Mud transport ship navigation time prediction method and system based on reinforcement learning and multi-mode fusion

Abstract

The invention provides a mud transport ship navigation time prediction method and system based on reinforcement learning and multi-mode fusion, and the method comprises the steps: integrating multi-source feature data through employing an attention mechanism, and extracting the incidence relation between features, so as to fully describe and construct a comprehensive state space containing internal and external factors; the model represents ship characteristics in a vector embedding mode, and the tide condition is calculated and represented by combining a prior tide-time physical principle; various data representation results adopt a self-attention mechanism to learn correlation, and a final output vector is used for describing a current environment; and a depth deterministic strategy gradient algorithm is adopted to output navigation behaviors including whether to wait or not and selection of an optimal navigation speed and a navigation path. According to the method, the usability of the algorithm and the completeness of the theory are comprehensively considered, the negative value of the distance between the deduced position and the actual position at the same time is adopted as a reward function, and it is ensured that the model can learn a stable navigation mode within a long time range.

Description

Technical Field

[0001] The present invention relates to the field of inland waterway intelligent shipping technology, and in particular to a method and system for predicting the sailing time of a mud transport ship based on reinforcement learning and multi-mode fusion. Background Art

[0002] Inland waterways, as crucial water transport infrastructure, serve multiple functions, including cargo transportation and flood control and drainage. Dredging projects are a key means of improving the navigability of inland waterways. However, the scheduling of mud carriers during dredging operations directly impacts project efficiency and cost control. Accurately predicting mud carrier travel times, particularly in complex inland waterway environments, poses a pressing technical challenge.

[0003] At present, research on ship sailing time prediction mainly relies on historical trajectory data analysis (such as AIS data) or statistical model-based methods. For example, by clustering a large number of historical navigation records, typical path patterns are extracted and time estimation is performed in combination with current environmental conditions. However, these methods have the following shortcomings: (1) Static modeling problem: Traditional methods usually assume that environmental conditions (such as flow rate and channel status) remain unchanged in a short period of time, while inland waterways are greatly affected by factors such as tidal cycles and seasonal hydrological changes, and are significantly dynamic. (2) Ignoring multi-source feature fusion: Existing studies rarely comprehensively consider multi-dimensional information such as tidal flow rate, ship behavior characteristics and channel constraints, resulting in limited prediction accuracy. (3) Lack of real-time optimization capabilities: Traditional methods are mostly offline analysis, which makes it difficult to respond to environmental changes in real time (such as sudden changes in water flow direction or channel construction restrictions), and cannot meet the needs of dynamic scheduling in dredging operations.

[0004] Tidal flow velocity is an important external factor affecting inland waterway navigation, especially in the waterway area close to the estuary. The periodic changes in tides will significantly change the local water flow speed and direction, thereby changing the choice of the optimal navigation mode for mud transport ships. For example, sailing downstream at high tide can significantly shorten the sailing time. In addition, the navigation behavior of mud transport ships in dredging scenarios is highly complex and uncertain. On the one hand, the operating habits of different ship drivers (such as acceleration / deceleration frequency, avoidance strategy) will lead to differences in sailing time; on the other hand, dredging operations require mud transport ships to frequently travel between dredging areas and unloading areas, which makes the ship behavior show a certain regularity. However, existing research has not fully explored these behavioral characteristics to optimize sailing time prediction.

[0005] In recent years, reinforcement learning, an emerging artificial intelligence algorithm for solving complex planning problems, has shown promising application prospects in the field of intelligent shipping. Decision-making models trained with reinforcement learning can accurately predict navigation times based on real-time environmental feedback (such as tidal flow and waterway conditions). However, the application of reinforcement learning in inland dredging scenarios is still in its exploratory stages, particularly with the lack of mature solutions for integrating multi-source features.

[0006] Patent application document CN116307069A discloses a method for predicting ship sailing times based on multi-task learning. The method comprises: acquiring historical AIS data; processing the historical AIS data into historical traffic flow and sailing time data for each time period of each channel according to the divided waterways, and collecting weather data for each channel in the future; inputting the historical traffic flow, sailing time data, and weather data into a traffic flow prediction sub-model and a sailing time prediction sub-model for training; constructing an overall model framework; training the overall model framework and outputting the final sailing time for each channel in the future; and calculating the overall sailing time for the entire route using a rolling forecast method. However, this patent fails to fully resolve existing technical problems and fails to meet the requirements of the present invention. Summary of the Invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for predicting the sailing time of a mud transport ship based on reinforcement learning and multi-mode fusion.

[0008] The method for predicting the sailing time of a mud transport vessel based on reinforcement learning and multi-mode fusion provided by the present invention includes:

[0009] Step 1: Integrate environmental data, ship characteristics, and channel constraints, extract the correlation between multi-source features through the attention mechanism, and construct a comprehensive state space;

[0010] The environmental data includes daily tide time and inland river environmental characteristics; the ship characteristics include ship ID and longitude and latitude position; the channel constraint conditions include navigation direction;

[0011] Step 2: Using the deep deterministic policy gradient (DDPG) algorithm, the navigation behavior decision is output based on the comprehensive state space, including speed selection and path planning. The DDPG algorithm optimizes the navigation decision through the reward function to minimize the error between the deduced position and the actual position, and uses the experience replay mechanism to improve the training stability.

[0012] Preferably, step 1 includes: calculating the daily high tide and low tide times based on the lunar calendar date to generate a tidal state vector; decomposing the timestamp into four dimensions of month, day, week, and hour, embedding them into vectors respectively and splicing them into a time feature vector; representing the ship ID by vector embedding to generate a ship feature vector; mapping the latitude and longitude position information into a position feature vector through linear transformation; and using a self-attention mechanism to fuse the tidal state vector, time feature vector, ship feature vector and position feature vector to generate a comprehensive state vector.

[0013] Preferably, the reward function is defined as the negative value of the square of the distance between the deduced position and the actual position, and the calculation formula is:

[0014] r=-d 2 ((lat t ,lon t ),(lat t+1 ,lon t+1 ))

[0015] Among them, r is the reward function, d is the distance calculation function, lat t ,lon t is the current latitude and longitude, lat t+1 ,lon t+1 To deduce the latitude and longitude of the location.

[0016] Preferably, the DDPG algorithm includes an Actor network and a Critic network:

[0017] The Actor network inputs the comprehensive state vector and outputs a continuous speed value, where a negative value indicates waiting;

[0018] The critic network evaluates the cumulative reward expectation of the state-action pair and updates the parameters by minimizing the mean square error between the target Q value and the predicted Q value;

[0019] The target network parameters gradually approach the main network parameters through soft updating. The update formula is:

[0020]

[0021] Where τ is the update rate, and Represent the parameters of the main network and the target network respectively.

[0022] Preferably, the policy gradient optimization formula of the Actor network is:

[0023]

[0024] in, represents the gradient of the Critic network Q for action a, Represents the gradient of the Actor network μ with respect to the parameter θ; s is the state vector at a certain moment, v is the speed decision result of the corresponding Actor network; E is the expected cumulative reward output by the Critic network in the future.

[0025] The mud transport ship sailing time prediction system based on reinforcement learning and multi-mode fusion provided by the present invention includes:

[0026] Module M1: Integrates environmental data, ship characteristics, and channel constraints, extracts the correlation between multi-source features through the attention mechanism, and constructs a comprehensive state space;

[0027] The environmental data includes daily tide time and inland river environmental characteristics; the ship characteristics include ship ID and longitude and latitude position; the channel constraint conditions include navigation direction;

[0028] Module M2: Adopts the deep deterministic policy gradient (DDPG) algorithm to output navigation behavior decisions based on the comprehensive state space, including speed selection and path planning. The DDPG algorithm optimizes navigation decisions through a reward function to minimize the error between the deduced position and the actual position, and uses an experience replay mechanism to improve training stability.

[0029] Preferably, the module M1 includes: calculating the daily high tide and low tide time based on the lunar calendar date to generate a tidal state vector; decomposing the timestamp into four dimensions of month, day, week and hour, embedding them into vectors respectively and splicing them into a time feature vector; representing the ship ID by vector embedding to generate a ship feature vector; mapping the latitude and longitude position information into a position feature vector through linear transformation; and using a self-attention mechanism to fuse the tidal state vector, time feature vector, ship feature vector and position feature vector to generate a comprehensive state vector.

[0030] Preferably, the reward function is defined as the negative value of the square of the distance between the deduced position and the actual position, and the calculation formula is:

[0031] r=-d 2 ((lat t ,lon t ),(lat t+1 ,lon t+1 ))

[0032] Among them, r is the reward function, d is the distance calculation function, lat t ,lon t is the current latitude and longitude, lat t+1 ,lon t+1 To deduce the latitude and longitude of the location.

[0033] Preferably, the DDPG algorithm includes an Actor network and a Critic network:

[0034] The Actor network inputs the comprehensive state vector and outputs a continuous speed value, where a negative value indicates waiting;

[0035] The critic network evaluates the cumulative reward expectation of the state-action pair and updates the parameters by minimizing the mean square error between the target Q value and the predicted Q value;

[0036] The target network parameters gradually approach the main network parameters through soft updating. The update formula is:

[0037]

[0038] Where τ is the update rate, and Represent the parameters of the main network and the target network respectively.

[0039] Preferably, the policy gradient optimization formula of the Actor network is:

[0040]

[0041] in, represents the gradient of the Critic network Q for action a, Represents the gradient of the Actor network μ with respect to the parameter θ; s is the state vector at a certain moment, v is the speed decision result of the corresponding Actor network; E is the expected cumulative reward output by the Critic network in the future.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) Higher prediction accuracy: By integrating multi-dimensional information such as tidal velocity, ship behavior patterns, and channel constraints, the accuracy of sailing time prediction is significantly improved. The prediction error of sailing time on multiple trajectories can be reduced by more than 20%.

[0044] (2) Effective multi-source data representation capability: The model can extract features of multiple data types through prior knowledge embedding and vector embedding, and learn association relationships based on the self-attention mechanism, so that the model can fully perceive the state space of navigation and generate effective representation vectors.

[0045] (3) Flexible dynamic adaptability: Utilizing the DDPG reinforcement learning algorithm, the reward function is designed to ensure the consistency between the agent’s behavior and the actual navigation behavior. Ultimately, the model can perceive navigation conditions, location, tides and other characteristics in real time and make correct navigation decisions accordingly, thus avoiding the limitations of traditional static modeling methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0047] Figure 1 It is the overall framework of the present invention;

[0048] Figure 2 Schematic diagram of the multi-source data fusion perception module;

[0049] Figure 3 Schematic diagram of the capture module for navigation mode. DETAILED DESCRIPTION

[0050] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0051] Example

[0052] This paper proposes a prediction algorithm for the sailing time of an inland mud transport vessel based on reinforcement learning and multi-modal feature fusion, aiming to address the shortcomings of existing technologies in terms of interactivity in dynamic environments, thereby improving the rationality of the navigation process deduction and further improving the accuracy of the sailing time prediction. Figure 1 As shown, the system consists of two main components: a state-space representation based on multi-source input fusion and a reinforcement learning neural network for capturing and learning navigation patterns. Together, these two components form a closed-loop planning and deduction framework encompassing environmental perception, navigation behavior decision-making, and decision-result feedback. This allows the system to perceive complex environments in real time, simulate real-world navigation patterns, and achieve high-precision predictions of navigation times based on the deduction results.

[0053] Specifically, Figure 1 The left half is the multimodal feature fusion module. It uses an attention mechanism to integrate multi-source heterogeneous data (including environmental data, ship characteristics, and channel constraints), fully explores the correlation between each feature, and forms an effective state space representation. Figure 1The right half is the reinforcement learning decision module, which uses the Deep Deterministic Policy Gradient (DDPG) algorithm to train the model. First, the state-space representation is fed into the Actor network, which then outputs a distribution of navigation behavior decisions. The state-space representation vector and the Monte Carlo sampling-based navigation behavior decisions are then fed into the Critic network, which then outputs a Q-value. The model is trained to ensure that the Actor network outputs decisions with the highest possible Q-values and that the Critic network outputs consistent and stable Q-values. Finally, the trained model is used to input environmental information and perform deductions based on the navigation behavior output by the model. The deduction results are used as the predicted navigation time.

[0054] Multi-source data fusion perception module

[0055] like Figure 2 The multi-source feature fusion module of this invention aims to efficiently fuse multi-source features through embedded representation and self-attention mechanism, generating a comprehensive vector representation that fully reflects the current environment state. This module specifically includes the following steps: feature embedding and extraction based on prior knowledge embedding, feature fusion based on the attention mechanism, and the construction of the final state space.

[0056] (1) Feature embedding and extraction:

[0057] In order to unify different types of features into the same dimensional space, different representation methods are used to process various types of input data and align them into vectors of the same dimension for subsequent construction of association relationships.

[0058] First, the daily tides are affected by the gravitational pull of the Earth and the Moon. When the Moon, Earth, and Sun are in a straight line, the tides are at their maximum, and when they are at right angles, the tides are at their minimum. To improve the model's usability, this paper abandons the method of estimating tides by querying tide tables and instead directly uses the lunar calendar date to estimate the current day's tides. The corresponding method is:

[0059] t u =(Moon(Date)%15-3)×0.8

[0060] Among them, t u Indicates the high tide time, Date indicates the current date, and Moon(·) indicates converting the input date to the corresponding lunar calendar date. In addition, the low tide time t d This usually occurs 6 hours after high tide time, i.e.:

[0061] t d =t u +6

[0062] According to this method, by inputting the current timestamp t, the current high tide or low tide state can be calculated. The calculation method is:

[0063]

[0064] Among them, s tide is a vector with all internal elements being learnable parameters. The above process represents the flood state with s tide and represents the ebb state with -s tide .

[0065] Since the navigation behavior is affected by the specific time at the same time, for example, whether it is working hours at present, the state space needs to include specific timestamp information. The implementation method is to split the current timestamp t into month t month , day t day , week t week , hour t hour into four dimensions, embed the four time dimensions into four vectors respectively, and splice them together, then there is:

[0066]

[0067] Among them, s time is the state representation vector generated according to the timestamp information.

[0068] In addition, since the drivers of different ships are different, resulting in different corresponding navigation habits, the model needs to distinguish different ships. The present invention uses the ship ID as the input and realizes the distinction of the ship's own characteristics through the method of vector embedding:

[0069] s ship = Embedding(ID ship )

[0070] Among them, s ship represents the representation result of the ship's own characteristics.

[0071] Finally, for the latitude and longitude information, since it is a continuous value and the values of latitude and longitude can effectively reflect the distance from the current position to the ocean, it is necessary to directly map the two-dimensional position information to the target dimension through a linear transformation:

[0072] s pos = W pose (lat, lon) T + b pose

[0073] Among them, s pos represents the feature extraction result obtained according to the latitude and longitude position information; lat and lon are latitude and longitude; W pose is the weight matrix; b pose is the bias vector.

[0074] (2) Feature fusion based on attention mechanism

[0075] In order to capture the complex correlation between multi-source features, the self-attention mechanism is introduced for feature fusion. The objects of feature fusion include tidal state s tide , the concatenated timestamp vector s time , the embedding vector s of the ship ID ship , and the mapping vector s of the position information pos First, each vector is linearly transformed to obtain the query vector (Query), key vector (Key) and value vector (Value). Then the attention weight matrix is calculated:

[0076]

[0077] Where d is the dimension of the key vector. Based on the output of the attention mechanism, a multi-layer perceptron with residual connections is used to implement nonlinear mapping, and layer normalization is used to stabilize the training process.

[0078] (3) State space construction

[0079] After fusion through the self-attention mechanism, the output vectors of the last layer at each position are concatenated and passed through a linear layer to generate a comprehensive state vector that describes the overall state of the current environment:

[0080] s=W s concat(s tide ,s time ,s ship ,s pos )+b s

[0081] Among them, W s is the weight matrix, b s is the bias vector. This vector contains three types of information: the current tidal state, the current timestamp, and the ship's characteristics (ID, current location). Ultimately, this state vector serves as input to the reinforcement learning algorithm, used to deduce sailing time and optimize decision-making.

[0082] The above-mentioned multi-feature fusion module achieves the following technical effects: Feature association mining: using the self-attention mechanism to effectively capture the complex correlation between multi-source features and improve feature expression capabilities; State space integrity: taking into account the characteristics of the environment, ships, etc., a vector representation that fully reflects the current environmental state is constructed; Dynamic adaptability: combining prior knowledge to infer tidal status and enhance the model's adaptability to dynamic environments.

[0083] Navigation mode extraction module based on reinforcement learning

[0084] Most existing flight time prediction algorithms rely on an end-to-end training model, taking various features as input and outputting a flight time estimate. While these methods work well in static environments, the actual flight process is dynamic, and the corresponding navigation decisions also change dynamically with the current state. While this doesn't necessarily mean that end-to-end models are ineffective in such scenarios, their lack of flexibility inevitably leads to unstable predictions and a decrease in long-term average accuracy.

[0085] (1) Problem background and basic settings

[0086] like Figure 3 ,First, in this scenario, as mentioned above, the state space S is the current ,navigation state and the surrounding environment, including ship characteristics, ,tidal conditions, location information, and timestamp information.

[0087] In addition, the behavior space A in this scenario corresponds to the following two aspects: whether to wait and the navigation speed v. Since the speed of a ship in the waiting state is necessarily 0 and the navigation direction is fixed (between the dredging point and the dumping point), to ensure the robustness of the model and the parseability of the output results, this paper sets the behavior space to only the navigation speed and defines that when the navigation speed is less than or equal to 0, it indicates that the ship is in the waiting behavior.

[0088] At the same time, the decision interval for navigation behavior is one hour. Once the navigation behavior at the previous moment is determined, the mud transport ship will travel along the waterway at a constant speed according to the output speed for the next hour. Therefore, based on the map information of the waterway, the position of the ship one hour later can be deduced according to the above principle. This deduction process is the state transition function:

[0089] s t+1 =f(s t ,v t )

[0090] Among them, f() represents the state change law, that is, according to the current position, navigation speed v t , the channel situation predicts the position at the next moment, s t represents the state space vector at time t, s t+1 Represents the state space vector at the next moment.

[0091] Finally, unlike most planning problems, the ultimate goal of the reinforcement learning model established in this invention is not to obtain the optimal navigation behavior decision, but to obtain the navigation decision that is closest to human behavior. Therefore, the reward function is set to the negative square of the distance between the deduced position and the actual position at the same time (i.e., the smaller the distance, the greater the reward). This encourages the model to output a strategy that is close to the actual navigation behavior, while also imposing a penalty on deduced results that deviate significantly from the actual position:

[0092] r=-d 2 ((lat t ,lon t ),(lat t+1 ,lon t+1 ))

[0093] Among them, r is the reward function, d is the distance calculation function, and the present invention adopts the waterway distance, that is, the total distance required to travel along the river between two locations.

[0094] (2) Model building process

[0095] Because the output navigation speed in navigation scenarios is a continuous value in the behavior space, this paper uses the Deep Deterministic Policy Gradient (DDPG) algorithm, a reinforcement learning algorithm that combines value functions and policy optimization. It is suitable for navigation decision-making problems in continuous action spaces. The DDPG algorithm is an improvement on the Deep Q-Learning (DQN) algorithm, specifically designed to solve planning problems in scenarios with continuous state and behavior spaces. Therefore, the DDPG algorithm is still a special case of the basic reinforcement learning algorithm Q-Learning.

[0096] The core concept of the DDPG algorithm is the actor-critic framework, which consists of two models: the actor network and the critic network. The actor network is used to define the policy function, while the critic network is used to fit the value function. The two networks improve policy performance through alternating optimization.

[0097] The policy function μ(Actor network) is a deterministic mapping that represents the policy function in a given state s. t The optimal sailing speed v t , its mathematical expression is:

[0098] v t =μ(s t |θ)

[0099] Here, θ represents the parameters of the Actor network. In this scenario, the policy function outputs a continuous value representing the navigation speed, where a negative value indicates no navigation and waiting in place.

[0100] The value function (Critic network) is used to evaluate the state-action pair (s t ,v t ), input the state vector s at a certain moment and the speed decision result v of the corresponding Actor network, and the Critic network outputs the expected cumulative reward in the future, which is mathematically expressed as:

[0101]

[0102] Where γ∈[0,1] is the discount factor, r t is the immediate reward at time t.

[0103] The critic network minimizes the gap between the target Q value and the predicted Q value through the mean square error loss function:

[0104]

[0105] Where y represents the target Q value and can be defined as:

[0106]

[0107] Here, s' and v' represent the state and behavior (navigation speed) at the next moment, respectively, and Q' represents the target Critic network. Represents the parameters of the target network. The model architecture and initial parameters of Q′ are exactly the same as those of Q, but its parameters are gradually approximated to the parameters of the main network using a soft update method. That is, after one or more trajectories are trained, the parameters of Q′ are set to the parameters of Q:

[0108]

[0109] Wherein, τ is the update rate, which is taken as 0.01 in the present invention. and Represent the parameters of the main network and the target network respectively.

[0110] The goal of optimizing the policy function in this invention is to maximize the value function By using the chain rule, the policy gradient formula can be expressed as:

[0111]

[0112] in, represents the gradient of the Critic network for action a, Represents the gradient of the Actor network with respect to the parameter θ. Through the above policy gradient formula, the Actor network can gradually optimize the policy parameters based on the feedback of the Critic network.

[0113] In order to improve sample utilization and reduce data correlation, this algorithm adopts the experience replay mechanism. The experience generated by each interaction (s t ,v t ,r t ,s t+1 ) is stored in the experience replay buffer, and a batch of experiences is randomly sampled for batch update during training.

[0114] Overall algorithm process

[0115] Based on the above theoretical modeling process, the overall process of the present invention is as follows: First, initialize the parameters θ and θ of the Actor network μ and the Critic network Q. Initialize the target network μ′ and Q′, and initialize the parameters of the target network to the main network parameters, and clear the experience buffer. Then, based on the timestamp, ship ID, location and other information, based on the multi-feature fusion module, generate a representation of the state and obtain the optimal strategy v considered by the current model through the Actor network. t =μ(s t |θ)+ε, where ε represents the exploration noise. It obeys a normal distribution with a mean of 0 and a standard deviation of 5. The Critic network is used to evaluate the overall quality of the “state-navigation speed” pair and record the interaction experience (s t ,v t ,r t ,s t+1 ) and store it in the buffer. Next, a batch of experience (s) is randomly sampled from the buffer. t ,v t ,r t ,s t+1 ), by minimizing the loss function based on the negative distance error Update the parameters of the Critic network Update the Actor network parameters θ using the policy gradient formula. Finally, regularly update the target network parameters using soft updates and periodically save the model parameters for subsequent reasoning and deployment.

[0116] During the inference phase, the model inputs state information every hour, and the model outputs the current speed. Based on the current speed, the model then infers the ship's travel distance and position one hour later. The new position, along with the corresponding time and tide information, is then fed back into the model to determine the next navigation decision. This process continues until the ship reaches the target location (dredging or blowing), and the total time spent in the simulation is returned as the predicted navigation time.

[0117] Effect verification

[0118] This model is programmed in Python3, and the neural network structure is built using the interface provided by PyTorch, and runs on the NVIDIA A6000 GPU. The present invention is verified on 14 real sampled ship trajectories. All trajectories are split into two segments in a ratio of 7:3 based on the timestamps, of which the first half is used to train the reinforcement learning model, and the second half is used to test the accuracy of the sailing time prediction. The experimental results show that the present invention can control the average error of the sailing time prediction to about 2.3 hours. Compared with the traditional prediction method based on "distance-average speed", the prediction error is reduced by about 50%, which shows that the present invention has a high adaptability to the dynamically changing sailing environment and the stability of the prediction effect.

[0119] This invention achieves accurate prediction and dynamic optimization of inland mud vessel travel times through the organic combination of multi-source feature fusion modeling and reinforcement learning algorithms. The multi-source feature fusion module fully utilizes environmental data, vessel characteristics, and channel constraints to construct a comprehensive state space. The reinforcement learning module outputs the optimal navigation strategy through a deep deterministic policy gradient algorithm, ensuring the model's efficient operation in complex dynamic environments. This design concept and framework significantly improves the rationality and accuracy of travel time predictions, providing intelligent technical support for inland dredging projects.

[0120] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0121] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for predicting the sailing time of a mud transport vessel based on reinforcement learning and multi-mode fusion, characterized in that: include: Step 1: Integrate environmental data, ship characteristics, and channel constraints, extract the correlation between multi-source features through the attention mechanism, and construct a comprehensive state space; The environmental data includes daily tide time and inland river environmental characteristics; the ship characteristics include ship ID and longitude and latitude position; the channel constraint conditions include navigation direction; Step 2: Using the deep deterministic policy gradient (DDPG) algorithm, the navigation behavior decision is output based on the comprehensive state space, including speed selection and path planning. The DDPG algorithm optimizes the navigation decision through the reward function to minimize the error between the deduced position and the actual position, and uses the experience replay mechanism to improve the training stability.

2. The method for predicting the sailing time of a mud transport vessel based on reinforcement learning and multi-mode fusion according to claim 1 is characterized in that: The step 1 includes: calculating the daily high tide and low tide times based on the lunar calendar date to generate a tidal state vector; decomposing the timestamp into four dimensions: month, day, week, and hour, embedding them into vectors respectively, and splicing them into a time feature vector; representing the ship ID through vector embedding to generate a ship feature vector; mapping the latitude and longitude position information into a position feature vector through linear transformation; and using a self-attention mechanism to fuse the tidal state vector, the time feature vector, the ship feature vector, and the position feature vector to generate a comprehensive state vector.

3. The method for predicting the sailing time of a mud transport vessel based on reinforcement learning and multi-mode fusion according to claim 1 is characterized in that: The reward function is defined as the negative value of the square of the distance between the predicted position and the actual position, and the calculation formula is: r=-d 2 ((years t lon t ),(years t+1 lon t+1 )) Among them, r is the reward function, d is the distance calculation function, lat t ,lon t is the current latitude and longitude, lat t+1 ,lon t+1 To deduce the latitude and longitude of the location.

4. The method for predicting the sailing time of a mud transport vessel based on reinforcement learning and multi-mode fusion according to claim 3 is characterized in that: The DDPG algorithm includes the Actor network and the Critic network: The Actor network inputs the comprehensive state vector and outputs a continuous speed value, where a negative value indicates waiting; The critic network evaluates the cumulative reward expectation of the state-action pair and updates the parameters by minimizing the mean square error between the target Q value and the predicted Q value; The target network parameters gradually approach the main network parameters through soft updating. The update formula is: where τ is the update rate, and Represent the parameters of the main network and the target network respectively.

5. The method for predicting the sailing time of a mud transport vessel based on reinforcement learning and multi-mode fusion according to claim 4 is characterized in that: The policy gradient optimization formula of the Actor network is: in, represents the gradient of the Critic network Q for action a, Represents the gradient of the Actor network μ with respect to the parameter θ; s is the state vector at a certain moment, v is the speed decision result of the corresponding Actor network; E is the expected cumulative reward output by the Critic network in the future.

6. A mud transport ship sailing time prediction system based on reinforcement learning and multi-mode fusion, characterized by: include: Module M1: Integrate environmental data, ship characteristics and channel constraints, extract the correlation between multi-source features through the attention mechanism, and construct a comprehensive state space; The environmental data includes daily tide time and inland river environmental characteristics; the ship characteristics include ship ID and longitude and latitude position; the channel constraint conditions include navigation direction; Module M2: Adopts the deep deterministic policy gradient (DDPG) algorithm to output navigation behavior decisions based on the comprehensive state space, including speed selection and path planning. The DDPG algorithm optimizes navigation decisions through a reward function to minimize the error between the deduced position and the actual position, and uses an experience replay mechanism to improve training stability.

7. The mud transport ship sailing time prediction system based on reinforcement learning and multi-mode fusion according to claim 6 is characterized in that: The module M1 includes: calculating the daily high tide and low tide times based on the lunar calendar date to generate a tidal state vector; decomposing the timestamp into four dimensions: month, day, week, and hour, embedding them into vectors respectively and splicing them into a time feature vector; representing the ship ID through vector embedding to generate a ship feature vector; mapping the latitude and longitude position information into a position feature vector through linear transformation; and using a self-attention mechanism to fuse the tidal state vector, time feature vector, ship feature vector, and position feature vector to generate a comprehensive state vector.

8. The mud transport ship sailing time prediction system based on reinforcement learning and multi-mode fusion according to claim 6 is characterized in that: The reward function is defined as the negative value of the square of the distance between the predicted position and the actual position, and the calculation formula is: r=-d 2 ((years t lon t ),(years t+1 lon t+1 )) Among them, r is the reward function, d is the distance calculation function, lat t ,lon t is the current latitude and longitude, lat t+1 ,lon t+1 To deduce the latitude and longitude of the location.

9. The mud transport ship sailing time prediction system based on reinforcement learning and multi-mode fusion according to claim 8 is characterized in that: The DDPG algorithm includes the Actor network and the Critic network: The Actor network inputs the comprehensive state vector and outputs a continuous speed value, where a negative value indicates waiting; The critic network evaluates the cumulative reward expectation of the state-action pair and updates the parameters by minimizing the mean square error between the target Q value and the predicted Q value; The target network parameters gradually approach the main network parameters through soft updating. The update formula is: where τ is the update rate, and Represent the parameters of the main network and the target network respectively.

10. The mud transport ship sailing time prediction system based on reinforcement learning and multi-mode fusion according to claim 9 is characterized in that: The policy gradient optimization formula of the Actor network is: in, represents the gradient of the Critic network Q for action a, Represents the gradient of the Actor network μ with respect to the parameter θ; s is the state vector at a certain moment, v is the speed decision result of the corresponding Actor network; E is the expected cumulative reward output by the Critic network in the future.

Images