Ship dynamic path adjusting method combining visual navigation and radar data

Abstract

The invention discloses a ship dynamic path adjusting method combining visual navigation and radar data. The ship dynamic path adjusting method comprises the following steps: S1, collecting data; s2, carrying out pretreatment; s3, performing data fusion to generate an environment perception model; s4, combining the generative adversarial network and the space-time diagram neural network to construct an environment prediction model; s5, performing initial path planning by using a heuristic search algorithm to generate an initial path; s6, refining and optimizing the path by using a fast extension random tree algorithm to generate an optimal path; s7, generating future possible environment change according to the environment prediction model, and calculating a coexistence reserve path for each prediction scene in combination with the environment perception model; s8, when the current path is predicted to have risks in the future, switching to a preset standby path in advance; and S9, automatically generating a navigation control instruction, and transmitting the instruction to a ship automatic driving system. According to the invention, intelligent optimization of the ship path is realized through a multi-sensor fusion and environment prediction method.

Description

Technical Field

[0001] The present invention relates to the field of ship navigation technology, and in particular to a ship dynamic path adjustment method combining visual navigation and radar data. Background Art

[0002] In the field of modern ship navigation, with the increase in global shipping demand and the complexity of the navigation environment, automated and intelligent ship navigation technology has become an important development direction of the maritime industry. Traditional ship navigation mainly relies on manual operation, combined with GPS, radar, sonar and other basic navigation tools for path planning. However, this method not only relies on the operator's experience and judgment, but also has problems such as slow response speed and difficulty in coping with complex and changing marine environments. Especially in areas with bad sea conditions, low visibility or dense ships, the limitations of traditional navigation methods are more prominent.

[0003] In the prior art, ship automated navigation systems usually use single sensor data, such as relying only on radar data or only on visual data for environmental perception. This single data source is prone to perception blind spots or misjudgments when facing complex marine environments. For example, in conditions of low visibility, the accuracy of pure visual navigation will be seriously affected, and radar systems may also have difficulty providing accurate perception results when facing small obstacles or complex ocean waves. This limitation directly affects the accuracy and safety of ship path planning and increases the risk of navigation.

[0004] In order to make up for the shortcomings of single sensor navigation, multi-sensor fusion technology has also begun to be introduced in existing technologies to comprehensively process data from different sensors to improve the accuracy of environmental perception. For example, radar data is fused with visual data, and methods such as Kalman filtering are used to improve the accuracy of environmental models. However, traditional multi-sensor fusion technology often only performs simple fusion at the data level, lacks in-depth understanding and prediction capabilities of multi-source data, and is difficult to cope with complex changes in dynamic environments. In addition, in actual applications, existing multi-sensor fusion systems still face problems such as difficulty in synchronizing sensor data and large impact of data noise, resulting in insufficient reliability and real-time performance of environmental perception models.

[0005] In terms of path planning, traditional ship navigation path planning algorithms such as Dijkstra algorithm and A* algorithm can generate the shortest path or optimal path in a static environment, but in a dynamic and complex marine environment, especially in the face of changing wind speed, wave height and other ship activities, the performance of these traditional algorithms is often unsatisfactory. They lack forward-looking considerations of environmental changes, which easily leads to the planned path being unable to respond to sudden environmental changes in a timely manner, increasing the frequency and complexity of path adjustments. Although some studies in recent years have introduced dynamic programming algorithms, such as rapidly expanding random trees (RRT) for path optimization, these algorithms still have the problem of disconnection between path planning and real-time environmental perception in practical applications, and it is difficult to effectively integrate environmental prediction models, resulting in delayed or inflexible path adjustments.

[0006] Another significant problem in existing technologies is the lack of effective prediction and prejudgment capabilities for future environmental changes. Most existing ship navigation systems rely only on current environmental perception data for decision-making, and lack forward-looking processing for possible future environmental changes. This limitation is particularly evident in dynamic and complex marine environments, where potential risks cannot be effectively avoided and temporary path adjustments are often required in the face of emergencies, which not only increases the complexity of navigation and fuel consumption, but may also affect the safety of the ship.

[0007] At the same time, existing automatic navigation systems usually lack backup path mechanisms and fault-tolerant designs for multi-path selection when making path adjustments. Traditional systems are mostly based on the selection of a single optimal path. When the path is not feasible due to sudden environmental changes, it is often necessary to temporarily recalculate the path, resulting in delayed response time and failure to avoid dangerous areas in time. In addition, there is less attention paid to human-computer interaction in existing technologies. Navigation systems are usually based on automation, while ignoring collaboration with ship operators in complex environments. There is a lack of suggestions and alarm mechanisms based on environmental predictions, making it difficult for operators to understand the decision logic of the navigation system and the reasons for path adjustments in a timely manner, thereby affecting the overall safety of navigation and the controllability of operations.

[0008] Therefore, how to provide a method for dynamic path adjustment of a ship that combines visual navigation and radar data is an urgent problem that technicians in this field need to solve. Summary of the invention

[0009] One purpose of the present invention is to propose a method for dynamic path adjustment of ships that combines visual navigation and radar data. The present invention realizes intelligent optimization and dynamic adjustment of ship paths through advanced technologies such as multi-sensor data fusion, generative adversarial networks, and spatiotemporal graph neural networks. By deeply integrating visual navigation and radar data and combining them with environmental prediction models, the system can proactively identify potential risks and optimize paths through heuristic search and rapidly expanding random tree algorithms. This method significantly improves the navigation capability of ships in complex sea conditions, has the advantages of high navigation safety, strong environmental adaptability, and flexible path adjustment, and ensures efficient and safe navigation of ships in various dynamic environments.

[0010] A method for adjusting a ship's dynamic path by combining visual navigation and radar data according to an embodiment of the present invention comprises the following steps:

[0011] S1. Use ship cameras to obtain real-time image data, use radar systems to obtain radar echo data of the surrounding environment, and collect environmental data through marine meteorological stations;

[0012] S2, preprocessing the collected image data, radar echo data and environmental data, including target detection, image segmentation and edge detection for image data, signal filtering, enhancement and feature extraction for radar echo data, and time series analysis for environmental data;

[0013] S3, performing multi-sensor data fusion on the pre-processed image data and radar echo data through the extended Kalman filter algorithm to generate an environment perception model;

[0014] S4, based on the environment perception model, historical navigation data and environmental data, the environment prediction model is constructed by combining the generative adversarial network and the spatiotemporal graph neural network;

[0015] S5. Based on the environment perception model and the environment prediction model, use the heuristic search algorithm to perform initial path planning and generate an initial path;

[0016] S6. Based on the initial path, the fast expanding random tree algorithm is used to refine and optimize the path to generate the optimal path;

[0017] S7, generating possible future environmental changes according to the environmental prediction model, and calculating and storing backup paths for each predicted scenario in combination with the environmental perception model;

[0018] S8. During navigation, the output of the environment perception model and the environment prediction model is monitored in real time. When it is predicted that the current path will have risks in the future, the preset backup path is switched in advance.

[0019] S9. Automatically generate navigation control instructions based on the environmental perception model, environmental prediction model and path planning results, and transmit the navigation control instructions to the ship's automatic driving system. In complex or high-risk environments, the system will issue warnings to the ship operator and provide path adjustment plans.

[0020] Optionally, the S3 specifically includes:

[0021] S31, using a convolutional neural network to extract features from the preprocessed image data;

[0022] S32, based on the image features extracted by the convolutional neural network, identify obstacles, ocean signs and other ships in navigation, and generate image feature vector I k ;

[0023] S33, perform amplitude detection and feature extraction on the preprocessed radar echo data to generate a radar feature vector R k , the strength of the radar signal is expressed as:

[0024]

[0025] Where θ represents the angle of the radar signal, r represents the distance of the target, and f c represents the radar carrier frequency, c represents the speed of light, R f (t) represents the radar signal after filtering;

[0026] S34, extract the image feature vector I k and radar feature vector R k Perform alignment in time and space;

[0027] S35, the aligned image feature vector I k and radar feature vector R k Input the extended Kalman filter to perform multi-sensor data fusion and estimation to generate an environmental perception model:

[0028]

[0029] Among them, x k+1 represents the state of the environment at time step k+1, x k represents the state vector at time step k, u k represents the input control of the system at time step k, h(x k ) represents the observed value predicted by the current state, f(x k ,u k ) represents the state transfer function, K k Represents the Kalman gain matrix:

[0030]

[0031] Among them, P k represents the state covariance matrix, H k represents the Jacobian matrix of the observation model, R k represents the observation noise covariance matrix;

[0032] S36. Finally, an environmental perception model is generated.

[0033] Optionally, the S4 specifically includes:

[0034] S41. Collect historical navigation data d , construct the time series form:

[0035] H d ={(p1,v1,θ1,w1,h1),(p2,v2,θ2,w2,h2),…,(p n ,v n ,θ n ,w n ,h n )};

[0036] Among them, n represents the time length of historical data, p i represents the position of the ship at time i, v i represents the speed of the ship at time i, θ i represents the heading vector of the ship at time i, w i represents the wind speed of the ship at time i, h i represents the wave height of the ship at time i;

[0037] S42. Obtaining environmental data E t , expressed as:

[0038] E t =(a t ,b t ,c t );

[0039] Among them, a t Indicates the current wind speed, b t Indicates the current wave height, c t Indicates the current ship flow density;

[0040] S43, the historical navigation data H d and environmental data E t Combined and input into the generative adversarial network model, the generator G receives the random noise vector z and the environment data E t As input, generate a future environment prediction sequence P f :

[0041] P f =G(z,E t )=σ(W G ·[z;E t ]+b G );

[0042] Among them, W G represents the weight matrix of the generator, b G represents the bias of the generator, σ represents the nonlinear activation function, [z; E t ] represents the random noise vector z and the environmental data E t The concatenation vector of

[0043] The discriminator D is used to evaluate the generated environment prediction sequence P f Compared with the real historical heading data H d differences;

[0044] S44, generate the environment prediction sequence P output by the adversarial network f Input into the space-time graph neural network for space-time dependency processing and construct the space-time graph G t (V,E), where V is a set of nodes, representing each time point and spatial position, and E is a set of edges, representing the association between time points and spatial positions.

[0045] S45. Perform temporal convolution on the features of each node to capture temporal dependencies:

[0046]

[0047] Among them, T represents the window size of temporal convolution, W τ represents the weight matrix of the temporal convolution kernel at time step τ, b τ represents the bias of temporal convolution, σ represents the activation function, represents the feature representation of node v at the t-τ time step, Represents the feature representation of node v at the t+1 time step;

[0048] S46. Perform spatial graph convolution on the features after temporal convolution to capture spatial dependencies:

[0049]

[0050] in, represents the feature representation of node v after the k+1th layer of graph convolution, N(v) represents the set of neighbor nodes of node v, and W k represents the spatial graph convolution weight matrix of the kth layer, p(u,v) represents the spatial position information between node u and node v, b k represents the spatial graph convolution bias of the kth layer;

[0051] S47. Generate a possible future environmental change scenario set S based on the output of the spatiotemporal graph neural network f ={S1,S2,…,S k}, each scene S i The possible changing trends of future predicted wind speed, wave height and ship flow density are included to build an environmental prediction model.

[0052] Optionally, the S5 specifically includes:

[0053] S51, obtaining initial data of current environmental status and future environmental changes from the environmental perception model and the environmental prediction model;

[0054] S52. Define the initial state vector x0 of the ship:

[0055] x0=(p0,v0,θ0,w0,h0);

[0056] Among them, p0 represents the initial position of the ship, v0 represents the initial speed of the ship, θ0 represents the heading vector of the ship, w0 represents the initial wind speed of the ship, and h0 represents the initial wave height of the ship;

[0057] S53, input the environmental perception data, environmental prediction data and initial state vector x0 into the heuristic search algorithm to perform initial path planning. The heuristic search algorithm uses an evaluation function f(x) to determine the priority of each state x. The evaluation function is composed of a path cost function g(x) and a heuristic estimation function h(x), which is expressed as:

[0058] f(x)=g(x)+h(x);

[0059] Among them, h(x) represents the heuristic estimate from the current state x to the target state g, expressed as Euclidean distance:

[0060]

[0061] Among them, x1, y1 represent the position coordinates of the current state x, and x2, y2 represent the position coordinates of the target state g;

[0062] S54, the path cost function g(x) is used to evaluate the cumulative path cost from the initial state x0 to the current state x:

[0063]

[0064] Among them, d(x i ,x i+1 ) indicates that the ship is in state x i To state x i+1 The distance between Represents state xi To state x i+1 The speed change rate between i+1 ) indicates that at time t i+1 The wind speed at the time, h(t i+1 ) indicates that at time t i+1 The wave height at the time, d(t i+1 ) indicates that at time t i+1 The ship flow density at the time, α, β, γ and δ represent weight coefficients, and k represents the total number of states in the cumulative path from the initial state x0 to the current state x;

[0065] S55, during the path planning process, dynamically adjusting the path cost function g(x) according to future environmental changes provided by the environmental prediction model;

[0066] S56, generate the initial path P of the ship through iterative calculation of the heuristic search algorithm init , starting from the current state x0 and eventually reaching the target state g, each state x in the path satisfies the optimal condition of the evaluation function f(x).

[0067] Optionally, the S6 specifically includes:

[0068] S61. Define the state space X of the target area, including all possible ship positions and headings, expressed as:

[0069] X={(x,y,θ,v)∣x,y∈R,θ∈[0,2π),v∈[v min ,v max ]};

[0070] Among them, x and y represent the coordinates of the ship's position, θ represents the heading angle, v represents the speed, and v min Indicates the minimum value of the ship's speed, v max Indicates the maximum value of the ship's speed;

[0071] S62. Randomly generate a new sample node x in the state space X rand =(x rand ,y rand ,θ rand ,v rand ), and find the path tree T that matches x rand The nearest node x near =(x near ,y near ,θ near ,v near ):

[0072]

[0073] Among them, λ1 and λ2 represent weight coefficients;

[0074] S63, by passing the sample node x rand Connect to the nearest node x near , generate a new node x new , the generation process takes into account the dynamic constraints of the ship and uses a nonlinear motion model:

[0075] x new =x near +Δt·f(x near ,u);

[0076] Among them, f(x near ,u) represents the motion equation of the ship, which is expressed as:

[0077]

[0078] Among them, v c represents the current speed, θ c Indicates the current heading angle, L c Indicates the length of the ship, represents the steering angle, a c represents acceleration, u represents control input;

[0079] S64. Check the generated new node x new Is it feasible under environmental constraints? Environmental constraints include the avoidance of dynamic obstacles, the stability of ship operation, and the constraints of future wind speed, wave height and ship flow density provided by the environmental prediction model. The environmental cost function C is introduced. env (x new ) for evaluation:

[0080]

[0081] Among them, d obs represents the distance between the new node and the nearest obstacle, represents the speed change rate, w(t) represents the wind speed at the current time, h(t) represents the wave height at the current time, d(t) represents the ship flow density at the current time, α new , β new , γ new , δ new and new represents the weight coefficient;

[0082] If C env (x new ) exceeds the predetermined threshold, the node is discarded and the sample point is regenerated; otherwise, x new Add to the path tree T;

[0083] S65, repeat steps S62 to S64 until a node x in the path tree T is goal Close enough to the target state g, the following termination conditions are met:

[0084] ∥x goal -g∥≤∈ and C env (x goal )≤η;

[0085] Among them, ∈ represents the approach threshold, and η represents the environmental cost threshold;

[0086] S66, backtrack from the path tree T to generate the optimal path P from the starting state x0 to the target state g opt .

[0087] Optionally, the S7 specifically includes:

[0088] S71. Define a real-time monitoring window W(t):

[0089] W(t)={x(t-Δt),x(t),x(t+Δt)};

[0090] Among them, Δt represents the width of the time window, and x(t) represents the state of the ship at time t;

[0091] S72. In each monitoring window W(t), a real-time risk assessment value of environmental changes is calculated based on the outputs of the environmental perception model and the environmental prediction model. The risk assessment value is calculated based on the potential impact of environmental changes on the ship path.

[0092] S73, comparing the real-time risk assessment value with a threshold value. If the risk assessment value exceeds the threshold value, it is determined that there is a potential risk in the current path and it is necessary to switch to an alternate path in advance.

[0093] S74. Use the selected backup path as the current new navigation path, notify the automatic driving system to switch the path, and update the real-time monitoring window W(t) to continue monitoring the new path.

[0094] The beneficial effects of the present invention are:

[0095] First, the present invention makes full use of the complementary advantages of visual navigation and radar data through multi-sensor data fusion technology to generate a more accurate and comprehensive environmental perception model. The application of the extended Kalman filter algorithm enables the fusion process of multi-sensor data to effectively reduce the impact of noise and enhance the accuracy and reliability of environmental perception. This deep fusion not only makes up for the limitations of a single sensor under different environmental conditions, but also provides a solid foundation for subsequent path planning and dynamic adjustment.

[0096] Secondly, the present invention introduces generative adversarial networks (GAN) and spatiotemporal graph neural networks (ST-GNN) to construct an environmental prediction model, which greatly improves the ability to predict future environmental changes. By generating future environmental change sequences through GAN and using ST-GNN to process complex spatiotemporal dependencies, the present invention can accurately predict key environmental factors such as future sea conditions, wind speeds, wave heights, and ship flow density. This forward-looking environmental prediction capability enables the system to pre-adjust the path before actual risks occur, significantly reducing the risks brought about by sudden environmental changes, reducing the frequency of temporary path adjustments, and improving the safety and efficiency of navigation.

[0097] In terms of path planning, the present invention combines the heuristic search algorithm and the rapidly expanding random tree (RRT) algorithm, which can not only generate the optimal path under static conditions, but also refine and optimize the path in real time in a dynamic environment. The heuristic search algorithm realizes the rapid planning of the initial path by comprehensively considering the path length and environmental impact, while the RRT algorithm further optimizes the path based on the initial path to cope with complex environmental changes. This multi-level and multi-stage path planning strategy ensures that the ship is always on the optimal path during the entire voyage and can effectively cope with complex changes in the marine environment.

[0098] In addition, the present invention designs a multi-path backup mechanism based on environmental prediction. By calculating and storing backup paths in advance, it can quickly switch to the optimal backup path when potential risks appear on the current path. This mechanism not only enhances the fault tolerance of the system under complex sea conditions, but also ensures the balance between the backup path length, operational stability and environmental adaptability through a multi-objective optimization algorithm, making the backup path switching more intelligent and efficient.

[0099] Finally, during the navigation process, the present invention timely detects and predicts potential risks on the path by real-time monitoring of the output of the environmental perception model and the environmental prediction model. When the system predicts that the current path will have risks in the future, it can make path adjustment decisions in advance to avoid temporary adjustments in emergency situations. In addition, the present invention also emphasizes the collaboration between the system and the ship operator. When the system detects high risks, it not only automatically adjusts the path, but also issues an alarm to the operator and provides path adjustment suggestions. This enhanced human-machine collaboration mode ensures that in complex or high-risk environments, the operator can understand the decision logic of the system in a timely manner, so as to make more reasonable operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0101] Figure 1A flow chart of a method for dynamic ship path adjustment combining visual navigation and radar data proposed by the present invention;

[0102] Figure 2 A flow chart for constructing an environment prediction model for a method for dynamic ship path adjustment combining visual navigation and radar data proposed by the present invention;

[0103] Figure 3 This is a schematic diagram of path planning and optimization of a ship dynamic path adjustment method combining visual navigation and radar data proposed by the present invention. DETAILED DESCRIPTION

[0104] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0105] refer to Figure 1-3 , a ship dynamic path adjustment method combining visual navigation and radar data, comprising the following steps:

[0106] S1. Use ship cameras to obtain real-time image data, use radar systems to obtain radar echo data of the surrounding environment, and collect environmental data through marine meteorological stations;

[0107] S2, preprocessing the collected image data, radar echo data and environmental data, including target detection, image segmentation and edge detection for image data, signal filtering, enhancement and feature extraction for radar echo data, and time series analysis for environmental data;

[0108] S3, performing multi-sensor data fusion on the pre-processed image data and radar echo data through the extended Kalman filter algorithm to generate an environment perception model;

[0109] S4, based on the environment perception model, historical navigation data and environmental data, the environment prediction model is constructed by combining the generative adversarial network and the spatiotemporal graph neural network;

[0110] S5. Based on the environment perception model and the environment prediction model, use the heuristic search algorithm to perform initial path planning and generate an initial path;

[0111] S6. Based on the initial path, the fast expanding random tree algorithm is used to refine and optimize the path to generate the optimal path;

[0112] S7, generating possible future environmental changes according to the environmental prediction model, and calculating and storing backup paths for each predicted scenario in combination with the environmental perception model;

[0113] S8. During navigation, the output of the environment perception model and the environment prediction model is monitored in real time. When it is predicted that the current path will have risks in the future, the preset backup path is switched in advance.

[0114] S9. Automatically generate navigation control instructions based on the environmental perception model, environmental prediction model and path planning results, and transmit the navigation control instructions to the ship's automatic driving system. In complex or high-risk environments, the system will issue warnings to the ship operator and provide path adjustment plans.

[0115] In this implementation, S3 specifically includes:

[0116] S31, using a convolutional neural network to extract features from the preprocessed image data;

[0117] S32, based on the image features extracted by the convolutional neural network, identify obstacles, ocean signs and other ships in navigation, and generate image feature vector I k ;

[0118] S33, perform amplitude detection and feature extraction on the preprocessed radar echo data to generate a radar feature vector R k , the strength of the radar signal is expressed as:

[0119]

[0120] Where θ represents the angle of the radar signal, r represents the distance of the target, and f c represents the radar carrier frequency, c represents the speed of light, R f (t) represents the radar signal after filtering;

[0121] S34, extract the image feature vector I k and radar feature vector R k Perform alignment in time and space;

[0122] S35, the aligned image feature vector I k and radar feature vector R k Input the extended Kalman filter to perform multi-sensor data fusion and estimation to generate an environmental perception model:

[0123]

[0124] Among them, x k+1 represents the state of the environment at time step k+1, x k represents the state vector at time step k, u k represents the input control of the system at time step k, h(x k ) represents the observed value predicted by the current state, f(x k,u k ) represents the state transfer function, K k Represents the Kalman gain matrix:

[0125]

[0126] Among them, P k represents the state covariance matrix, H k represents the Jacobian matrix of the observation model, R k represents the observation noise covariance matrix;

[0127] S36. Finally, an environmental perception model is generated.

[0128] In this implementation manner, the S4 specifically includes:

[0129] S41. Collect historical navigation data d , construct the time series form:

[0130] H d ={(p1,v1,θ1,w1,h1),(p2,v2,θ2,w2,h2),…,(p n ,v n ,θ n ,w n ,h n )};

[0131] Among them, n represents the time length of historical data, p i represents the position of the ship at time i, v i represents the speed of the ship at time i, θ i represents the heading vector of the ship at time i, w i represents the wind speed of the ship at time i, h i represents the wave height of the ship at time i;

[0132] S42. Obtaining environmental data E t , expressed as:

[0133] E t =(a t ,b t ,c t );

[0134] Among them, a t Indicates the current wind speed, b t Indicates the current wave height, c t Indicates the current ship flow density;

[0135] S43, the historical navigation data H d and environmental data E tCombined and input into the generative adversarial network model, the generator G receives the random noise vector z and the environment data E t As input, generate a future environment prediction sequence P f :

[0136] P f =G(z,E t )=σ(W G ·[z;E t ]+b G );

[0137] Among them, W G represents the weight matrix of the generator, b G represents the bias of the generator, σ represents the nonlinear activation function, [z; E t ] represents the random noise vector z and the environmental data E t The concatenation vector of

[0138] The discriminator D is used to evaluate the generated environment prediction sequence P f Compared with the real historical heading data H d differences;

[0139] S44, generate the environment prediction sequence P output by the adversarial network f Input into the space-time graph neural network for space-time dependency processing and construct the space-time graph G t (V,E), where V is a set of nodes, representing each time point and spatial position, and E is a set of edges, representing the association between time points and spatial positions.

[0140] S45. Perform temporal convolution on the features of each node to capture temporal dependencies:

[0141]

[0142] Among them, T represents the window size of temporal convolution, W τ represents the weight matrix of the temporal convolution kernel at time step τ, b τ represents the bias of temporal convolution, σ represents the activation function, represents the feature representation of node v at the t-τ time step, Represents the feature representation of node v at the t+1 time step;

[0143] S46. Perform spatial graph convolution on the features after temporal convolution to capture spatial dependencies:

[0144]

[0145] in, represents the feature representation of node v after the k+1th layer of graph convolution, N(v) represents the set of neighbor nodes of node v, and W k represents the spatial graph convolution weight matrix of the kth layer, p(u,v) represents the spatial position information between node u and node v, b k represents the spatial graph convolution bias of the kth layer;

[0146] S47. Generate a possible future environmental change scenario set S based on the output of the spatiotemporal graph neural network f ={S1,S2,…,S k}, each scene S i The possible changing trends of future predicted wind speed, wave height and ship flow density are included to build an environmental prediction model.

[0147] In this implementation manner, S5 specifically includes:

[0148] S51, obtaining initial data of current environmental status and future environmental changes from the environmental perception model and the environmental prediction model;

[0149] S52. Define the initial state vector x0 of the ship:

[0150] x0=(p0,v0,θ0,w0,h0);

[0151] Among them, p0 represents the initial position of the ship, v0 represents the initial speed of the ship, θ0 represents the heading vector of the ship, w0 represents the initial wind speed of the ship, and h0 represents the initial wave height of the ship;

[0152] S53, input the environmental perception data, environmental prediction data and initial state vector x0 into the heuristic search algorithm to perform initial path planning. The heuristic search algorithm uses an evaluation function f(x) to determine the priority of each state x. The evaluation function is composed of a path cost function g(x) and a heuristic estimation function h(x), which is expressed as:

[0153] f(x)=g(x)+h(x);

[0154] Among them, h(x) represents the heuristic estimate from the current state x to the target state g, expressed as Euclidean distance:

[0155]

[0156] Among them, x1, y1 represent the position coordinates of the current state x, and x2, y2 represent the position coordinates of the target state g;

[0157] S54, the path cost function g(x) is used to evaluate the cumulative path cost from the initial state x0 to the current state x:

[0158]

[0159] Among them, d(x i ,x i+1 ) indicates that the ship is in state x i To state x i+1 The distance between Represents state x i To state x i+1 The speed change rate between i+1 ) indicates that at time t i+1 The wind speed at the time, h(t i+1 ) indicates that at time t i+1 The wave height at the time, d(t i+1 ) indicates that at time t i+1 The ship flow density at the time, α, β, γ and δ represent weight coefficients, and k represents the total number of states in the cumulative path from the initial state x0 to the current state x;

[0160] S55, during the path planning process, dynamically adjusting the path cost function g(x) according to future environmental changes provided by the environmental prediction model;

[0161] S56, generate the initial path P of the ship through iterative calculation of the heuristic search algorithm init , starting from the current state x0 and eventually reaching the target state g, each state x in the path satisfies the optimal condition of the evaluation function f(x).

[0162] In this implementation manner, S6 specifically includes:

[0163] S61. Define the state space X of the target area, including all possible ship positions and headings, expressed as:

[0164] X={(x,y,θ,v)∣x,y∈R,θ∈[0,2π),v∈[v min ,v max ]};

[0165] Among them, x and y represent the coordinates of the ship's position, θ represents the heading angle, v represents the speed, and v min Indicates the minimum value of the ship's speed, v max Indicates the maximum value of the ship's speed;

[0166] S62. Randomly generate a new sample node x in the state space X rand =(x rand ,y rand ,θ rand ,v rand ), and find the path tree T that matches x rand The nearest node x near =(xnear ,y near ,θ near ,v near ):

[0167]

[0168] Among them, λ1 and λ2 represent weight coefficients;

[0169] S63, by passing the sample node x rand Connect to the nearest node x near , generate a new node x new , the generation process takes into account the dynamic constraints of the ship and uses a nonlinear motion model:

[0170] x new =x near +Δt·f(x near ,u);

[0171] Among them, f(x near ,u) represents the motion equation of the ship, which is expressed as:

[0172]

[0173] Among them, v c represents the current speed, θ c Indicates the current heading angle, L c Indicates the length of the ship, represents the steering angle, a c represents acceleration, u represents control input;

[0174] S64. Check the generated new node x new Is it feasible under environmental constraints? Environmental constraints include the avoidance of dynamic obstacles, the stability of ship operation, and the constraints of future wind speed, wave height and ship flow density provided by the environmental prediction model. The environmental cost function C is introduced. env (x new ) for evaluation:

[0175]

[0176] Among them, d obs represents the distance between the new node and the nearest obstacle, represents the speed change rate, w(t) represents the wind speed at the current time, h(t) represents the wave height at the current time, d(t) represents the ship flow density at the current time, α new , β new , γ new , δ new and new represents the weight coefficient;

[0177] If C env (x new ) exceeds the predetermined threshold, the node is discarded and the sample point is regenerated; otherwise, x new Add to the path tree T;

[0178] S65, repeat steps S62 to S64 until a node x in the path tree T is goal Close enough to the target state g, the following termination conditions are met:

[0179] ∥x goal -g∥≤∈ and C env (x goal )≤η;

[0180] Among them, ∈ represents the approach threshold, and η represents the environmental cost threshold;

[0181] S66, backtrack from the path tree T to generate the optimal path P from the starting state x0 to the target state g opt .

[0182] In this implementation manner, the S7 specifically includes:

[0183] S71. Define a real-time monitoring window W(t):

[0184] W(t)={x(t-Δt),x(t),x(t+Δt)};

[0185] Among them, Δt represents the width of the time window, and x(t) represents the state of the ship at time t;

[0186] S72. In each monitoring window W(t), a real-time risk assessment value of environmental changes is calculated based on the outputs of the environmental perception model and the environmental prediction model. The risk assessment value is calculated based on the potential impact of environmental changes on the ship path.

[0187] S73, comparing the real-time risk assessment value with a threshold value. If the risk assessment value exceeds the threshold value, it is determined that there is a potential risk in the current path and it is necessary to switch to an alternate path in advance.

[0188] S74. Use the selected backup path as the current new navigation path, notify the automatic driving system to switch the path, and update the real-time monitoring window W(t) to continue monitoring the new path.

[0189] Embodiment 1:

[0190] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a transatlantic cargo transportation mission. In a transatlantic cargo transportation mission, an autonomous cargo ship equipped with the latest intelligent navigation system departs from the Port of Rotterdam in the Netherlands and plans to go to the Port of New York in the United States. During the voyage, the cargo ship will cross multiple sea areas with complex and changeable weather, especially in the central North Atlantic, where it often faces strong winds, huge waves and high-density ship traffic. This environment poses a severe challenge to traditional navigation systems, because traditional systems are usually difficult to cope with complex and changeable environments, especially at night or in low visibility conditions, and are prone to navigation errors or delays, increasing navigation risks.

[0191] During this voyage, the ship applied the ship dynamic path adjustment method combining visual navigation and radar data proposed in this invention for the first time. The system combines visual navigation data with radar echo data through multi-sensor fusion technology to generate an accurate environment perception model. At the same time, an environment prediction model was constructed through generative adversarial networks and spatiotemporal graph neural networks, which can predict future sea state changes in advance and provide reliable data support for path planning.

[0192] When departing from the Port of Rotterdam, the ship first obtains real-time image data through the shipboard camera and uses the radar system to obtain radar echo data of the surrounding environment. In addition, through communication with the marine meteorological station, the system obtains environmental data such as current wind speed, wave height and surrounding ship traffic in real time. After preprocessing and multi-sensor fusion, these data generate an environmental perception model that accurately depicts the environmental conditions around the ship.

[0193] After leaving the port, the ship sailed along the planned route, but when entering the central North Atlantic region, the system predicted based on the generative adversarial network model and the spatiotemporal graph neural network that the region would encounter a Category 10 storm in the next six hours, accompanied by waves over 5 meters high. Traditional navigation systems can usually only make temporary adjustments when a storm occurs, but the system of the present invention can identify this risk in advance and optimize the path in advance. The system replanned the initial path through a heuristic search algorithm, combined with an environmental prediction model and an environmental perception model, and applied a rapidly expanding random tree algorithm to refine the path, ensuring the safe navigation of the ship in complex sea conditions.

[0194] During the path planning process, the system not only took into account the changes in wind speed and wave height, but also processed future ship traffic data through the spatiotemporal graph neural network to avoid possible collision risks. In the end, the system generated an optimal path to avoid the center of the storm and stored several backup paths in advance. Thanks to the predictive adjustment, the ship successfully avoided the frontal attack of the storm and returned to the optimal route after the storm weakened, avoiding potential dangers.

[0195] During the voyage, the system continuously monitors the safety of the current path through the real-time monitoring window, and continuously evaluates the impact of environmental changes on the ship based on the output of the environmental perception model and the environmental prediction model. For example, when the ship approaches a high-density ship area in the central North Atlantic, the system detects that the current path may have a collision risk, so it immediately switches to the previously stored backup path and successfully avoids collisions with other ships. This dynamic adjustment based on real-time monitoring and backup paths effectively improves the safety and efficiency of navigation.

[0196] The system's performance was fully demonstrated throughout the voyage. The voyage time was shortened from the scheduled 15 days to 14 days, and no dangerous situations occurred during the voyage despite the severe sea conditions and complex ship traffic. In addition, fuel consumption was reduced by about 5%, thanks to the system's route optimization during the storm, avoiding ineffective navigation and excessive fuel consumption.

[0197] Table 1 Comparison of key indicators between traditional navigation method and ship dynamic path adjustment method of the present invention

[0198]

[0199] Table 1 shows that in the comparison between the traditional navigation method and the ship dynamic path adjustment method of the present invention, significant differences are reflected in multiple key indicators, showing the superiority of the present invention.

[0200] First, in terms of sailing time, the traditional navigation method usually takes 15 days to complete the voyage. Through the optimized path planning of the method of the present invention, it only takes 14 days, shortening the sailing time by 1 day, reducing the total sailing time by about 7%. This improvement not only improves the overall sailing efficiency, but also brings economic benefits to ship operations.

[0201] In terms of fuel consumption, the traditional method has a large fuel consumption of 100% due to untimely path adjustment and insufficient optimization of route planning. In contrast, the method of the present invention reduces fuel consumption to 95% through precise path optimization and early response to environmental changes, saving 5% of fuel. This fuel saving not only reduces operating costs, but also reduces carbon emissions, which has positive significance for environmental protection.

[0202] In terms of storm identification and path adjustment response time, traditional methods often respond temporarily when a storm occurs, lacking early warning capabilities. However, the present invention, through generative adversarial networks and spatiotemporal graph neural networks, can identify storms 8 hours in advance and start path adjustment 6 hours before the storm occurs. This early identification and adjustment capability greatly reduces navigation risks and ensures that ships can safely avoid storms.

[0203] The detection and handling of high-density ship areas is another significant improvement. Traditional methods usually rely on radar monitoring, which can only detect other ships when approaching, posing a risk of collision. The method of the present invention detects the ship ahead 3,000 meters in advance and quickly adjusts the path, increasing the safety distance and avoiding potential collision risks. This shows that the present invention has higher safety when dealing with complex navigation environments.

[0204] In terms of the number of path adjustments, the traditional method needs to adjust the path frequently because it does not respond to environmental changes in a timely manner. However, the present invention reduces the number of main path adjustments to 3 times and fine-tuning to 6 times through the pre-judgment and backup path mechanism. With this pre-judgment capability, the path adjustment of the present invention in complex sea conditions is more accurate and efficient.

[0205] In terms of backup path usage, traditional navigation methods usually rely on a single path and lack a backup path mechanism, resulting in a slow response time when dealing with emergencies. The method of the present invention greatly enhances fault tolerance and improves navigation safety by storing three backup paths and actually using one.

[0206] In terms of safety incidents, traditional methods are prone to risks such as collisions in complex environments. However, the present invention, through early detection and path adjustment, no safety incidents occurred during the entire navigation process, thus significantly improving safety.

[0207] In terms of navigation efficiency, the traditional method is less efficient when greatly affected by the environment, while the present invention improves the overall navigation efficiency by dynamically adjusting the path to avoid adverse environments. The enhancement of the system's real-time monitoring function makes risk detection more frequent and in advance, with an assessment every 30 minutes, which greatly improves the sensitivity and response capabilities to environmental changes.

[0208] Finally, the introduction of the environmental prediction model makes the present invention significantly more forward-looking than traditional methods, and can predict environmental changes 6 hours in advance, enhancing the ability to respond to factors such as wind speed, waves, and ship flow density. This enhanced environmental prediction capability, combined with real-time path adjustment, enables the present invention to perform well in avoiding collision risks and improving navigation safety and efficiency.

[0209] In summary, the above comparative analysis shows that the ship dynamic path adjustment method of the present invention performs superiorly in terms of sailing time, fuel consumption, safety, path planning efficiency and environmental response capability, demonstrating obvious technological progress and practical application value.

[0210] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for dynamic ship path adjustment combining visual navigation and radar data, characterized in that: The steps include: S1. Use ship cameras to obtain real-time image data, use radar systems to obtain radar echo data of the surrounding environment, and collect environmental data through marine meteorological stations; S2, preprocessing the collected image data, radar echo data and environmental data, including target detection, image segmentation and edge detection for image data, signal filtering, enhancement and feature extraction for radar echo data, and time series analysis for environmental data; S3, performing multi-sensor data fusion on the pre-processed image data and radar echo data through the extended Kalman filter algorithm to generate an environment perception model; S4, based on the environment perception model, historical navigation data and environmental data, the environment prediction model is constructed by combining the generative adversarial network and the spatiotemporal graph neural network; S5. Based on the environment perception model and the environment prediction model, use the heuristic search algorithm to perform initial path planning and generate an initial path; S6. Based on the initial path, the fast expanding random tree algorithm is used to refine and optimize the path to generate the optimal path; S7, generating possible future environmental changes according to the environmental prediction model, and calculating and storing backup paths for each predicted scenario in combination with the environmental perception model; S8. During navigation, the output of the environment perception model and the environment prediction model is monitored in real time. When it is predicted that the current path will have risks in the future, the preset backup path is switched in advance. S9. Automatically generate navigation control instructions based on the environmental perception model, environmental prediction model and path planning results, and transmit the navigation control instructions to the ship's automatic driving system. In complex or high-risk environments, the system will issue warnings to the ship operator and provide path adjustment plans.

2. A method for dynamic ship path adjustment combining visual navigation and radar data according to claim 1, characterized in that: The S3 specifically includes: S31, using a convolutional neural network to extract features from the preprocessed image data; S32, based on the image features extracted by the convolutional neural network, identify obstacles, ocean signs and other ships in navigation, and generate image feature vector I k ; S33, perform amplitude detection and feature extraction on the preprocessed radar echo data to generate a radar feature vector R k , the strength of the radar signal is expressed as: Where θ represents the angle of the radar signal, r represents the distance of the target, and f c represents the radar carrier frequency, c represents the speed of light, R f (t) represents the radar signal after filtering; S34, extract the image feature vector I k and radar feature vector R k Perform alignment in time and space; S35, the aligned image feature vector I k and radar feature vector R k Input the extended Kalman filter to perform multi-sensor data fusion and estimation to generate an environmental perception model: Among them, x k+1 represents the state of the environment at time step k+1, x k represents the state vector at time step k, u k represents the input control of the system at time step k, h(x k ) represents the observed value predicted by the current state, f(x k ,u k ) represents the state transfer function, K k Represents the Kalman gain matrix: Among them, P k represents the state covariance matrix, H k represents the Jacobian matrix of the observation model, R k represents the observation noise covariance matrix; S36. Finally, an environmental perception model is generated.

3. A method for dynamic ship path adjustment combining visual navigation and radar data according to claim 1, characterized in that: The S4 specifically includes: S41. Collect historical navigation data d , construct the time series form: H d ={(p1,v1,θ1,w1,h1),(p2,v2,θ2,w2,h2),…,(p n ,v n ,θ n ,w n ,h n )}; Among them, n represents the time length of historical data, p i represents the position of the ship at time i, v i represents the speed of the ship at time i, θ i represents the heading vector of the ship at time i, w i represents the wind speed of the ship at time i, h i represents the wave height of the ship at time i; S42. Obtaining environmental data E t , expressed as: E t =(a t ,b t ,c t ); Among them, a t Indicates the current wind speed, b t Indicates the current wave height, c t Indicates the current ship flow density; S43, the historical navigation data H d and environmental data E t Combined and input into the generative adversarial network model, the generator G receives the random noise vector z and the environment data E t As input, generate a future environment prediction sequence P f : P f =G(z,E t )=σ(W G ·[That t ]+b G ); Among them, W G represents the weight matrix of the generator, b G represents the bias of the generator, σ represents the nonlinear activation function, [z; E t ] represents the random noise vector z and the environmental data E t The concatenation vector of The discriminator D is used to evaluate the generated environment prediction sequence P f Compared with the real historical heading data H d differences; S44, generate the environment prediction sequence P output by the adversarial network f Input into the space-time graph neural network for space-time dependency processing and construct the space-time graph G t (V,E), where V is a set of nodes, representing each time point and spatial position, and E is a set of edges, representing the association between time points and spatial positions. S45. Perform temporal convolution on the features of each node: Among them, T represents the window size of temporal convolution, W τ represents the weight matrix of the temporal convolution kernel at time step τ, b τ represents the bias of temporal convolution, σ represents the activation function, represents the feature representation of node v at the t-τ time step, Represents the feature representation of node v at the t+1 time step; S46. Perform spatial graph convolution on the features after temporal convolution: in, represents the feature representation of node v after the k+1th layer of graph convolution, N(v) represents the set of neighbor nodes of node v, and W k represents the spatial graph convolution weight matrix of the kth layer, p(u,v) represents the spatial position information between node u and node v, b k represents the spatial graph convolution bias of the kth layer; S47. Generate a possible future environmental change scenario set S based on the output of the spatiotemporal graph neural network f ={S1,S2,…,S k }, each scene S i The possible changing trends of future predicted wind speed, wave height and ship flow density are included to build an environmental prediction model.

4. A method for dynamic ship path adjustment combining visual navigation and radar data according to claim 1, characterized in that: The S5 specifically includes: S51, obtaining initial data of current environmental status and future environmental changes from the environmental perception model and the environmental prediction model; S52. Define the initial state vector x0 of the ship: x0=(p0,v0,θ0,w0,h0); Among them, p0 represents the initial position of the ship, v0 represents the initial speed of the ship, θ0 represents the heading vector of the ship, w0 represents the initial wind speed of the ship, and h0 represents the initial wave height of the ship; S53, input the environmental perception data, environmental prediction data and initial state vector x0 into the heuristic search algorithm to perform initial path planning. The heuristic search algorithm uses an evaluation function f(x) to determine the priority of each state x. The evaluation function is composed of a path cost function g(x) and a heuristic estimation function h(x), which is expressed as: f(x)=g(x)+h(x); Among them, h(x) represents the heuristic estimate from the current state x to the target state g, expressed as Euclidean distance: Among them, x1, y1 represent the position coordinates of the current state x, and x2, y2 represent the position coordinates of the target state g; S54, the path cost function g(x) is used to evaluate the cumulative path cost from the initial state x0 to the current state x: Among them, d(x i ,x i+1 ) indicates that the ship is in state x i To state x i+1 The distance between Represents state x i To state x i+1 The speed change rate between i+1 ) indicates that at time t i+1 The wind speed at the time, h(t i+1 ) indicates that at time t i+1 The wave height at the time, d(t i+1 ) indicates that at time t i+1 The ship flow density at the time, α, β, γ and δ represent weight coefficients, and k represents the total number of states in the cumulative path from the initial state x0 to the current state x; S55, during the path planning process, dynamically adjusting the path cost function g(x) according to future environmental changes provided by the environmental prediction model; S56, generate the initial path P of the ship through iterative calculation of the heuristic search algorithm init , starting from the current state x0 and eventually reaching the target state g, each state x in the path satisfies the optimal condition of the evaluation function f(x).

5. The method for dynamic ship path adjustment combining visual navigation and radar data according to claim 1, characterized in that: The S6 specifically includes: S61. Define the state space X of the target area, including all possible ship positions and headings, expressed as: X={(x,y,θ,v)∣x,y∈R,θ∈[0,2π),v∈[v min ,v max ]}; Among them, x and y represent the coordinates of the ship's position, θ represents the heading angle, v represents the speed, and v min Indicates the minimum value of the ship's speed, v max Indicates the maximum value of the ship's speed; S62. Randomly generate a new sample node x in the state space X rand =(x rand ,y rand ,θ rand ,v rand ), and find the path tree T that matches x rand The nearest node x near =(x near ,y near ,θ near ,v near ): Among them, λ1 and λ2 represent weight coefficients; S63, by passing the sample node x rand Connect to the nearest node x near , generate a new node x new , the generation process takes into account the dynamic constraints of the ship and uses a nonlinear motion model: x new =x near +Δt·f(x near ,u); Among them, f(x near ,u) represents the motion equation of the ship, which is expressed as: Among them, v c represents the current speed, θ c Indicates the current heading angle, L c Indicates the length of the ship, represents the steering angle, a c represents acceleration, u represents control input; S64. Check the generated new node x new Is it feasible under environmental constraints? Environmental constraints include the avoidance of dynamic obstacles, the stability of ship operation, and the constraints of future wind speed, wave height and ship flow density provided by the environmental prediction model. The environmental cost function C is introduced. env (x new ) for evaluation: Among them, d obs represents the distance between the new node and the nearest obstacle, represents the speed change rate, w(t) represents the wind speed at the current time, h(t) represents the wave height at the current time, d(t) represents the ship flow density at the current time, α new , β new , γ new , δ new and new represents the weight coefficient; If C env (x new ) exceeds the predetermined threshold, the node is discarded and the sample point is regenerated; otherwise, x new Add to the path tree T; S65, repeat steps S62 to S64 until a node x in the path tree T is goal It is close enough to the target state g and satisfies the following termination conditions: ∥x goal -g∥≤∈ and C env (x goal )≤η; Among them, ∈ represents the approach threshold, and η represents the environmental cost threshold; S66, backtrack from the path tree T to generate the optimal path P from the starting state x0 to the target state g opt .

6. A method for dynamic ship path adjustment combining visual navigation and radar data according to claim 1, characterized in that: The S7 specifically includes: S71. Define a real-time monitoring window W(t): W(t)={x(t-Δt),x(t),x(t+Δt)}; Among them, Δt represents the width of the time window, and x(t) represents the state of the ship at time t; S72. In each monitoring window W(t), a real-time risk assessment value of environmental changes is calculated based on the outputs of the environmental perception model and the environmental prediction model. The risk assessment value is calculated based on the potential impact of environmental changes on the ship path. S73, comparing the real-time risk assessment value with a threshold value. If the risk assessment value exceeds the threshold value, it is determined that there is a potential risk in the current path and it is necessary to switch to an alternate path in advance. S74. Use the selected backup path as the current new navigation path, notify the automatic driving system to switch the path, and update the real-time monitoring window W(t) to continue monitoring the new path.

Images