A real-time ship navigation route planning method based on augmented reality

Abstract

The present invention proposes a real-time ship navigation route planning method based on augmented reality, including: collecting various environmental data around the ship in real time and preprocessing the various environmental data; processing video data through a convolutional neural network to extract environmental features, extracting dynamic features through a long short-term memory network, extracting its spatial features through a convolutional neural network, using a two-layer multi-layer perceptron to extract meteorological features, and fusing to obtain fused features; using a generative adversarial network to generate various sea conditions and meteorological scenarios, conducting risk assessment and generating a risk map, generating an avoidance strategy, and adjusting the route to avoid high-risk areas; combining real-time environmental perception data and historical navigation data, and dynamically adjusting the navigation route by using a reinforcement learning algorithm to obtain an optimized navigation route; through the real-time synchronization of the AR display and the navigation system, and combining with the augmented reality technology for superimposed display. The solution of the present invention improves the safety and navigation efficiency of the ship.

Description

Technical Field

[0001] The present invention belongs to the field of ship navigation route planning, and particularly relates to a real-time ship navigation route planning method based on augmented reality. Background Art

[0002] With the rapid development of the global shipping industry, the importance of ship navigation safety and efficiency has become increasingly prominent. Traditional ship navigation systems mainly rely on means such as manual observation, electronic nautical charts, and the Automatic Identification System (AIS). However, these methods have obvious limitations. First, manual observation is easily affected by weather, visibility, and human factors, and it is difficult to provide real-time and accurate environmental perception information. Second, although electronic nautical charts can provide basic information on navigation routes, they lack the support of real-time dynamic data and cannot cope with sudden environmental changes and risks. In addition, although the AIS system can provide the position and movement information of other ships, in high-density navigation areas and complex sea conditions, relying solely on AIS information is difficult to ensure navigation safety.

[0003] Some existing advanced navigation systems attempt to introduce radar and video surveillance technologies to improve the accuracy of environmental perception. However, these systems usually operate independently, lacking effective fusion and comprehensive analysis of multi-modal data, resulting in the problem of information silos and making it difficult to provide comprehensive environmental perception and risk assessment. In addition, although some systems introduce artificial intelligence technologies such as machine learning and deep learning for route optimization and risk prediction, their application scenarios are limited and they lack the ability to dynamically adjust to real-time environmental data. Especially in ocean voyages and complex sea conditions, these systems often cannot respond and adjust in real time, affecting navigation safety and efficiency. Summary of the Invention

[0004] The purpose of the present invention is to propose a real-time ship navigation route planning method based on augmented reality. By fusing on-board video, AIS information, radar data, and meteorological data, and applying deep learning and Generative Adversarial Network (GAN) technologies, it realizes real-time optimization and risk assessment of ship routes. The augmented reality technology then intuitively displays the optimized route and risk prompts in the crew's field of vision, greatly enhancing the intuitiveness and accuracy of navigation decisions. This method not only overcomes the information silo problem of traditional navigation systems but also makes up for the deficiencies of existing intelligent navigation systems in real-time dynamic adjustment and comprehensive risk assessment, providing a new solution for ship navigation.

[0005] To achieve the above object, the present invention provides a real-time ship navigation route planning method based on augmented reality, and the method includes:

[0006] S1. Collect various environmental data around the ship in real time and preprocess the various environmental data; the various environmental data includes video data, AIS data, radar data, and meteorological data;

[0007] S2. Process the video data through a convolutional neural network to extract environmental features, process the AIS data through a long short-term memory network to extract dynamic features, process the radar data through a convolutional neural network to extract its spatial features, use a two-layer multi-layer perceptron to process the meteorological data to extract meteorological features, use an attention mechanism to align the features of different modalities and fuse them to obtain fused features, and then use a bidirectional long short-term memory network to perform temporal modeling on the fused features to generate the final environmental perception result;

[0008] S3. According to the final environmental perception result, use a generative adversarial network to generate various sea conditions and meteorological scenarios, conduct risk assessment and generate a risk map, and according to the results of the risk assessment and the risk map, generate an avoidance strategy to adjust the route to avoid high-risk areas;

[0009] S4. According to the results of the risk assessment, combined with real-time environmental perception data and historical navigation data, use a reinforcement learning algorithm to dynamically adjust the navigation route to obtain an optimized navigation route;

[0010] S5. Synchronize the optimized navigation route, high-risk areas, and important reminder information in real time with the navigation system through AR display, and combine with augmented reality technology for overlay display;

[0011] Among them, according to the results of the risk assessment and the risk map, generate an avoidance strategy to adjust the route to avoid high-risk areas, specifically including:

[0012] Introduce a path smoothness term α smooth and a path adjustment cost term β cost , generate an avoidance strategy, expressed as follows:

[0013]

[0014] Among them, P new represents the new avoidance strategy, R total (P) represents the risk score of the old avoidance strategy, Smoothness(P) represents the path smoothness metric, and AdjustmentCost represents the cost of adjusting from the current path to the new path;

[0015] Among them, the path smoothness metric is expressed as follows:

[0016]

[0017] Among them, θ jDenote the heading angle of the j-th waypoint on path P, and m represents the total number of waypoints;

[0018] The cost of adjusting from the current path to the new path is expressed as follows:

[0019] AdjustmentCost(P) = δ fuel ·FuelCost(P) + δ time ·TimeDelay(P)

[0020] where FuelCost(P) represents the additional fuel consumption caused by adjusting the path, TimeDelay(P) represents the time delay caused by path adjustment, and δ fuel and δ time represent the corresponding weight coefficients.

[0021] Preferably, the preprocessing includes multi-modal data time synchronization processing, video data preprocessing, AIS data preprocessing, radar data preprocessing, and meteorological data preprocessing; among them, the video data preprocessing includes denoising processing and frame extraction; the AIS data preprocessing includes duplicate removal, filtering processing, and data interpolation processing; the radar data preprocessing includes echo denoising and data gridding processing; the meteorological data preprocessing includes data smoothing processing and data standardization processing.

[0022] Preferably, the video data is processed by a convolutional neural network to extract environmental features, which is expressed as follows:

[0023] F conv (t) = ReLU(Conv(I(t), W) + b)

[0024] where I(t) represents the input video frame, W represents the convolutional kernel weight, b represents the bias, Conv represents the convolution operation, ReLU represents the activation function, and F conv (t) represents the output feature of the current convolutional layer;

[0025] The pooling calculation formula of the convolutional neural network is expressed as follows:

[0026] F pool (t) = MaxPool(F conv (t))

[0027] where F pool (t) is the pooled feature map, and MaxPool represents the max pooling operation;

[0028] The AIS data is processed by a long short-term memory network to extract dynamic features, which is expressed as follows:

[0029] i t = σ(W i·[h t-1 , A(t)] + b i )

[0030] f t = σ(W f ·[h t-1 , A(t)] + b f )

[0031] o t = σ(W o ·[h t-1 , A(t)] + b o )

[0032]

[0033] h t = o t * tanh(C t )

[0034] where A(t) represents the input AIS data, C t represents the cell state, h t represents the hidden state, b i , b f , b o , b C represents the bias vector, σ represents the sigmoid activation function, tanh represents the tanh activation function, h t-1 represents the hidden state at t - 1, C t-1 represents the cell state at t - 1, represents the candidate cell state at the current time step, i t represents the input gate activation value at time t, W i , W f , W o , W C represents the weight matrix, o t represents the output gate activation value at time t, f t represents the forget gate activation value at time t;

[0035] The processing of radar data by the convolutional neural network to extract its spatial features is represented as follows:

[0036] F conv_rad (t) = ReLU(Conv(R(t), W rad ) + b rad )

[0037] where R(t) represents the input radar data, W rad represents the convolutional kernel weight, b rad represents the bias, ReLU represents the activation function, Fconv_rad (t) represents the output of the current convolutional neural network;

[0038] The pooling calculation formula is:

[0039] F pool_rad (t) = MaxPool(F conv_rad (t))

[0040] where F pool_rad (t) represents the feature map after pooling, and MaxPool represents the max pooling operation;

[0041] Using two-layer multi-layer perceptrons to process meteorological data and extract meteorological features is expressed as follows:

[0042] The first fully connected layer:

[0043] F fc1 (t) = ReLU(W fc1 ·W(t) + b fc1 )

[0044] where W(t) represents the input meteorological data, W fc1 represents the weight, b fc1 represents the bias, and F fc1 (t) represents the output of the first fully connected layer;

[0045] The second fully connected layer:

[0046] F met (t) = ReLU(W fc2 ·F fc1 (t) + b fc2 ))

[0047] where F fc1 (t) represents the output features of the first layer, W fc2 represents the weight of the second layer, b fc2 represents the bias, and F met (t) represents the output of the second fully connected layer.

[0048] Preferably, using the attention mechanism to align the features of different modalities and fuse them to obtain the fused features, and then using the bidirectional long short-term memory network to perform temporal modeling on the fused features to generate the final environmental perception result, which is expressed as follows:

[0049] Let the attention weight be α, and the aligned feature be F aligned (t), then the calculation of the attention weight:

[0050] α vid = softmax(W a1 ·F vid (t))

[0051] Among them, α vid represents the video attention weight, W a1 is the attention weight matrix of video features, and softmax is the softmax function;

[0052] α AIS = softmax(W a2 ·F AIS (t))

[0053] Among them, α AIS represents the AIS attention weight, W a2 is the attention weight matrix of AIS features;

[0054] α rad = softmax(W a3 ·F rad (t))

[0055] Among them, α rad represents the radar attention weight, W a3 is the attention weight matrix of radar features;

[0056] α met = softmax(W a4 ·F met (t)))

[0057] Among them, α met represents the meteorological attention weight, W a4 is the attention weight matrix of meteorological features;

[0058] Feature alignment and fusion:

[0059] F aligned (t)= α vid ·F vid (t)+ α AIS ·F AIS (t)+ α rad ·F rad (t)+ α met ·F met (t)

[0060] Among them, F vid (t) represents the video feature map, F AIS (t) represents the AIS feature, F rad (t) represents the radar feature map, F met (t) represents the meteorological feature, F aligned (t) is the aligned multi-modal feature;

[0061] Use a bidirectional long short-term memory network to perform temporal modeling on the fused features, extract temporal dynamic information, and generate the final environmental perception result E(t), which is expressed as follows:

[0062] E(t) = BiLSTM(F aligned (t))

[0063] Among them, BiLSTM represents the bidirectional long short-term memory network operation, and E(t) is the environmental perception result at time t.

[0064] Preferably, the structure of the generative adversarial network is constructed as follows:

[0065] The input of the generator G is the noise vector z and the multi-modal fused feature F aligned (t), and the output is the generated simulated scenario G(z, F aligned (t)), where the network structure of the generator G is a multi-layer perceptron network;

[0066] The input of the discriminator D is the real scenario X real and the generated scenario G(z, F aligned (t)), and the output is the judgment result D(X real , G(z, F aligned (t))), where the network structure of the discriminator D is: a convolutional neural network;

[0067] Use an alternating training method to train the generator G and the discriminator D respectively; among them, the loss function of the generator G is expressed as follows:

[0068]

[0069] Among them, p z (z) represents the noise distribution, p data (F aligned ) represents the distribution of the multi-modal fused feature, z represents the noise vector, and E represents the expectation;

[0070] The loss function of the discriminator D is expressed as follows

[0071]

[0072] Among them, p data (X real ) represents the distribution of the real scenario;

[0073] According to different input noise vectors z and multi-modal fused features F aligned (t), generate multiple simulated scenarios G(z i , F aligned (t)).

[0074] Preferably, a dynamic risk factor λ based on historical data is introduced dynamic and a non-linear term γ based on environmental complexity env For each generated simulation scenario G(z i ,F aligned (t)), a risk assessment is performed, expressed as follows:

[0075] R i = M risk (G(z i ,F aligned (t)))+λ dynamic ·HistoryFactor(G(z i ,F aligned (t)))+γ env ·EnvComplexity(G(z i ,F aligned (t)))

[0076] Wherein, M risk represents the basic risk assessment model; λ dynamic represents the historical dynamic risk factor; γ env represents the non-linear term of environmental complexity; HistoryFactor represents the risk pattern weight in historical data; wherein, the HistoryFactor reflects the historical risk under similar navigation conditions, and when calculating, it is necessary to find the historical scenario in the historical database that best matches the current scenario G(z i ,F aligned (t)), and weight according to the average risk value of these scenarios, expressed as follows:

[0077]

[0078] Wherein, k represents the number of selected historical scenarios, w j represents the weight of the jth historical scenario matching the current scenario, R hist (j) represents the risk score of historical scenario j;

[0079] The risk scores of all generated scenarios are comprehensively analyzed to obtain the comprehensive risk score R total under the overall navigation environment, expressed as follows:

[0080]

[0081] Wherein, ω i represents the dynamic weighting factor, and N represents the number of generated scenarios.

[0082] Preferably, the S4 specifically includes:

[0083] S401. Define the state st , representing the environmental state of the ship at time t; define the action a t , representing the navigation decision of the ship at time t;

[0084] S402. The reward function R(s t ,a t ) is used to evaluate the effect of each state-action pair. Among them, the reward function R(s t ,a t ) includes a safety reward R safety , an efficiency reward R efficiency , and a risk penalty R risk ;

[0085] S403. Using the policy gradient method, design a policy network π(a t |s t ; θ), where θ is the parameter of the policy network. The input of the policy network is the state s t , and the output is the probability distribution of the action a t ;

[0086] S404. Design a value network V(s t ; φ), where φ is the parameter of the value network, used to estimate the value of the state s t . Among them, the input of the value network is the state s t , and the output is the value of the state V(s t );

[0087] S405. Using the policy gradient method, optimize the parameter θ of the policy network π(a t |s t ; θ), and use the value network V(s t ; φ) to optimize the parameter φ of the policy network π(a t |s t ; θ);

[0088] S406. Using the optimized policy network π(a t |s t ; θ) for real-time path optimization, and according to the real-time environment perception result E(t) and the current state s t , sample the action a t from the policy network to adjust the ship's route.

[0089] Preferably, the reward function R(s t ,a t ) includes a safety reward R safety , an efficiency reward R efficiency , and a risk penalty R risk , and is expressed as follows:

[0090] Safety Reward R safety (s t ,a t ):

[0091]

[0092] where d i represents the distance to the i-th surrounding vessel, ∈ represents a small constant to prevent the denominator from being zero, and n represents the number of surrounding vessels;

[0093] Efficiency Reward R efficiency (s t ,a t ):

[0094]

[0095] where vel desired represents the desired speed, and vel t represents the actual speed;

[0096] Risk Penalty R risk (s t ,a t ):

[0097] R risk (s t ,a t ) = R total (s t )

[0098] where R total (s t ) represents the comprehensive risk score;

[0099] Total Reward Function R(s t ,a t ):

[0100] R(s t ,a t ) = λ 1 ·R safety (s t ,a t ) + λ 2 ·R efficiency (s t ,a t ) - λ 3 ·R risk (s t ,a t )

[0101] where λ 1 , λ 2 , λ 3 represents the weight coefficient.

[0102] Preferably, in the real-time synchronization of the AR display and the navigation system, a joint optimization formula is introduced, and combined with the environmental complexity and the comprehensive risk score, the visual effect of the path display is dynamically adjusted, as shown below:

[0103]

[0104] where λ display1 and λ display2 respectively represent the adjustment parameters of path smoothness and information transparency, and θ j represents the heading angle of the j-th waypoint on path P; Transparency(R i ) represents the transparency of the risk area R i , which is dynamically adjusted according to the risk level.

[0105] Preferably, for a specific navigation scenario, a risk accumulation term δ risk is designed to record the cumulative risk value that occurred in a similar environment in history, and help the crew anticipate potential dangerous areas in advance, as shown below:

[0106]

[0107] where γ env is the risk accumulation coefficient.

[0108] The beneficial technical effects of the present invention are at least as follows:

[0109] (1) Through the deep learning algorithm, the present invention integrates on-board video, AIS, radar, and meteorological data to provide a more comprehensive and accurate environmental perception. Compared with the prior art, the present invention can overcome the information island problem, realize the comprehensive analysis of different data sources, and provide real-time environmental perception information.

[0110] (2) The present invention uses GAN technology to generate various possible sea conditions and meteorological scenarios for comprehensive risk assessment. Existing systems rely mostly on static and single data sources for risk assessment, while the present invention realizes dynamic and diversified risk prediction through GAN technology, greatly improving the accuracy and reliability of risk assessment.

[0111] (3) The present invention adopts a reinforcement learning algorithm to dynamically adjust and optimize the route according to real-time environmental data and historical navigation data. Compared with the static path planning of traditional and existing systems, the present invention can respond to environmental changes in real time, provide a dynamic path optimization scheme, and ensure the efficiency and safety of navigation.

[0112] (4) Through augmented reality technology, the optimized shipping lanes, risk areas, and important prompt information are superimposed and displayed in the crew's field of vision, providing intuitive navigation and decision-making support. The information display of traditional navigation systems is relatively single and abstract, while the AR display of the present invention can improve the intuitiveness and operability of information, helping the crew better understand and respond to the navigation environment. Description of the Drawings

[0113] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the following drawings.

[0114] Figure 1 It is a flowchart of a real-time ship navigation route planning method based on augmented reality according to the present invention. Detailed Embodiments

[0115] The embodiments of the present invention are described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0116] As Figure 1 shown, a real-time ship navigation route planning method based on augmented reality provided by an embodiment of the present invention includes the following steps S1 - S5:

[0117] S1. Real-time collect various environmental data around the ship and preprocess the various environmental data; the various environmental data include video data, AIS data, radar data, and meteorological data.

[0118] Specifically, a variety of sensor devices are configured around the ship, including high-resolution video cameras, AIS devices, radars, and meteorological sensors. Ensure that the devices can work properly in various navigation environments. The video cameras are set at different key positions of the ship to cover a 360-degree field of vision around the ship and collect video data with a resolution of 1080p or higher. The AIS device is configured to receive and send information such as ship identification, position, speed, heading, and timestamp, and communicate with other ships and shore-based devices. The radar sets the radar antenna angle and sensitivity to ensure that surrounding ships, obstacles, and sea conditions can be detected, and the collection frequency is set to 10 times per second. The meteorological sensor is installed with temperature, humidity, wind speed, wind direction, and pressure sensors to collect environmental meteorological data in real time.

[0119] Furthermore, when collecting data, in order to ensure the temporal consistency of each data source, a global time synchronization mechanism is used. The high-precision GPS timestamp is adopted to synchronize the collection times of all devices, ensuring that the data can be aligned at the same time point.

[0120] Specifically, the time synchronization formula:

[0121] T s =T GPS +Δt (1)

[0122] Among them, T s is the synchronized timestamp, T GPS is the GPS timestamp, and Δt is the time delay compensation of the device. The specific time delay compensation amount can be determined through the calibration process of the device.

[0123] Specifically, the video data preprocessing includes the following:

[0124] Denoising processing: A custom multi-scale Gaussian filter is used to remove the noise in the video. The multi-scale filtering formula:

[0125]

[0126] Among them, I smooth (x,y) is the denoised video frame, is the Gaussian kernel with a standard deviation of σ i , w i is the weight, I(x,y) is the original video frame, and n is the number of scales.

[0127] Frame extraction: Key frames are extracted from the video stream at fixed intervals to reduce the load of data processing. The key frame extraction formula:

[0128] F key (t)=I(t) if t mod N=0 (3)

[0129] Among them, F key (t) is the key frame, I(t) is the video frame at time t, and N is the extraction interval.

[0130] The AIS data preprocessing includes the following:

[0131] Duplicate removal and filtering: The AIS data is processed for duplicate removal to ensure that only the latest data is retained at each time point, and abnormal and error data is filtered out. The duplicate removal and filtering formula:

[0132]

[0133] Among them, A filtered (t) is the filtered AIS data, a i(t) is the i-th AIS data point at time t, and ε is the abnormal data set.

[0134] Data interpolation: Use the Bezier curve interpolation method based on the navigation track to fill in the missing time points of the data and ensure data continuity. Bezier curve interpolation formula:

[0135] P(t) = (1 - t) 2 P 0 + 2(1 - t)tP 1 + t 2 P 2 (5)

[0136] Where P(t) is the interpolated AIS position data, P 0 is the previous position point, P 1 is the current position point, P 2 is the next position point, and t is the interpolation parameter.

[0137] Radar data preprocessing includes the following:

[0138] Echo denoising: Use an adaptive median filter to remove noise from the radar data. Adaptive median filter formula:

[0139] R filtered (x, y) = median{R(x + i, y + j)} (6)

[0140] Where R filtered (x, y) is the denoised radar data, R(x, y) is the original radar data, and (i, j) is the filter window size, which is adaptively adjusted according to the noise level.

[0141] Data gridding: Map the radar data to a fixed polar coordinate grid for convenient subsequent processing. Gridding formula:

[0142] R grid (ρ, θ) = R(x, y) (7)

[0143] Where R grid (ρ, θ) is the gridded radar data, ρ is the distance, θ is the angle, and (x, y) is the radar coordinate.

[0144] Meteorological data preprocessing includes the following:

[0145] Data smoothing: Use the double exponential smoothing method to smooth the meteorological data. Double exponential smoothing formula:

[0146] W smooth (t) = αW(t) + (1 - α)W smooth (t - 1) + β(W(t) - W(t - 1)) (8)

[0147] Among them, W smooth (t) is the smoothed meteorological data, W(t) is the original meteorological data, and α and β are smoothing coefficients.

[0148] Data standardization: Standardize the meteorological data to ensure that the data is on the same scale. The standardization formula:

[0149]

[0150] Among them, W norm (t) is the standardized meteorological data, μ W and σ W are the mean and standard deviation of the meteorological data respectively.

[0151] S2. Process the video data through a convolutional neural network to extract environmental features, process the AIS data through a long short-term memory network to extract dynamic features, process the radar data through a convolutional neural network to extract its spatial features, use a two-layer multi-layer perceptron to process the meteorological data to extract meteorological features, use an attention mechanism to align and fuse the features of different modalities to obtain fused features, and then use a bidirectional long short-term memory network to perform temporal modeling on the fused features to generate the final environmental perception result.

[0152] Specifically, use a convolutional neural network (CNN) to extract the spatial features of video frames. Assume that the input video frame is I(t), and after three layers of convolution and pooling operations, the feature map F vid (t) is obtained.

[0153] Among them, the parameters of the convolutional layer are that the convolutional kernel size is 3×3, the stride is 1, the padding is 1, and the activation function is ReLU. The parameters of the pooling layer are max pooling, the window size is 2×2, and the stride is 2. The convolution calculation formula:

[0154] F conv (t) = ReLU(Conv(I(t), W) + b) (10)

[0155] Among them, I(t) is the input video frame, W is the convolutional kernel weight, b is the bias, Conv represents the convolution operation, and ReLU is the activation function.

[0156] The pooling calculation formula:

[0157] F pool (t) = MaxPool(F conv (t)) (11)

[0158] Among them, F pool (t) is the pooled feature map, and MaxPool represents the max pooling operation.

[0159] Furthermore, the AIS data feature extraction uses a long short-term memory network (LSTM) to process AIS data and extract time-dynamic features. Assuming the input AIS data is A(t), after passing through the LSTM layer, the AIS feature F AIS (t) is obtained. The calculation formula of the LSTM unit is expressed as

[0160] i t = σ(W i ·[h t-1 , A(t)] + b i ) (12)

[0161] f t = σ(W f ·[h t-1 , A(t)] + b f ) (13)

[0162] o t = σ(W o ·[h t-1 , A(t)] + b o ) (14)

[0163]

[0164] h t = o t * tanh(C t ) (17)

[0165] Among them, A(t) represents the AIS (Automatic Identification System) data input at time t. The AIS data includes the position information, speed, course, and other dynamic information of the ship, and this information is used to extract time-series features in the LSTM network. h t-1 represents the hidden state at time t-1, indicating the memory of the LSTM unit for the information of the previous time step. This hidden state contains the comprehensive information of all previous time steps and is used to calculate the input gate, forget gate, output gate, and candidate cell state at the current time step t. i t represents the activation value of the input gate at time t, indicating how the current AIS data A(t) affects the cell state C t . The input gate controls the contribution degree of the new input to the update of the cell state. f t represents the activation value of the forget gate at time t, indicating the retention degree of the previous time step C t-1 cell state. The forget gate determines which past information will be forgotten or retained. o t represents the activation value of the output gate at time t, controlling how the cell state C t affects the final hidden state h t. The output gate determines the degree of influence from the cell state to the hidden state. represents the candidate cell state at time step t, which is generated based on the current input A(t) and the hidden state h at the previous time step. t-1 It represents the possible update of the cell state in the absence of a gating mechanism. C t-1 represents the cell state at time step t - 1, which contains the cumulative information passed down by the LSTM unit from previous time steps. This state combines the information of the forget gate f t and the input gate i t to generate the cell state C at the current time step. t C t represents the cell state at time step t, which is the core memory of the LSTM unit and is used to store information related to the current task. It combines the cell state C at the previous time step t-1 and the candidate cell state at the current time step h t represents the hidden state at time step t, which represents the output of the LSTM unit at time t. This is calculated based on the current cell state C t and the output gate o t and is used as the final output of the LSTM and is passed to the next time step. W i , W f , W o , W C represent weight matrices, which are used to calculate the input gate, forget gate, output gate, and candidate cell state respectively. These weights are optimized through the training process to accurately reflect the temporal dynamic characteristics in AIS data. b i , b f , b o , b C represent bias vectors, which correspond to the calculations of the input gate, forget gate, output gate, and candidate cell state respectively. The bias is used to adjust the activation value of each gate to help the model better fit the data. σ represents the sigmoid activation function, which is applied to the calculations of the input gate, forget gate, and output gate, and the output value is between [0, 1], controlling the degree of information passing. tanh represents the tanh activation function, which is applied to the calculations of the candidate cell state and the final hidden state, and the output value is between [-1, 1], used to generate a richer state representation.

[0166] Radar data feature extraction uses a convolutional neural network (CNN) to process radar data and extract its spatial features. Assuming the input radar data is R(t), after three layers of convolution and pooling operations, the feature map F rad (t) is obtained. The parameters of the convolution and pooling layers are the same as those in video data feature extraction. Convolution calculation formula:

[0167] Fconv_rad R(t) = ReLU(Conv(R(t), W rad ) + b rad ) (18)

[0168] Where R(t) is the input radar data, W rad is the convolutional kernel weight, and b rad is the bias.

[0169] Pooling calculation formula:

[0170] F pool_rad (t) = MaxPool(F conv_rad (t)) (19)

[0171] Where F pool_rad (t) is the feature map after pooling, and MaxPool represents the max pooling operation.

[0172] Meteorological data feature extraction uses a two-layer multi-layer perceptron (MLP) to process meteorological data and extract key features. Assume the input meteorological data is W(t), and after two fully connected operations, the meteorological feature F met (t) is obtained.

[0173] Among them, the first fully connected layer:

[0174] F fc1 (t) = ReLU(W fc1 ·W(t) + b fc1 ) (20)

[0175] Where W(t) is the input meteorological data, W fc1 is the weight, and b fc1 is the bias.

[0176] The second fully connected layer:

[0177] F met (t) = ReLU(W fc2 ·F fc1 (t) + b fc2 ) (21)

[0178] Where F fc1 (t) is the output feature of the first layer, W fc2 is the weight of the second layer, and b fc2 is the bias.

[0179] Furthermore, an attention mechanism (AttentionMechanism) is used to align features of different modalities. Let the attention weight be α, and the aligned feature be F aligned (t).

[0180] Among them, the calculation of attention weights:

[0181] α vid = softmax(W a1 )·F vid (t)) (22)

[0182] Among them, W a1 is the attention weight matrix of video features, and softmax is the softmax function.

[0183] α AIS = softmax(W a2 ·F AIS (t)) (23)

[0184] Among them, W a2 is the attention weight matrix of AIS features.

[0185] α rad = softmax(W a3 )·F rad (t)) (24)

[0186] Among them, W a3 is the attention weight matrix of radar features.

[0187] α met = softmax(W a4 ·F met (t)) (25)

[0188] Among them, W a4 is the attention weight matrix of meteorological features.

[0189] Feature alignment and fusion:

[0190] F aligned (t)= α vid ·F vid (t)+ α AIS ·F AIS (t)+ α rad ·F rad (t)+ α met ·F met (t) (26)

[0191] Among them, F aligned (t) is the aligned multi-modal feature.

[0192] Furthermore, a bidirectional long short-term memory network (BiLSTM) is used to perform temporal modeling on the fused features, extract their temporal dynamic information, and generate the final environmental perception result E(t).

[0193] E(t) = BiLSTM(F aligned (t)) (27)

[0194] Wherein, BiLSTM represents a bidirectional long short-term memory network operation, and E(t) is the environmental perception result at time t.

[0195] S3. According to the final environmental perception result, use a generative adversarial network to generate various sea conditions and meteorological scenarios, conduct risk assessment and generate a risk map, and generate an avoidance strategy based on the risk assessment result and the risk map to adjust the route to avoid high-risk areas.

[0196] Specifically, construct a generator network (Generator) G and a discriminator network (Discriminator) D. The generator is used to generate simulated sea conditions and meteorological scenarios, and the discriminator is used to judge whether the generated scenarios are realistic.

[0197] The input of the generator G is the noise vector z and the multimodal fusion feature F aligned (t), and the output is the generated simulated scenario G(z, F aligned (t)).

[0198] The generator network structure is a multi-layer perceptron (MLP) network. The input is the noise vector z and the fusion feature F aligned (t), and after being processed by the fully connected layer and the activation function, a simulated scenario is generated.

[0199] G(z, F aligned (t)) = ReLU(W g ·[z, F aligned (t)] + b g ) (28)

[0200] Wherein, W g is the weight matrix of the generator, and b g is the bias vector.

[0201] The input of the discriminator D is the real scenario X real and the generated scenario G(z, F aligned (t)), and the output is the judgment result D(X real , G(z, F aligned (t))).

[0202] The discriminator network structure is a convolutional neural network (CNN). The input is scenario data, and after passing through the convolutional layer and the pooling layer, the discrimination result is output.

[0203] D(X) = σ(Conv(X, W d ) + b d ) (29)

[0204] Among them, X is the input scenario, and W d is the weight matrix of the discriminator, and b d is the bias vector, and σ is the sigmoid activation function.

[0205] Furthermore, an alternating training method is used to train the generator G and the discriminator D respectively.

[0206] Among them, the loss function of the generator G:

[0207]

[0208] Among them, p z (z) represents the noise distribution, and p data (F aligned ) represents the distribution of the multi-modal fusion features.

[0209] Among them, the loss function of the discriminator D:

[0210]

[0211] Among them, p data (X real ) represents the distribution of the real scenario.

[0212] After the training is completed, the generator G can generate various possible sea conditions and meteorological scenarios. By inputting different noise vectors z and multi-modal fusion features F aligned (t), multiple simulated scenarios G(z i ,F aligned (t)) are generated.

[0213] {G(z i ,F aligned (t))∣i = 1, 2, …, N} (32)

[0214] Among them, N is the number of generated scenarios, and z i is different noise vectors.

[0215] Furthermore, a risk assessment is performed on each generated simulated scenario G(z i ,F aligned (t)). Each simulated scenario represents a possible navigation environment, including different sea conditions, weather conditions, and the dynamics of surrounding vessels.

[0216] To accurately evaluate the risk of each simulated scenario, a multi-dimensional risk assessment model M risk is used, which not only considers the traditional physical distance, heading, and speed, but also introduces a dynamic risk factor λ dynamic based on historical data and a non-linear term γ env based on environmental complexity, making the risk assessment more targeted.

[0217] R i = M risk (G(z i , F aligned (t))) + λ dynamic ·HistoryFactor(G(z i , F aligned (t))) + γ env ·EnvComplexity(G(z i , F aligned (t))) (33)

[0218] Wherein, M risk represents a basic risk assessment model, based on basic factors such as physical distance, speed, and course. λ dynamic represents a historical dynamic risk factor, based on risk patterns extracted from historical data under similar navigation conditions. γ env represents a non - linear term of environmental complexity, evaluating the complexity of the environment in the current scenario (such as multi - vessel intersections, bad weather, etc.). HistoryFactor represents the weight of risk patterns under similar conditions in historical data.

[0219] Wherein, HistoryFactor reflects the historical risk under similar navigation conditions. When calculating, it is necessary to find the historical scenario in the historical database that best matches the current scenario G(z i , F aligned (t)), and weight according to the average risk value of these scenarios. The calculation formula:

[0220]

[0221] Wherein, k is the number of selected historical scenarios, w j is the weight of the j - th historical scenario matching the current scenario (determined by the matching degree), R h ist (j) is the risk score of historical scenario j.

[0222] Numerical example: Suppose the current scenario matches 5 historical scenarios, and the matching degree weights are 0.3, 0.2, 0.2, 0.2, 0.1 respectively, and the risk scores of the historical scenarios are 4, 3, 5, 4, 2 respectively. Then HistoryFactor is:

[0223] HistoryFactor = 0.3×4 + 0.2×3 + 0.2×5 + 0.2×4 + 0.1×2 = 3.8 (35)

[0224] Wherein, EnvComplexity represents a measure of environmental complexity, considering the diversity and complexity in the environment, such as vessel density, weather change frequency, etc.

[0225] Among them, EnvComplexity quantifies the complexity of the current navigation environment, considering the combined effects of factors such as vessel density, sea condition changes, and weather conditions. The calculation formula:

[0226] EnvComplexity(G(z i ,F aligned (t)))=α·ρ + β·Δsea + γ·WeatherVar (36)

[0227] Among them, ρ is the vessel density (number of vessels in a unit sea area), Δsea is the amplitude of sea condition changes (such as wave height changes), WeatherVar is the frequency of weather changes, and α, β, γ are the corresponding weight coefficients. Assume there are 8 vessels in the current sea area (ρ = 8), the wave height change is 2 meters (Δsea = 2), the weather changes 2 times per hour (WeatherVar = 2), and the weight coefficients are 0.5, 0.3, 0.2 respectively. Then EnvComplexity is:

[0228] EnvComplexity = 0.5×8 + 0.3×2 + 0.2×2 = 4.6 (37)

[0229] Furthermore, a comprehensive analysis is conducted on the risk scores of all generated scenarios to obtain the comprehensive risk score R of the overall navigation environment total . During the calculation process, a dynamic weighting factor ω i is introduced, which is adjusted according to the occurrence probability and possible severity of the scenario, so that the comprehensive risk score not only reflects the average risk but also pays special attention to high - risk scenarios.

[0230]

[0231] Among them, N is the number of generated scenarios. ω i is the dynamic weighting factor of scenario i, reflecting the occurrence probability and risk severity of the scenario. Numerical example: Assume the risk scores of 3 scenarios are 5, 3, 4 respectively, the occurrence probabilities are 0.5, 0.3, 0.2 respectively, and their risk severity weights are 1.2, 1.0, 1.5 respectively. Then the comprehensive risk score R total is:

[0232] R total =(0.5×1.2)×5+(0.3×1.0)×3+(0.2×1.5)×4=4.2 (39)

[0233] The comprehensive risk score not only provides an assessment of the overall risk but also is used to generate a risk map. The risk map marks high-risk areas and shows the risk levels of different areas through color gradients, providing intuitive information support for subsequent path optimization.

[0234] Furthermore, based on the comprehensive risk score and the generated risk map, an avoidance strategy is formulated. The generation of the avoidance strategy is not only based on the comprehensive risk score but also introduces a path smoothness term α smooth and a path adjustment cost term β cost , ensuring that the new path is safe while avoiding unnecessary frequent adjustments, improving the stability and economy of navigation, as shown below:

[0235]

[0236] Smoothness(P) represents a path smoothness metric that ensures the path has no excessive sharp turns and frequent changes. The calculation formula:

[0237]

[0238] where θ j is the heading angle of the j-th waypoint on path P, and m is the total number of waypoints.

[0239] Numerical example: Assume there are 4 waypoints on the path, and their heading angles are 30°, 45°, 50°, and 40° respectively. Then the smoothness Smoothness(P) is:

[0240] Smoothness(P) = |45 - 30| + |50 - 45| + |40 - 50| = 25° (42)

[0241] AdjustmentCost(P) represents the path adjustment cost, which evaluates the cost required to adjust from the current path to the new path (such as fuel consumption, time delay, etc.). AdjustmentCost evaluates the cost of adjusting from the current path to the new path, considering factors such as additional fuel consumption and time delay. The calculation formula:

[0242] AdjustmentCost(P) = δ fuel ·FuelCost(P) + δ time ·TimeDelay(P) (43)

[0243] where FuelCost(P) is the additional fuel consumption caused by adjusting the path, TimeDelay(P) is the time delay caused by the path adjustment, δ fuel and δ timeis the corresponding weight coefficient. Numerical example: Suppose the adjustment path increases fuel consumption by 100 liters and causes a time delay of 30 minutes. The fuel weight is 2 and the time delay weight is 1.5. Then the path adjustment cost AdjustmentCost(P) is:

[0244] AdjustmentCost(P) = 2×100 + 1.5×30 = 245 (44)

[0245] S4. According to the results of risk assessment, combined with real-time environmental perception data and historical navigation data, use the reinforcement learning algorithm to dynamically adjust the navigation route to obtain an optimized navigation route

[0246] Specifically, define the state s t , representing the environmental state of the ship at time t, including position, speed, heading, environmental perception results, meteorological data, and information about surrounding ships, etc.

[0247] s t = {pos t , vel t , heading t , E(t), weather t , ships t} (45)

[0248] Among them, pos t is the position, vel t is the speed, heading t is the heading, E(t) is the environmental perception result, weather t is the meteorological data, ships t is the information about surrounding ships.

[0249] Define the action a t , representing the navigation decision of the ship at time t, including the steering angle and speed adjustment.

[0250] a t = {steer t , accel t} (46)

[0251] Among them, steer t is the steering angle, accel t is the speed adjustment.

[0252] The reward function R(s t , a t ) is used to evaluate the effect of each state-action pair, encouraging safe and efficient navigation, including:

[0253] Safety reward Rsafety :

[0254]

[0255] where d i is the distance to the i-th surrounding vessel, ∈ is a small constant to prevent the denominator from being zero, and n is the number of surrounding vessels.

[0256] Efficiency reward R efficiency :

[0257]

[0258] where vel desired is the desired speed.

[0259] Risk penalty R risk :

[0260] R risk (s t , a t ) = R total (s t ) (49)

[0261] where R total (s t ) is the comprehensive risk score calculated in step 3.

[0262] Total reward function:

[0263] R(s t , a t ) = λ 1 ·R safety (s t , a t ) + λ 2 ·R efficiency (s t , a t ) - λ 3 ·R risk (s t , a t ) (50)

[0264] where λ 1 , λ 2 , λ 3 are weight coefficients.

[0265] Furthermore, using the policy gradient method in deep reinforcement learning, design a policy network π(a t |s t ; θ), where θ are the parameters of the policy network. The input of the policy network is the state s t , and the output is the probability distribution of the action a t .

[0266] π(a t |s t ;θ) = softmax(f(s t ;θ)) (51)

[0267] where f(s t ;θ) represents the forward propagation process of the policy network, and softmax is the softmax function.

[0268] Furthermore, a value network V(s t ;φ) is designed, where φ are the parameters of the value network, used to estimate the value of state s t . The input of the value network is the state s t , and the output is the value of the state V(s t ).

[0269] V(s t ;φ) = g(s t ;φ) (52)

[0270] where g(s t ;φ) represents the forward propagation process of the value network.

[0271] Furthermore, the policy gradient method is used to optimize the parameters θ of the policy network π(a t |s t ;θ).

[0272] Policy gradient calculation:

[0273]

[0274] where J(θ) is the policy objective function, A t is the advantage function, and T is the time step.

[0275] Advantage function calculation:

[0276] A t = R(s t , a t ) + γV(s t+1 ;φ) - V(s t ;φ) (54)

[0277] where γ is the discount factor.

[0278] Furthermore, the value network V(s t ;φ) is used to optimize the parameters φ of the policy network π(a t |s t ;θ). Loss function of the value network:

[0279]

[0280] Furthermore, during actual navigation, the optimized policy network π(a t |s t ; θ) is used for path optimization. According to the real-time environment perception result E(t) and the current state s t , an action a t is sampled from the policy network to adjust the ship's route.

[0281] a t ~π(a t |s t ; θ)(56)

[0282] Update the state s t , repeat the above process, continuously optimize the path, and ensure the safety and efficiency of navigation.

[0283] S5. Synchronize the optimized navigation route, high-risk areas, and important prompt information in real time through AR display and overlay them for display in combination with augmented reality technology.

[0284] Specifically, the AR display system includes AR glasses or a head-mounted display (HMD), which is closely integrated with the ship's multi-modal data acquisition and path optimization system. The AR device must work properly in various harsh marine environments and support functions such as waterproof, dustproof, and anti-glare. In addition, the AR device should have characteristics such as high resolution, wide viewing angle, and low latency to ensure real-time and clear display of navigation information.

[0285] Furthermore, data is obtained in real time from the path optimization system and the multi-modal data fusion module, and a dedicated low-latency wireless network protocol is used to ensure efficient data transmission and real-time update. An adaptive delay compensation model is adopted to dynamically adjust the refresh frequency of the display content according to the transmission delay to ensure the stability and smoothness of the display:

[0286] ν(t) = ν max ·e -ατ(t) (57)

[0287] where ν(t) represents the refresh frequency of the current display content. ν max represents the maximum refresh frequency allowed by the system. α represents an adjustment parameter that controls the sensitivity to delay. τ(t) represents the current network delay.

[0288] Furthermore, the data from multi-modal data acquisition (step 1), including video, AIS, radar, and meteorological data, will be fused through a deep learning model (step 2) to generate a unified environment perception result F aligned(t). This result has been used for risk assessment in the simulated scenarios generated by the GAN (step 3) and will now be continuously applied in the AR display.

[0289] Specifically, based on the environment perception result F aligned (t) generated by deep learning and GAN, and the comprehensive risk score R total , the AR display content is generated. These contents include optimized routes, safety area and risk area prompts. A dynamic information weighting model is adopted to dynamically adjust the display priorities of each information layer to ensure that key information such as risk prompts and route guidance is always in the core position:

[0290]

[0291] Among them, I display (t) represents the display content at time t. w i (t) represents the weight of the i-th type of information, which is dynamically adjusted based on risk assessment and path optimization. I i (t) represents the i-th type of information, such as routes, safety areas, risk prompts, etc.

[0292] Furthermore, the best route P new from the reinforcement learning path optimization (step 4) is displayed in the form of a dynamic path in the AR. The display of the path combines color coding (green represents safety, red represents high risk) and dynamically adjusts the transparency according to the multi-modal fusion result to ensure that high-risk areas can attract the attention of the crew.

[0293] Furthermore, the environmental information is overlaid. The environmental information (such as weather, sea conditions, and dynamic of surrounding vessels) comes from the multi-modal data fusion module (step 2) and is overlaid and displayed in the AR field of view in a transparent manner. Through the hierarchical display technology, the less important information (such as weather changes) is placed at the edge position to ensure that the key navigation information is always clearly visible.

[0294] Furthermore, to ensure the smoothness of the path display and the transparency of the information, a joint optimization formula is introduced, which combines the environmental complexity and the comprehensive risk score to dynamically adjust the visual effect of the path display:

[0295]

[0296] Among them, λ display1 and λ display2 respectively represent the adjustment parameters of the path smoothness and the information transparency, θ j represents the course angle of the j-th waypoint on the path P. Transparency(R i ) represents the transparency of the risk area R i , which is dynamically adjusted according to the risk level.

[0297] Furthermore, for specific nautical scenarios, a risk accumulation term δ is added to the path display risk , which is used to record the cumulative risk values that occurred in similar environments historically, helping the crew to anticipate potential dangerous areas in advance:

[0298]

[0299] Among them, γ env is the risk accumulation coefficient.

[0300] Furthermore, based on the real-time multi-modal environment perception E(t), the AR display content is dynamically updated to ensure the timeliness and accuracy of information. The content update is based on the real-time data from steps 2 and 3, as well as the latest results of the reinforcement learning path optimization.

[0301] Gesture recognition: Combining the crew's operation behaviors recognized in the path optimization module, gesture recognition is realized. The crew can adjust the display view or switch information layers through gestures such as swiping and clicking.

[0302] Voice interaction: Based on the deep learning-based speech recognition model, the crew can control the AR display content through voice commands, such as "display the risk area", "adjust the route", etc. The voice interaction directly interacts with the data of the path optimization module.

[0303] Furthermore, an interaction response function χ(t) is introduced in the dynamic interaction design, which adjusts the system response speed according to the crew's operation frequency and environmental changes:

[0304]

[0305] Among them, κ represents the baseline value of the response speed. β represents the adjustment parameter, which controls the sensitivity of the system to the operation frequency. σ represents the current operation frequency. σ 0 represents the set baseline value of the operation frequency.

[0306] Furthermore, the ambient light data obtained from the multi-modal data acquisition module will be used to automatically adjust the brightness and contrast of the AR display to adapt to different lighting conditions. Especially in the case of strong light reflection or insufficient light, the readability of the display content is improved through dynamic adjustment.

[0307] Furthermore, to ensure the clarity of the display content under various lighting conditions, a brightness adjustment formula based on the ambient light intensity is designed:

[0308]

[0309] Among them, B display (t) represents the display brightness at time t. B baseRepresents the reference brightness value, applicable to ordinary lighting conditions. L ambient L(t) represents the current ambient light intensity. opt Represents the ideal ambient light intensity. ∈ represents the adjustment parameter to smooth the brightness change.

[0310] Furthermore, in combination with the image enhancement algorithm, the strong light reflection areas are automatically identified and processed to ensure that the contrast and color saturation of these areas are effectively improved. The visual enhancement function combined with the environmental simulation results generated by GAN enables the accurate identification of high-risk areas under any lighting conditions.

[0311] Furthermore, aiming at the special requirements of complex lighting conditions at sea, an ambient light adaptive regularization term ρ is designed env , and this term is added to the brightness adjustment formula to ensure stability in the case of strong reflection or low contrast:

[0312]

[0313] where ρ env represents the ambient light adaptive coefficient. Represents the gradient change rate of the ambient light, reflecting the sudden change of light.

[0314] Furthermore, the AR display system is integrated with all the previously developed modules (multi-modal data acquisition and preprocessing, deep learning data fusion, GAN environmental simulation, reinforcement learning path optimization) to ensure the coherence of the data flow and the real-time nature of the processing.

[0315] It is connected to the core module of the ship navigation system through a unified interface, and standardized APIs are used for data interaction and synchronization of display content.

[0316] Furthermore, actual navigation tests are carried out, focusing on verifying the performance of the system in a dynamic environment. By collecting the operation feedback of the crew, the AR display content and interaction methods are further optimized. Combining the test data, the algorithm parameters of each module are tuned to ensure the stability and reliability of the system under different sea conditions and lighting conditions.

[0317] Furthermore, the performance of the AR display system is monitored in real time, especially its performance in high-risk situations. Combining the feedback of the path optimization module, potential problems are detected and processed in a timely manner to ensure the stability of the system during long voyages.

[0318] It should be noted that the above-described work process is only illustrative and does not limit the protection scope of the present invention. In actual applications, those skilled in the art can select some or all of them according to actual needs to achieve the purpose of the solution of this embodiment, and no limitation is made here.

[0319] In addition, for the technical details not described in detail in this embodiment, reference may be made to the parameter operation method provided in any embodiment of the present invention, which will not be elaborated herein.

[0320] It should be noted that in this document, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including a..." does not exclude the presence of another identical element in the process, method, article or system comprising that element.

[0321] The serial numbers of the above embodiments of the present invention are only for description and do not represent the superiority or inferiority of the embodiments.

[0322] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as a read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0323] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied to other related technical fields, shall be similarly included in the patent protection scope of the present invention.

Claims

1. A real-time ship navigation route planning method based on augmented reality, characterized in that: The method comprises: S1. Real-time collection of various environmental data around the ship, and pre-processing of the various environmental data; the various environmental data include video data, AIS data, radar data and meteorological data; S2. Process video data through convolutional neural networks to extract environmental features, process AIS data through long short-term memory networks to extract dynamic features, process radar data through convolutional neural networks to extract its spatial features, use two-layer multi-layer perceptrons to process meteorological data to extract meteorological features, use attention mechanisms to align features of different modalities and fuse them to obtain fused features, and then use bidirectional long short-term memory networks to perform time series modeling on the fused features to generate the final environmental perception results; S3. Based on the final environmental perception results, a variety of sea conditions and weather scenarios are generated using a generative adversarial network to conduct risk assessment and generate a risk map. Based on the risk assessment results and the risk map, an avoidance strategy is generated to adjust the route to avoid high-risk areas. S4. Based on the results of risk assessment, combined with real-time environmental perception data and historical navigation data, the navigation route is dynamically adjusted using a reinforcement learning algorithm to obtain an optimized navigation route; S5. The optimized navigation route, high-risk areas and important reminder information are displayed through real-time synchronization of AR display and navigation system, combined with augmented reality technology overlay; Among them, based on the results of risk assessment and risk map, avoidance strategies are generated to adjust routes to avoid high-risk areas, including: Introduce path smoothing items based on risk assessment results and risk maps and path adjustment cost , generate the avoidance strategy, which is expressed as follows: ； in, represents a new avoidance strategy, represents the risk score of the old avoidance strategy, represents the path smoothness measure, represents the cost of adjusting from the current path to the new path; The path smoothness metric is expressed as follows: ； in, Indicates the path Previous The heading angle of the waypoint, Indicates the total number of waypoints; The cost of adjusting from the current path to the new path is expressed as follows: ； in, represents the additional fuel consumption caused by adjusting the route, Indicates the time delay caused by path adjustment, and Represents the corresponding weight coefficient.

2. The method for real-time ship navigation route planning based on augmented reality according to claim 1, characterized in that: The preprocessing includes multimodal data time synchronization processing, video data preprocessing, AIS data preprocessing, radar data preprocessing and meteorological data preprocessing; wherein the video data preprocessing includes denoising processing and frame extraction; the AIS data preprocessing includes deduplication and filtering processing and data interpolation processing; the radar data preprocessing includes echo denoising and data gridding processing; the meteorological data preprocessing includes data smoothing processing and data standardization processing.

3. The method for real-time ship navigation route planning based on augmented reality according to claim 1, characterized in that: The video data is processed by a convolutional neural network to extract environmental features, which is expressed as follows: ； in, represents the input video frame, represents the convolution kernel weight, Indicates bias, represents the convolution operation, represents the activation function, Represents the output features of the current convolutional layer; The pooling calculation formula of the convolutional neural network is expressed as follows: ； in, is the feature map after pooling, Represents the maximum pooling operation; The AIS data is processed by the long short-term memory network to extract dynamic features, which are expressed as follows: ； ； ； ； ； ； in, Indicates the input AIS data, Indicates the cell state, Indicates the hidden state, represents the bias vector, represents the sigmoid activation function, represents the tanh activation function, represents the hidden state at time t-1, represents the cell state at t-1, represents the candidate cell state at the current time step, Indicates time The input gate activation value at time , represents the weight matrix, Indicates time The output gate activation value at time , Indicates time The forget gate activation value at the moment; The radar data is processed by convolutional neural network to extract its spatial features, which are expressed as follows: ； in, Represents input radar data, represents the convolution kernel weight, Indicates bias, represents the activation function, Represents the output of the current convolutional neural network; The pooling calculation formula is: ； in, represents the feature map after pooling, Represents the maximum pooling operation; The two-layer multi-layer perceptron is used to process meteorological data and extract meteorological features, which are expressed as follows: The first layer is fully connected: ； in, Indicates input meteorological data, represents the weight, Indicates bias, Represents the output of the first fully connected layer; The second layer is fully connected: ； in, represents the first layer output features, represents the second layer weight, Indicates bias, Represents the output of the second fully connected layer.

4. The method for real-time ship navigation route planning based on augmented reality according to claim 3, characterized in that: The attention mechanism is used to align the features of different modalities and fuse them to obtain fused features, and then a bidirectional long short-term memory network is used to perform temporal modeling on the fused features to generate the final environmental perception result, which is expressed as follows: Assume the attention weight is , the aligned features are , then the attention weight calculation is: ； in, represents the video attention weight, is the attention weight matrix of video features, is the softmax function; ； in, represents the AIS attention weight, is the attention weight matrix of AIS features; ； in, represents the radar attention weight, is the attention weight matrix of radar features; ； in, represents the meteorological attention weight, is the attention weight matrix of meteorological features; Feature alignment and fusion: ； in, represents the video feature map, Indicates AIS characteristics, represents the radar characteristic graph, Indicates meteorological characteristics. is the aligned multimodal feature; Use a bidirectional long short-term memory network to perform temporal modeling on the fusion features, extract temporal dynamic information, and generate the final environmental perception results , which is expressed as follows: ； in, represents the bidirectional long short-term memory network operation, For time Environmental perception results.

5. The method for real-time ship navigation route planning based on augmented reality according to claim 1, characterized in that: The structure of the generative adversarial network is constructed as follows: Generator The input is a noise vector And multimodal fusion features , the output is the generated simulation scenario , where the generator The network structure is a multi-layer perceptron network; Discriminator The input is the real scene and generate scenarios , the output is the judgment result , where the discriminator The network structure is: convolutional neural network; Use the alternating training method to train the generator separately and the discriminator ; Among them, the generator The loss function is expressed as follows: ； in, represents the noise distribution, represents the distribution of multimodal fusion features, represents the noise vector, express expectations; Discriminator The loss function is expressed as follows ； in, Represents the distribution of real scenarios; According to the input of different noise vector And multimodal fusion features , generate multiple simulation scenarios .

6. A real-time ship navigation route planning method based on augmented reality according to claim 5, characterized in that: Introduced dynamic risk factors based on historical data and nonlinear terms based on environmental complexity For each generated simulation scenario Conduct a risk assessment, expressed as follows: ； in, represents the basic risk assessment model; represents the historical dynamic risk factor; Represents the nonlinear term of environmental complexity; represents the risk model weight in historical data; wherein, Reflects historical risks under similar sailing conditions. The calculation requires comparing the historical database with the current scenario. The best matching historical scenarios, weighted by the average risk value of these scenarios, are expressed as follows: ； in, represents the number of historical scenarios selected, Indicates the first The weight of each historical scenario, Representing historical situation risk score; Comprehensively analyze the risk scores of all generated scenarios to obtain a comprehensive risk score under the overall navigation environment , which is expressed as follows: ； in, represents the dynamic weighting factor, Indicates the number of scenarios generated.

7. The method for real-time ship navigation route planning based on augmented reality according to claim 1, characterized in that: The S4 specifically includes: S401, define status , indicating that the ship is at time The state of the environment; define actions , indicating that the ship is at time navigation decisions; S402. Reward Function Used to evaluate the effect of each state-action pair, where the reward function Safety bonus included , efficiency rewards and risk penalties ; S403. Use the policy gradient method to design a policy network ,in is the parameter of the policy network, and the input of the policy network is the state , the output is action The probability distribution of S404. Design a value network ,in is the parameter of the value network, used to estimate the state The value of, where the input of the value network is the state , the output is the value of the state ; S405. Optimize the policy network using the policy gradient method Parameters , using the value network Optimizing Strategy Network Parameters ; S406: Use the optimized strategy network Perform real-time path optimization and perceive the results based on real-time environment and current status , sample actions from the policy network , adjust the ship's route.

8. The method for real-time ship navigation route planning based on augmented reality according to claim 7, characterized in that: The reward function Safety bonus included , efficiency rewards and risk penalties , which has the following representation: Safety Rewards : ； in, Indicates The distance to the surrounding ships, represents a small constant that prevents the denominator from being zero, Indicates the number of surrounding ships; Efficiency Rewards : ； in, represents the expected speed, Indicates actual speed; Risk Penalty : ； in, represents the comprehensive risk score; Total Reward Function : ； in, Represents the weight coefficient.

9. The method for real-time ship navigation route planning based on augmented reality according to claim 1, characterized in that: In the real-time synchronization between AR display and navigation system, a joint optimization formula is introduced to dynamically adjust the visual effect of path display by combining environmental complexity and comprehensive risk score, which is expressed as follows: ； in, and They represent the adjustment parameters of path smoothness and information transparency, Indicates the path Previous The heading angle of each waypoint; Indicates risk areas transparency and dynamically adjust according to risk levels.

10. The method for real-time ship navigation route planning based on augmented reality according to claim 1, characterized in that: Design risk accumulation items for specific sailing scenarios , used to record the cumulative risk values ​​that occurred in similar environments in history, to help crew members predict potential dangerous areas in advance, as shown below: ； in, is the cumulative risk factor.

Images