Real-time ship navigation route planning method based on augmented reality

Abstract

The invention provides a real-time ship navigation route planning method based on augmented reality, and the method comprises the steps: collecting various environmental data around a ship in real time, and carrying out the preprocessing of the various environmental data; processing video data through a convolutional neural network, extracting environment features, extracting dynamic features through a long-short-term memory network, extracting spatial features through the convolutional neural network, extracting meteorological features by using a two-layer multi-layer perceptron, and fusing to obtain fused features; a generative adversarial network is utilized to generate various sea conditions and meteorological scenes, risk assessment is carried out, a risk map is generated, an avoidance strategy is generated, and a route is adjusted to avoid a high-risk area; in combination with the real-time environment perception data and the historical navigation data, dynamically adjusting the navigation route by adopting a reinforcement learning algorithm to obtain an optimized navigation route; the display is performed through real-time synchronization of AR display and a navigation system in combination with augmented reality technology superposition. According to the scheme, the safety and navigation efficiency of the ship are improved.

Description

Technical Field

[0001] The present invention belongs to the field of ship navigation route planning, and in particular relates to a real-time ship navigation route planning method based on augmented reality. Background Art

[0002] With the rapid development of the global shipping industry, the importance of ship navigation safety and efficiency has become increasingly prominent. Traditional ship navigation systems mainly rely on manual observation, electronic charts, and automatic identification systems (AIS). However, these methods have obvious limitations. First, manual observation is easily affected by weather, visibility, and human factors, and it is difficult to provide real-time and accurate environmental perception information. Secondly, although electronic charts can provide basic information on navigation routes, they lack the support of real-time dynamic data and cannot cope with sudden environmental changes and risks. In addition, although the AIS system can provide the location and movement information of other ships, it is difficult to ensure the safety of navigation in high-density navigation areas and complex sea conditions by relying solely on AIS information.

[0003] Some existing advanced navigation systems attempt to introduce radar and video surveillance technologies to improve the accuracy of environmental perception. However, these systems usually operate independently and lack effective fusion and comprehensive analysis of multimodal data, resulting in information island problems and difficulty in providing comprehensive environmental perception and risk assessment. In addition, although some systems have introduced artificial intelligence technologies such as machine learning and deep learning for route optimization and risk prediction, their application scenarios are limited and lack the ability to dynamically adjust real-time environmental data. Especially in ocean voyages and complex sea conditions, these systems are often unable to respond and adjust in real time, affecting the safety and efficiency of navigation. Summary of the invention

[0004] The purpose of this invention is to propose a real-time ship navigation route planning method based on augmented reality. By integrating shipboard video, AIS information, radar data and meteorological data, deep learning and generative adversarial network (GAN) technology are applied to achieve real-time optimization and risk assessment of ship routes. Augmented reality technology intuitively displays the optimized routes and risk warnings in the crew's field of vision, greatly improving the intuitiveness and accuracy of navigation decisions. This method not only overcomes the information island problem of traditional navigation systems, but also makes up for the shortcomings of existing intelligent navigation systems in real-time dynamic adjustment and comprehensive risk assessment, providing a new solution for ship navigation.

[0005] In order to achieve the above object, the present invention provides a real-time ship navigation route planning method based on augmented reality, the method comprising:

[0006] S1. Real-time collection of various environmental data around the ship, and pre-processing of the various environmental data; the various environmental data include video data, AIS data, radar data and meteorological data;

[0007] S2. Process video data through convolutional neural networks to extract environmental features, process AIS data through long short-term memory networks to extract dynamic features, process radar data through convolutional neural networks to extract its spatial features, use two-layer multi-layer perceptrons to process meteorological data to extract meteorological features, use attention mechanisms to align features of different modalities and fuse them to obtain fused features, and then use bidirectional long short-term memory networks to perform time series modeling on the fused features to generate the final environmental perception results;

[0008] S3. Based on the final environmental perception results, a variety of sea conditions and weather scenarios are generated using a generative adversarial network to conduct risk assessment and generate a risk map. Based on the risk assessment results and the risk map, an avoidance strategy is generated to adjust the route to avoid high-risk areas.

[0009] S4. Based on the results of risk assessment, combined with real-time environmental perception data and historical navigation data, the navigation route is dynamically adjusted using a reinforcement learning algorithm to obtain an optimized navigation route;

[0010] S5. The optimized navigation route, high-risk areas and important reminder information are displayed through real-time synchronization of AR display and navigation system, combined with augmented reality technology overlay;

[0011] Among them, based on the results of risk assessment and risk map, avoidance strategies are generated to adjust routes to avoid high-risk areas, including:

[0012] According to the risk assessment results and risk map, the path smoothing term α is introduced smooth and the path adjustment cost term β cost , the production avoidance strategy is expressed as follows:

[0013]

[0014] Among them, P new represents the new avoidance strategy, R total (P) represents the risk score of the old avoidance strategy, Smoothness(P) represents the path smoothness measure, and AdjustmentCost represents the cost of adjusting from the current path to the new path;

[0015] The path smoothness metric is expressed as follows:

[0016]

[0017] Among them, θ jrepresents the heading angle of the jth waypoint on path P, and m represents the total number of waypoints;

[0018] The cost of adjusting from the current path to the new path is expressed as follows:

[0019] AdjustmentCost(P) = δ fuel FuelCost(P)+δ time TimeDelay(P)

[0020] Among them, FuelCost(P) represents the additional fuel consumption caused by adjusting the route, TimeDelay(P) represents the time delay caused by the route adjustment, and δ fuel and δ time Represents the corresponding weight coefficient.

[0021] Preferably, the preprocessing includes multimodal data time synchronization processing, video data preprocessing, AIS data preprocessing, radar data preprocessing and meteorological data preprocessing; wherein the video data preprocessing includes denoising processing and frame extraction; the AIS data preprocessing includes deduplication and filtering processing and data interpolation processing; the radar data preprocessing includes echo denoising and data gridding processing; the meteorological data preprocessing includes data smoothing processing and data standardization processing.

[0022] Preferably, the video data is processed by a convolutional neural network to extract environmental features, which are expressed as follows:

[0023] F conv (t) = ReLU(Conv(I(t),W)+b)

[0024] Where I(t) represents the input video frame, W represents the convolution kernel weight, b represents the bias, Conv represents the convolution operation, ReLU represents the activation function, and F conv (t) represents the output feature of the current convolutional layer;

[0025] The pooling calculation formula of the convolutional neural network is expressed as follows:

[0026] F pool (t) = MaxPool(F conv (t))

[0027] Among them, F pool (t) is the feature map after pooling, and MaxPool represents the maximum pooling operation;

[0028] The AIS data is processed by the long short-term memory network to extract dynamic features, which are expressed as follows:

[0029] i t =σ(W i·[h t-1 ,A(t)]+b i )

[0030] f t =σ(W f ·[h t-1 ,A(t)]+b f )

[0031] o t =σ(W o ·[h t-1 ,A(t)]+b o )

[0032]

[0033] h t =o t *tanh(C t )

[0034] Where A(t) represents the input AIS data, C t Indicates the cell state, h t represents the hidden state, b i ,b f ,b o ,b C represents the bias vector, σ represents the sigmoid activation function, tanh represents the tanh activation function, and h t-1 represents the hidden state at time t-1, C t-1 represents the cell state at t-1, represents the candidate cell state at the current time step, i t represents the input gate activation value at time t, W i ,W f ,W o ,W C represents the weight matrix, o t represents the output gate activation value at time t, f t Represents the activation value of the forget gate at time t;

[0035] The radar data is processed by convolutional neural network to extract its spatial features, which are expressed as follows:

[0036] F conv_rad (t) = ReLU(Conv(R(t),W rad )+b rad )

[0037] Where R(t) represents the input radar data, W rad represents the convolution kernel weight, b rad represents bias, ReLU represents activation function, Fconv_rad (t) represents the output of the current convolutional neural network;

[0038] The pooling calculation formula is:

[0039] F pool_rad (t) = MaxPool(F conv_rad (t))

[0040] Among them, F pool_rad (t) represents the feature map after pooling, and MaxPool represents the maximum pooling operation;

[0041] The two-layer multi-layer perceptron is used to process meteorological data and extract meteorological features, which are expressed as follows:

[0042] The first layer is fully connected:

[0043] F fc1 (t) = ReLU(W fc1 ·W(t)+b fc1 )

[0044] Where W(t) represents the input meteorological data, W fc1 represents the weight, b fc1 Indicates bias, F fc1 (t) represents the output of the first fully connected layer;

[0045] The second layer is fully connected:

[0046] F met (t) = ReLU(W fc2 ·F fc1 (t)+b fc2 ))

[0047] Among them, F fc1 (t) represents the output feature of the first layer, W fc2 represents the second layer weight, b fc2 Indicates bias, F met (t) represents the output of the second fully connected layer.

[0048] Preferably, the attention mechanism is used to align the features of different modalities and fuse them to obtain fused features, and then a bidirectional long short-term memory network is used to perform temporal modeling on the fused features to generate the final environmental perception result, which is expressed as follows:

[0049] Let the attention weight be α and the aligned feature be F aligned (t), then the attention weight is calculated as:

[0050] α vid =softmax(W a1 ·F vid (t))

[0051] Among them, α vid represents the video attention weight, W a1 is the attention weight matrix of video features, and softmax is the softmax function;

[0052] α AIS =softmax(W a2 ·F AIS (t))

[0053] Among them, α AIS represents the AIS attention weight, W a2 is the attention weight matrix of AIS features;

[0054] α rad =softmax(W a3 ·F rad (t))

[0055] Among them, α rad represents the radar attention weight, W a3 is the attention weight matrix of radar features;

[0056] α met =softmax(W a4 ·F met (t)))

[0057] Among them, α met represents the meteorological attention weight, W a4 is the attention weight matrix of meteorological features;

[0058] Feature alignment and fusion:

[0059] F aligned (t) = α vid ·F vid (t)+α AIS ·F AIS (t)+α rad ·F rad (t)+α met ·F met (t)

[0060] Among them, F vid (t) represents the video feature map, F AIS (t) represents the AIS feature, F rad (t) represents the radar characteristic graph, F met (t) represents meteorological characteristics, F aligned (t) is the aligned multimodal feature;

[0061] A bidirectional long short-term memory network is used to perform temporal modeling on the fusion features, extract temporal dynamic information, and generate the final environmental perception result E(tv, expressed as follows:

[0062] E(t)=BiLSTM(F aligned (t))

[0063] Among them, BiLSTM represents the bidirectional long short-term memory network operation, and E(t) is the environmental perception result at time t.

[0064] Preferably, the structure of the generative adversarial network is constructed as follows:

[0065] The input of the generator G is the noise vector z and the multimodal fusion feature F aligned (t), the output is the generated simulation scenario G(z,F aligned (t)), where the network structure of the generator G is a multi-layer perceptron network;

[0066] The input of the discriminator D is the real scene X real and generate scenarios G(z,F aligned (t)), the output is the judgment result D(X real ,G(z,F aligned (t))), where the network structure of the discriminator D is: convolutional neural network;

[0067] Use the alternating training method to train the generator G and the discriminator D separately; the loss function of the generator G is expressed as follows:

[0068]

[0069] Among them, p z (z) represents the noise distribution, p data (F aligned ) represents the distribution of multimodal fusion features, z represents the noise vector, and E represents the expectation;

[0070] The loss function of the discriminator D is expressed as follows

[0071]

[0072] Among them, p data (X real ) represents the distribution of the real scenario;

[0073] According to the input of different noise vectors z and multimodal fusion features F aligned (t), generate multiple simulation scenarios G(z i ,F aligned (t)).

[0074] Preferably, a dynamic risk factor λ based on historical data is introduced dynamic and a nonlinear term γ based on the complexity of the environment env For each generated simulation scenario G(z i ,F aligned (t)) to conduct a risk assessment, expressed as follows:

[0075] R i =M risk (G(z i ,F aligned (t)))+λ dynamic ·HistoryFactor(G(z i ,F aligned (t)))+γ env ·EnvComplexity(G(z i ,F aligned (t)))

[0076] Among them, M risk represents the basic risk assessment model; dynamic represents the historical dynamic risk factor; γ env represents the nonlinear term of environmental complexity; HistoryFactor represents the risk pattern weight in historical data; wherein, the HistoryFactor reflects the historical risk under similar navigation conditions, and the calculation requires the comparison between the historical database and the current scenario G(z i ,F aligned (t)) are the best matching historical scenarios and are weighted according to the average risk value of these scenarios, expressed as follows:

[0077]

[0078] Where k represents the number of selected historical scenarios, w j represents the weight of the jth historical scenario that matches the current scenario, R hist (j) represents the risk score of historical scenario j;

[0079] The risk scores of all generated scenarios are comprehensively analyzed to obtain the comprehensive risk score R under the overall navigation environment. total , which is expressed as follows:

[0080]

[0081] Among them, ω i represents the dynamic weighting factor, and N represents the number of generated scenarios.

[0082] Preferably, the S4 specifically includes:

[0083] S401, define state st , represents the environmental state of the ship at time t; define action a t , represents the navigation decision of the ship at time t;

[0084] S402, reward function R(s t ,a t ) is used to evaluate the effect of each state-action pair, where the reward function R(s t ,a t ) including safety bonus R safety , Efficiency Reward R efficiency and risk penalty R risk ;

[0085] S403, using the policy gradient method, design a policy network π(a t |s t ; θ), where θ is the parameter of the policy network, and the input of the policy network is the state s t , the output is action a t The probability distribution of

[0086] S404、Design a value network V(s t ; φ), where φ is the parameter of the value network, used to estimate the state s t The value of the network, where the input of the value network is the state s t , the output is the value of the state V(s t );

[0087] S405, using the policy gradient method to optimize the policy network π (a t |s t ;θ) parameters θ, using the value network V(s t ;φ) optimizes the strategy networkπ(a t |s t ; θ) parameter φ;

[0088] S406, using the optimized strategy network π(a t |s t ; θ) performs real-time path optimization and uses the real-time environment perception result E(t) and the current state s t , sample action a from the policy network t , adjust the ship's route.

[0089] Preferably, the reward function R(s t ,a t ) including safety bonus R safety , Efficiency Reward R efficiency and risk penalty R risk , which has the following representation:

[0090] Safety Reward R safety (s t ,a t ):

[0091]

[0092] Among them, d i represents the distance to the i-th surrounding ship, ∈ represents a small constant to prevent the denominator from being zero, and n represents the number of surrounding ships;

[0093] Efficiency Reward R efficiency (s t ,a t ):

[0094]

[0095] Among them, vel desired represents the expected speed, vel t Indicates actual speed;

[0096] Risk penalty R risk (s t ,a t ):

[0097] R risk (s t ,a t )=R total (s t )

[0098] Among them, R total (s t ) represents the comprehensive risk score;

[0099] The total reward function R(s t ,a t ):

[0100] R(s t ,a t )=λ1·R safety (s t ,a t )+λ2·R efficiency (s t ,a t )-λ3·R risk (s t ,a t )

[0101] Among them, λ1, λ2, λ3 represent weight coefficients.

[0102] Preferably, in the real-time synchronization between the AR display and the navigation system, a joint optimization formula is introduced to dynamically adjust the visual effect of the path display by combining the environmental complexity and the comprehensive risk score, as shown below:

[0103]

[0104] Among them, λ display1 and λ display2 They represent the adjustment parameters of path smoothness and information transparency, θ j represents the heading angle of the j-th waypoint on path P; Transparency (R i ) represents the risk area R i The transparency of the information is adjusted dynamically according to the risk level.

[0105] Preferably, for a specific navigation scenario, the risk accumulation term δ is designed risk , used to record the cumulative risk values ​​that occurred in similar environments in history, to help crew members predict potential dangerous areas in advance, as shown below:

[0106]

[0107] Among them, γ env is the cumulative risk factor.

[0108] The beneficial technical effects of the present invention are at least as follows:

[0109] (1) The present invention uses a deep learning algorithm to integrate shipborne video, AIS, radar and meteorological data to provide more comprehensive and accurate environmental perception. Compared with the prior art, the present invention can overcome the information island problem, realize comprehensive analysis of different data sources, and provide real-time environmental perception information.

[0110] (2) The present invention uses GAN technology to generate a variety of possible sea conditions and weather scenarios for comprehensive risk assessment. Existing systems rely on static and single data sources for risk assessment, while the present invention uses GAN technology to achieve dynamic and diversified risk prediction, greatly improving the accuracy and reliability of risk assessment.

[0111] (3) The present invention uses a reinforcement learning algorithm to dynamically adjust and optimize routes based on real-time environmental data and historical navigation data. Compared with the static path planning of traditional and existing systems, the present invention can respond to environmental changes in real time, provide dynamic path optimization solutions, and ensure efficient and safe navigation.

[0112] (4) Through augmented reality technology, the optimized routes, risk areas and important reminder information are superimposed and displayed in the crew's field of vision, providing intuitive navigation and decision support. The information display of traditional navigation systems is relatively simple and abstract, while the AR display of the present invention can improve the intuitiveness and operability of information, helping crew members better understand and respond to the navigation environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0114] Figure 1 This is a flow chart of a real-time ship navigation route planning method based on augmented reality in the present invention. DETAILED DESCRIPTION

[0115] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0116] like Figure 1 As shown, an embodiment of the present invention provides a real-time ship navigation route planning method based on augmented reality, and the method includes the following steps S1-S5:

[0117] S1. Collect various environmental data around the ship in real time and pre-process the various environmental data; the various environmental data include video data, AIS data, radar data and meteorological data.

[0118] A variety of sensor equipment is configured around the ship, including high-resolution video cameras, AIS equipment, radars and meteorological sensors. Ensure that the equipment can work properly in various navigation environments. Video cameras are set at different key positions of the ship, covering a 360-degree field of view around the ship and collecting video data with a resolution of 1080p or higher. The AIS equipment is configured to receive and send information such as ship identification, position, speed, heading, timestamp, etc., and communicate with other ships and shore-based equipment. The radar sets the radar antenna angle and sensitivity to ensure that it can detect surrounding ships, obstacles and sea conditions, and the collection frequency is set to 10 times per second. The meteorological sensor installs temperature, humidity, wind speed, wind direction and air pressure sensors to collect environmental meteorological data in real time.

[0119] Furthermore, when collecting data, in order to ensure the time consistency of each data source, a global time synchronization mechanism is used. High-precision GPS timestamps are used to synchronize the collection time of all devices to ensure that the data can be aligned at the same time point.

[0120] Specifically, the time synchronization formula is:

[0121] T s =T GPS +Δt (1)

[0122] Among them, T s is the timestamp after synchronization, T GPS is the GPS timestamp, and Δt is the time delay compensation of the device. The specific time delay compensation amount can be determined through the calibration process of the device.

[0123] Specifically, video data preprocessing includes the following:

[0124] Denoising: Use a custom multi-scale Gaussian filter to remove noise from the video. Multi-scale filtering formula:

[0125]

[0126] Among them, I smooth (x,y) is the denoised video frame, The standard deviation is σ i Gaussian kernel, w i is the weight, I(x,y) is the original video frame, and n is the number of scales.

[0127] Frame extraction: Extract key frames from the video stream at fixed intervals to reduce the load of data processing. Key frame extraction formula:

[0128] F key (t)=I(t) if t mod N=0 (3)

[0129] Among them, F key (t) is the key frame, I(t) is the video frame at time t, and N is the extraction interval.

[0130] AIS data preprocessing includes the following:

[0131] Deduplication and filtering: Deduplication of AIS data is performed to ensure that only the latest data is retained at each time point, and abnormal and erroneous data is filtered out. Deduplication and filtering formula:

[0132]

[0133] Among them, A filtered (t) is the filtered AIS data, a i(t) is the i-th AIS data point at time t, and ε is the anomaly dataset.

[0134] Data interpolation: Use the Bezier curve interpolation method based on the navigation trajectory to fill in the missing time points of data to ensure data continuity. Bezier curve interpolation formula:

[0135] P(t)=(1-t) 2 P0+2(1-t)tP1+t 2 P2 (5)

[0136] Wherein, P(t) is the interpolated AIS position data, P0 is the previous position point, P1 is the current position point, P2 is the next position point, and t is the interpolation parameter.

[0137] Radar data preprocessing includes the following:

[0138] Echo denoising: Adaptive median filter is used to remove noise from radar data. The adaptive median filter formula is:

[0139] R filtered (x,y)=median{R(x+i,y+j)} (6)

[0140] Among them, R filtered (x, y) is the denoised radar data, R(x, y) is the original radar data, and (i, j) is the filter window size, which is adaptively adjusted according to the noise level.

[0141] Data gridding: Map the radar data to a fixed polar coordinate grid to facilitate subsequent processing. The gridding formula is:

[0142] R grid (ρ,θ)=R(x,y) (7)

[0143] Among them, R grid (ρ,θ) is the gridded radar data, ρ is the distance, θ is the angle, and (x,y) is the radar coordinate.

[0144] Meteorological data preprocessing includes the following:

[0145] Data smoothing: Double exponential smoothing method is used to smooth meteorological data. The double exponential smoothing formula is:

[0146] W smooth (t) = αW(t) + (1-α)W smooth (t-1)+β(W(t)-W(t-1)) (8)

[0147] Among them, W smooth(t) is the smoothed meteorological data, W(t) is the original meteorological data, and α and β are smoothing coefficients.

[0148] Data standardization: Standardize the meteorological data to ensure that the data is on the same scale. The standardization formula is:

[0149]

[0150] Among them, W norm (t) is the standardized meteorological data, μ W and σ W are the mean and standard deviation of the meteorological data respectively.

[0151] S2. Process video data through convolutional neural networks to extract environmental features, process AIS data through long short-term memory networks to extract dynamic features, process radar data through convolutional neural networks to extract its spatial features, use two-layer multi-layer perceptrons to process meteorological data to extract meteorological features, use attention mechanisms to align features of different modalities and fuse them to obtain fused features, and then use bidirectional long short-term memory networks to perform time series modeling on the fused features to generate the final environmental perception results.

[0152] Specifically, a convolutional neural network (CNN) is used to extract the spatial features of the video frame. Assuming the input video frame is I(t), after three layers of convolution and pooling operations, the feature map F is obtained. vid (t).

[0153] The convolution layer parameters are: kernel size 3×3, stride 1, padding 1, activation function ReLU. The pooling layer parameters are max pooling, window size 2×2, stride 2. Convolution calculation formula:

[0154] F conv (t)=ReLU(Conv(I(t),W)+b) (10)

[0155] Among them, I(t) is the input video frame, W is the convolution kernel weight, b is the bias, Conv represents the convolution operation, and ReLU is the activation function.

[0156] Pooling calculation formula:

[0157] F pool (t) = MaxPool(F conv (t)) (11)

[0158] Among them, F pool (t) is the feature map after pooling, and MaxPool represents the maximum pooling operation.

[0159] Furthermore, the AIS data feature extraction uses the long short-term memory network (LSTM) to process the AIS data and extract the temporal dynamic features. Assuming that the input AIS data is A(t), after passing through the LSTM layer, the AIS feature F is obtained. AIS (t). The calculation formula of LSTM unit is expressed as

[0160] i t =σ(W i ·[h t-1 ,A(t)]+b i ) (12)

[0161] f t =σ(W f ·[h t-1 ,A(t)]+b f ) (13)

[0162] o t =σ(W o ·[h t-1 ,A(t)]+b o ) (14)

[0163]

[0164] h t =o t *tanh(C t ) (17)

[0165] Where A(t) represents the AIS (Automatic Identification System) data input at time t. AIS data contains the ship's location information, speed, heading, and other dynamic information, which is used to extract time series features in the LSTM network. t-1 Represents the hidden state at time t-1, which represents the LSTM unit's memory of information from previous time steps. This hidden state contains the comprehensive information of all previous time steps and is used to calculate the input gate, forget gate, output gate, and candidate cell states in the current time step t. t represents the input gate activation value at time t, indicating how the current AIS data A(t) affects the cell state C t The input gate controls the contribution of new input to the cell state update. t Represents the activation value of the forget gate at time t, indicating the activation of the previous time step C t-1 The degree to which the cell state is retained. The forget gate determines which past information will be forgotten or retained. t Represents the output gate activation value at time t, controlling the cell state C t How to affect the final hidden state h tThe output gate determines the extent of influence from the cell state to the hidden state. Represents the candidate cell state at time t, which is based on the current input A(t) and the hidden state h of the previous time step t-1 It represents the possible updates of the cell state in the absence of a gating mechanism. t-1 Represents the cell state at time t-1, which contains the accumulated information passed down by the LSTM unit from past time steps. This state is combined with the forget gate f t and input gate i t The information is used to generate the cell state C at the current time step t . C t Represents the cell state at time t, which is the core memory of the LSTM unit and is used to store information related to the current task. It combines the cell state C of the previous time step t-1 and the candidate cell states at the current time step h t represents the hidden state at time t, which represents the output of the LSTM unit at time t. This is based on the current cell state C t and output gate o t The calculated value is used as the final output of LSTM and is passed to the next time step. i ,W f ,W o ,W C represents the weight matrix, which is used to calculate the input gate, forget gate, output gate and candidate cell state. These weights are optimized through the training process to accurately reflect the temporal dynamic characteristics in AIS data. i ,b f ,b o ,b C Represents the bias vector, which corresponds to the calculation of the input gate, forget gate, output gate and candidate cell state. The bias is used to adjust the activation value of each gate to help the model fit the data better. σ represents the sigmoid activation function, which is applied to the calculation of the input gate, forget gate and output gate. The output value is between [0,1] and controls the degree of information passing. Tanh represents the tanh activation function, which is applied to the calculation of the candidate cell state and the final hidden state. The output value is between [-1,1] and is used to generate a richer state representation.

[0166] Radar data feature extraction uses convolutional neural networks (CNN) to process radar data and extract its spatial features. Assuming the input radar data is R(t), after three layers of convolution and pooling operations, the feature map F is obtained. rad (t). The convolution and pooling layer parameters are the same as those used in video data feature extraction. Convolution calculation formula:

[0167] Fconv_rad (t) = ReLU(Conv(R(t),W rad )+b rad ) (18)

[0168] Where R(t) is the input radar data, W rad is the convolution kernel weight, b rad For bias.

[0169] Pooling calculation formula:

[0170] F pool_rad (t) = MaxPool(F conv_rad (t)) (19)

[0171] Among them, F pool_rad (t) is the feature map after pooling, and MaxPool represents the maximum pooling operation.

[0172] Meteorological data feature extraction uses a two-layer multi-layer perceptron (MLP) to process meteorological data and extract key features. Assuming that the input meteorological data is W(t), after two layers of full connection operations, the meteorological feature F is obtained. met (t).

[0173] Among them, the first layer is fully connected:

[0174] F fc1 (t) = ReLU(W fc1 ·W(t)+b fc1 ) (20)

[0175] Where W(t) is the input meteorological data, W fc1 is the weight, b fc1 For bias.

[0176] The second layer is fully connected:

[0177] F met (t) = ReLU(W fc2 ·F fc1 (t)+b fc2 ) (twenty one)

[0178] Among them, F fc1 (t) is the first layer output feature, W fc2 is the second layer weight, b fc2 For bias.

[0179] Furthermore, the attention mechanism is used to align the features of different modalities. Let the attention weight be α and the aligned feature be F aligned (t).

[0180] Among them, the attention weight calculation is:

[0181] α vid =softmax(W a1 )·F vid (t)) (22)

[0182] Among them, W a1 is the attention weight matrix of video features, and softmax is the softmax function.

[0183] α AIS =softmax(W a2 ·F AIS (t)) (23)

[0184] Among them, W a2 is the attention weight matrix of AIS features.

[0185] α rad =softmax(W a3 )·F rad (t)) (24)

[0186] Among them, W a3 is the attention weight matrix of radar features.

[0187] α met =softmax(W a4 ·F met (t)) (25)

[0188] Among them, W a4 is the attention weight matrix of meteorological features.

[0189] Feature alignment and fusion:

[0190] F aligned (t) = α vid ·F vid (t)+α AIS ·F AIS (t)+α rad ·F rad (t)+α met ·F met (t) (26)

[0191] Among them, F aligned (t) is the aligned multimodal feature.

[0192] Furthermore, a bidirectional long short-term memory network (BiLSTM) is used to perform temporal modeling on the fusion features, extract their temporal dynamic information, and generate the final environmental perception result E(t).

[0193] E(t)=BiLSTM(F aligned (t)) (27)

[0194] Among them, BiLSTM represents the bidirectional long short-term memory network operation, and E(t) is the environmental perception result at time t.

[0195] S3. Based on the final environmental perception results, a variety of sea conditions and weather scenarios are generated using a generative adversarial network to conduct risk assessment and generate a risk map. Based on the risk assessment results and the risk map, an avoidance strategy is generated to adjust the route to avoid high-risk areas.

[0196] Specifically, a generator network (Generator) G and a discriminator network (Discriminator) D are constructed. The generator is used to generate simulated sea conditions and weather scenarios, and the discriminator is used to determine whether the generated scenarios are realistic.

[0197] The input of the generator G is the noise vector z and the multimodal fusion feature F aligned (t), the output is the generated simulation scenario G(z,F aligned (t)).

[0198] The generator network structure is a multi-layer perceptron (MLP) network, and the input is the noise vector z and the fusion feature F aligned (t), after being processed by the fully connected layer and activation function, a simulation scenario is generated.

[0199] G(z,F aligned (t)) = ReLU(W g ·[z,F aligned (t)]+b g ) (28)

[0200] Among them, W g is the weight matrix of the generator, b g is the bias vector.

[0201] The input of the discriminator D is the real scene X real and generate scenarios G(z,F aligned (t)), the output is the judgment result D(X real ,G(z,F aligned (t))).

[0202] The discriminator network structure is a convolutional neural network (CNN). The input is scene data, which passes through convolutional layers and pooling layers to output the discrimination results.

[0203] D(X)=σ(Conv(X,W d )+b d ) (29)

[0204] Where X is the input scenario, W d is the weight matrix of the discriminator, b d is the bias vector, and σ is the sigmoid activation function.

[0205] Furthermore, an alternating training method is used to train the generator G and the discriminator D separately.

[0206] Among them, the loss function of the generator G is:

[0207]

[0208] Among them, p z (z) represents the noise distribution, p data (F aligned ) represents the distribution of multimodal fusion features.

[0209] Among them, the loss function of the discriminator D is:

[0210]

[0211] Among them, p data (X real ) represents the distribution of the real scenario.

[0212] After training, the generator G can generate a variety of possible sea conditions and weather scenarios. By inputting different noise vectors z and multimodal fusion features F aligned (t), generate multiple simulation scenarios G(z i ,F aligned (t)).

[0213] {G(z i ,F aligned (t))|i=1,2,…,N} (32)

[0214] Where N is the number of generated scenarios, z i are different noise vectors.

[0215] Furthermore, for each generated simulation scenario G(z i ,F aligned (t)) Risk assessment. Each simulation scenario represents a possible navigation environment, including different sea states, weather conditions and the dynamics of surrounding ships.

[0216] In order to accurately assess the risk of each simulation scenario, a multidimensional risk assessment model M is used. risk , which not only considers the traditional physical distance, heading and speed, but also introduces a dynamic risk factor λ based on historical data dynamic and a nonlinear term γ based on the complexity of the environment env , making risk assessment more targeted.

[0217] R i =M risk (G(z i ,F aligned (t)))+λ dynamic ·HistoryFactor(G(z i ,F aligned (t)))+γ env ·EnvComplexity(G(z i ,F aligned (t))) (33)

[0218] Among them, M risk Represents the basic risk assessment model, based on basic factors such as physical distance, speed, and heading. dynamic Represents the historical dynamic risk factor, which is based on the risk pattern extracted from historical data under similar sailing conditions. env It represents the nonlinear term of environmental complexity, and evaluates the complexity of the environment under the current scenario (such as multiple ship intersections, bad weather, etc.). HistoryFactor represents the risk pattern weight under similar conditions in historical data.

[0219] Among them, HistoryFactor reflects the historical risk under similar navigation conditions. When calculating, it is necessary to find the historical database that is consistent with the current scenario G(z i ,F aligned (t)) the most matching historical scenario and weighted according to the average risk value of these scenarios. Calculation formula:

[0220]

[0221] Among them, k is the number of historical scenarios selected, w j is the weight of the jth historical scenario that matches the current scenario (determined by the degree of matching), R h ist (j) is the risk score of historical scenario j.

[0222] Numerical example: Assume that the current scenario matches 5 historical scenarios, and the matching weights are 0.3, 0.2, 0.2, 0.2, and 0.1 respectively. The risk scores of the historical scenarios are 4, 3, 5, 4, and 2 respectively. Then the HistoryFactor is:

[0223] HistoryFactor=0.3×4+0.2×3+0.2×5+0.2×4+0.1×2=3.8 (35)

[0224] Among them, EnvComplexity represents the measure of environmental complexity, taking into account the diversity and complexity in the environment, such as ship density, frequency of weather changes, etc.

[0225] Among them, EnvComplexity quantifies the complexity of the current navigation environment, taking into account the combined impact of factors such as ship density, sea state changes, and weather conditions. Calculation formula:

[0226] EnvComplexity(G(z i ,F aligned (t)))=α·ρ+β·Δsea+γ·WeatherVar (36)

[0227] Among them, ρ is the ship density (the number of ships in a unit sea area), Δsea is the amplitude of sea condition change (such as wave height change), WeatherVar is the frequency of weather change, and α, β, γ are the corresponding weight coefficients. Assume that there are 8 ships in the current sea area (ρ=8), the wave height change is 2 meters (Δsea=2), the weather changes twice per hour (WeatherVar=2), and the weight coefficients are 0.5, 0.3, and 0.2 respectively, then EnvComplexity is:

[0228] EnvComplexity=0.5×8+0.3×2+0.2×2=4.6 (37)

[0229] Furthermore, the risk scores of all generated scenarios are comprehensively analyzed to obtain the comprehensive risk score R under the overall navigation environment. total In the calculation process, a dynamic weighting factor ω is introduced i , which adjusts for the probability of occurrence and possible severity of the scenario, so that the overall risk score not only reflects the average risk but also pays special attention to high-risk scenarios.

[0230]

[0231] Where N is the number of scenarios generated. i is the dynamic weighting factor of scenario i, reflecting the probability of scenario occurrence and the severity of risk. Numerical example: Assuming there are three scenarios with risk scores of 5, 3, and 4, respectively, with occurrence probabilities of 0.5, 0.3, and 0.2, respectively, and their risk severity weights of 1.2, 1.0, and 1.5, respectively, then the comprehensive risk score R total for:

[0232] R total =(0.5×1.2)×5+(0.3×1.0)×3+(0.2×1.5)×4=4.2 (39)

[0233] The comprehensive risk score not only provides an overall risk assessment, but is also used to generate a risk map. The risk map marks high-risk areas and displays the risk levels of different areas through color gradients, providing intuitive information support for subsequent path optimization.

[0234] Furthermore, an avoidance strategy is formulated based on the comprehensive risk score and the generated risk map. The generation of the avoidance strategy is not only based on the comprehensive risk score, but also introduces a path smoothing term α smooth and the path adjustment cost term β cost , ensuring the new path is safe while avoiding unnecessary frequent adjustments, improving the stability and economy of navigation, as shown below:

[0235]

[0236] Smoothness (P) represents the path smoothness measure, ensuring that the path does not have too many sharp turns and frequent changes. The calculation formula is:

[0237]

[0238] Among them, θ j is the heading angle of the j-th waypoint on path P, and m is the total number of waypoints.

[0239] Numerical example: Assume that there are 4 waypoints on the path, and their heading angles are 30°, 45°, 50°, and 40° respectively. The smoothness (P) is:

[0240] Smoothness(P)=|45-30|+|50-45|+|40-50|=25° (42)

[0241] AdjustmentCost (P) represents the cost of path adjustment, which evaluates the cost (such as fuel consumption, time delay, etc.) required to adjust from the current path to the new path. AdjustmentCost evaluates the cost of adjusting from the current path to the new path, taking into account factors such as additional fuel consumption and time delay. Calculation formula:

[0242] AdjustmentCost(P) = δ fuel FuelCost(P)+δ time ·TimeDelay(P) (43)

[0243] Among them, FuelCost(P) is the additional fuel consumption caused by adjusting the route, TimeDelay(P) is the time delay caused by the route adjustment, and δ fuel and δ timeis the corresponding weight coefficient. Numerical example: Assume that adjusting the route will increase fuel consumption by 100 liters and cause a time delay of 30 minutes. The fuel weight is 2 and the time delay weight is 1.5. Then the route adjustment cost AdjustmentCost(P) is:

[0244] AdjustmentCost(P)=2×100+1.5×30=245 (44)

[0245] S4. Based on the results of risk assessment, combined with real-time environmental perception data and historical navigation data, the navigation route is dynamically adjusted using a reinforcement learning algorithm to obtain an optimized navigation route.

[0246] Specifically, define the state s t , represents the environmental status of the ship at time t, including position, speed, heading, environmental perception results, meteorological data and information of surrounding ships.

[0247] s t = {pos t ,vel t ,heading t ,E(t),weather t ,ships t} (45)

[0248] Among them, pos t For location, vel t For speed, heading t is the heading, E(t) is the environmental perception result, weather t For weather data, ships t Information about surrounding vessels.

[0249] Define action a t , represents the navigation decision of the ship at time t, including the steering angle and speed adjustment.

[0250] a t = {steer t ,accel t} (46)

[0251] Among them, steer t is the steering angle, accel t Adjust for speed.

[0252] Reward function R(s t ,a t ) is used to evaluate the effect of each state-action pair, encouraging safe and efficient navigation, including:

[0253] Safety Reward Rsafety :

[0254]

[0255] Among them, d i is the distance to the ith surrounding ship, ∈ is a small constant to prevent the denominator from being zero, and n is the number of surrounding ships.

[0256] Efficiency Reward R efficiency :

[0257]

[0258] Among them, vel desired is the expected speed.

[0259] Risk penalty R risk :

[0260] R risk (s t ,a t )=R total (s t ) (49)

[0261] Among them, R total (s t ) is the comprehensive risk score calculated in step 3.

[0262] Total reward function:

[0263] R(s t ,a t )=λ1·R safety (s t ,a t )+λ2·R efficiency (s t ,a t )-λ3·R risk (s t ,a t ) (50)

[0264] Among them, λ1, λ2, λ3 are weight coefficients.

[0265] Furthermore, using the policy gradient method in deep reinforcement learning, we design a policy network π(a t |s t ; θ), where θ is the parameter of the policy network. The input of the policy network is the state s t , the output is action a t The probability distribution of .

[0266] π(a t |s t ; θ) = softmax(f(st ;θ)) (51)

[0267] Among them, f(s t ; θ) represents the forward propagation process of the policy network, and softmax is the softmax function.

[0268] Furthermore, we design a value network V(s t ; φ), where φ is the parameter of the value network, used to estimate the state s t The input of the value network is the state s t , the output is the value of the state V(s t ).

[0269] V(s t ; φ) = g(s t ;φ) (52)

[0270] Among them, g(s t ; φ) represents the forward propagation process of the value network.

[0271] Furthermore, the policy gradient method is used to optimize the policy network π(a t |s t ;θ) parameter θ.

[0272] Policy gradient calculation:

[0273]

[0274] Among them, J(θ) is the policy objective function, A t is the advantage function and T is the time step.

[0275] Advantage function calculation:

[0276] A t =R(s t ,a t )+γV(s t+1 ;φ)-V(s t ;φ) (54)

[0277] Among them, γ is the discount factor.

[0278] Furthermore, using the value network V(s t ;φ) optimizes the strategy networkπ(a t |s t ; θ) parameter φ. The loss function of the value network is:

[0279]

[0280] Furthermore, in actual navigation, the optimized strategy network π(at |s t ; θ) to optimize the path. According to the real-time environment perception result E(t) and the current state s t , sample action a from the policy network t , adjust the ship's route.

[0281] a t ~π(a t |s t ;θ)(56)

[0282] Update Status t , repeat the above process, continuously optimize the path, and ensure the safety and efficiency of navigation.

[0283] S5. The optimized navigation routes, high-risk areas and important reminder information will be displayed through real-time synchronization of AR display and navigation system, combined with augmented reality technology overlay.

[0284] Specifically, the AR display system includes AR glasses or head-mounted displays (HMD), which are closely integrated with the ship's multimodal data collection and path optimization system. AR equipment must work normally in various harsh marine environments and support functions such as waterproof, dustproof, and anti-glare. In addition, AR equipment should have high resolution, wide viewing angle, and low latency to ensure real-time and clear display of navigation information.

[0285] Furthermore, data is obtained in real time from the path optimization system and multimodal data fusion module, and a dedicated low-latency wireless network protocol is used to ensure efficient data transmission and real-time update. An adaptive delay compensation model is used to dynamically adjust the refresh frequency of the displayed content according to the transmission delay to ensure the stability and smoothness of the display:

[0286] ν(t)=ν max ·e -ατ(t) (57)

[0287] Where ν(t) represents the refresh rate of the current display content. max Indicates the maximum refresh rate allowed by the system. α is a tuning parameter that controls the sensitivity to latency. τ(t) is the current network latency.

[0288] Furthermore, data from multimodal data collection (step 1), including video, AIS, radar, and meteorological data, will be fused through a deep learning model (step 2) to generate a unified environmental perception result F aligned (t). This result has already been used for risk assessment in the GAN-generated simulation scenario (step 3) and will now be further applied in the AR display.

[0289] Specifically, the environmental perception results F generated by deep learning and GANaligned (t) and the comprehensive risk score R total , generating AR display content. These contents include optimized routes, safe areas and risk area prompts. A dynamic information weighting model is used to dynamically adjust the display priority of each information layer to ensure that key information such as risk warnings and route guidance are always at the core:

[0290]

[0291] Among them, I display (t) indicates the display content at time t. i (t) represents the weight of the i-th type of information, which is dynamically adjusted based on risk assessment and path optimization. i (t) represents the i-th category of information, such as routes, safety areas, risk warnings, etc.

[0292] Further, the best route P from the reinforcement learning path optimization (step 4) is new The path is displayed in AR in the form of a dynamic path. The path display combines color coding (green for safety, red for high risk) and dynamically adjusts the transparency based on the multimodal fusion results to ensure that high-risk areas can attract the crew's attention.

[0293] Furthermore, environmental information (such as weather, sea conditions, and surrounding ship dynamics) is superimposed from the multimodal data fusion module (step 2) and displayed in the AR field of view in a transparent manner. Through layered display technology, less important information (such as weather changes) is placed at the edge to ensure that key navigation information is always clearly visible.

[0294] Furthermore, to ensure the smoothness and information transparency of the path display, a joint optimization formula is introduced to dynamically adjust the visual effect of the path display by combining the environmental complexity and the comprehensive risk score:

[0295]

[0296] Among them, λ display1 and λ display2 They represent the adjustment parameters of path smoothness and information transparency, θ j Represents the heading angle of the j-th waypoint on path P. Transparency(R i ) represents the risk area R i The transparency of the information is adjusted dynamically according to the risk level.

[0297] Furthermore, for specific navigation scenarios, a risk accumulation term δ is added to the path display. risk , used to record the cumulative risk values ​​that occurred in similar environments in history, helping crew members to predict potential dangerous areas in advance:

[0298]

[0299] Among them, γ env is the cumulative risk factor.

[0300] Furthermore, based on the real-time multimodal environment perception E(t), the AR display content is dynamically updated to ensure the real-time and accuracy of the information. The content is updated based on the real-time data from steps 2 and 3, as well as the latest results of the reinforcement learning path optimization.

[0301] Gesture recognition: Combined with the crew's operation behavior identified in the path optimization module, gesture recognition is implemented. Crew members can adjust the display view or switch information layers through gestures such as sliding and clicking.

[0302] Voice interaction: Based on the deep learning speech recognition model, crew members can control the AR display content through voice commands, such as "show risk area", "adjust route", etc. Voice interaction directly interacts with the data of the route optimization module.

[0303] Furthermore, the interactive response function χ(t) is introduced into the dynamic interactive design to adjust the system response speed according to the crew's operation frequency and environmental changes:

[0304]

[0305] Among them, κ represents the reference value of response speed. β represents the adjustment parameter, the sensitivity of the control system to the operating frequency. σ represents the current operating frequency. σ0 represents the set operating frequency reference value.

[0306] Furthermore, the ambient light data obtained from the multimodal data acquisition module will be used to automatically adjust the brightness and contrast of the AR display to adapt to different lighting conditions. In particular, in the case of strong light reflection or insufficient light, the readability of the displayed content can be improved through dynamic adjustment.

[0307] Furthermore, to ensure the clarity of displayed content under various lighting conditions, a brightness adjustment formula based on ambient light intensity is designed:

[0308]

[0309] Among them, B display (t) represents the display brightness at time t. base Indicates the reference brightness value, applicable to normal lighting conditions. ambient (t) represents the current ambient light intensity. L opt Represents the ideal ambient light intensity. ∈ represents the adjustment parameter to smooth the brightness change.

[0310] Furthermore, combined with the image enhancement algorithm, strong light reflection areas are automatically identified and processed to ensure that the contrast and color saturation of these areas are effectively improved. The visual enhancement function is combined with the environmental simulation results generated by GAN, so that high-risk areas can be accurately identified under any lighting conditions.

[0311] Furthermore, in response to the special needs of complex lighting conditions at sea, an ambient light adaptive regularization term ρ is designed. env , add this item to the brightness adjustment formula to ensure stability in strong reflection or low contrast conditions:

[0312]

[0313] Among them, ρ env Represents the ambient light adaptation coefficient. Indicates the gradient change rate of ambient light, reflecting sudden changes in lighting.

[0314] Furthermore, the AR display system is integrated with all previously developed modules (multimodal data acquisition and preprocessing, deep learning data fusion, GAN environment simulation, and reinforcement learning path optimization) to ensure the consistency of data flow and the real-time processing.

[0315] It is connected to the core module of the ship navigation system through a unified interface, and uses standardized APIs for data interaction and synchronization of display content.

[0316] Furthermore, actual navigation tests were conducted to verify the performance of the system in a dynamic environment. The AR display content and interaction methods were further optimized by collecting operational feedback from crew members. Based on the test data, the algorithm parameters of each module were tuned to ensure the stability and reliability of the system under different sea conditions and lighting conditions.

[0317] Furthermore, the performance of the AR display system is monitored in real time, especially in high-risk scenarios. Combined with the feedback from the path optimization module, potential problems can be detected and handled in a timely manner to ensure the stability of the system during long-term navigation.

[0318] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0319] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the parameter operation method provided in any embodiment of the present invention, and will not be repeated here.

[0320] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0321] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0322] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0323] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time ship navigation route planning method based on augmented reality, It is characterized in that The method comprises: S1. Real-time collection of various environmental data around the ship, and pre-processing of the various environmental data; the various environmental data include video data, AIS data, radar data and meteorological data; S2. Process video data through convolutional neural networks to extract environmental features, process AIS data through long short-term memory networks to extract dynamic features, process radar data through convolutional neural networks to extract its spatial features, use two-layer multi-layer perceptrons to process meteorological data to extract meteorological features, use attention mechanisms to align features of different modalities and fuse them to obtain fused features, and then use bidirectional long short-term memory networks to perform time series modeling on the fused features to generate the final environmental perception results; S3. Based on the final environmental perception results, a variety of sea conditions and weather scenarios are generated using a generative adversarial network to conduct risk assessment and generate a risk map. Based on the risk assessment results and the risk map, an avoidance strategy is generated to adjust the route to avoid high-risk areas. S4. Based on the results of risk assessment, combined with real-time environmental perception data and historical navigation data, the navigation route is dynamically adjusted using a reinforcement learning algorithm to obtain an optimized navigation route; S5. The optimized navigation route, high-risk areas and important reminder information are displayed through real-time synchronization of AR display and navigation system, combined with augmented reality technology overlay; Among them, based on the results of risk assessment and risk map, avoidance strategies are generated to adjust routes to avoid high-risk areas, including: According to the risk assessment results and risk map, the path smoothing term α is introduced smooth and the path adjustment cost term β cost , the production avoidance strategy is expressed as follows: Among them, P new represents the new avoidance strategy, R total (P) represents the risk score of the old avoidance strategy, Smoothness(P) represents the path smoothness measure, and AdjustmentCost represents the cost of adjusting from the current path to the new path; The path smoothness metric is expressed as follows: Among them, θ j represents the heading angle of the jth waypoint on path P, and m represents the total number of waypoints; The cost of adjusting from the current path to the new path is expressed as follows: AdjustmentCost(P)=δ fuel ·FuelCost(P)+δ time ·TimeDelay(P) Among them, FuelCost(P) represents the additional fuel consumption caused by adjusting the route, TimeDelay(P) represents the time delay caused by the route adjustment, and δ fuel and δ time Represents the corresponding weight coefficient.

2. A real-time ship navigation route planning method based on augmented reality according to claim 1, It is characterized in that The preprocessing includes multimodal data time synchronization processing, video data preprocessing, AIS data preprocessing, radar data preprocessing and meteorological data preprocessing; wherein, the video data preprocessing includes denoising processing and frame extraction; the AIS data preprocessing includes deduplication and filtering processing and data interpolation processing; the radar data preprocessing includes echo denoising and data gridding processing; the meteorological data preprocessing includes data smoothing processing and data standardization processing.

3. A real-time ship navigation route planning method based on augmented reality according to claim 1, It is characterized in that The video data is processed by convolutional neural network to extract environmental features, which are expressed as follows: F conv (t)=ReLU(Conv(I(t),W)+b) Where I(t) represents the input video frame, W represents the convolution kernel weight, b represents the bias, Conv represents the convolution operation, ReLU represents the activation function, and F conv (t) represents the output feature of the current convolutional layer; The pooling calculation formula of the convolutional neural network is expressed as follows: F pool (t)=MaxPool(F conv (t)) Among them, F pool (t) is the feature map after pooling, and MaxPool represents the maximum pooling operation; The AIS data is processed by the long short-term memory network to extract dynamic features, which are expressed as follows: i t =σ(W i ·[h t-1 ,A(t)]+b i ) f t =σ(W f ·[h t-1 ,A(t)]+b f ) o t =σ(W o ·[h t-1 ,A(t)]+b o ) h t =o t *tanh(C t ) Where A(t) represents the input AIS data, C t Indicates the cell state, h t represents the hidden state, b i , b f , b o , b C represents the bias vector, σ represents the sigmoid activation function, tanh represents the tanh activation function, and h t-1 represents the hidden state at time t-1, C t-1 represents the cell state at t-1, represents the candidate cell state at the current time step, i t represents the input gate activation value at time t, W i , W f , W o , W C represents the weight matrix, o t represents the output gate activation value at time t, f t Represents the activation value of the forget gate at time t; The radar data is processed by convolutional neural network to extract its spatial features, which are expressed as follows: F conv_rad (t)=ReLU(Conv(R(t),W rad )+b rad ) Where R(t) represents the input radar data, W rad represents the convolution kernel weight, b rad represents bias, ReLU represents activation function, F conv_rad (t) represents the output of the current convolutional neural network; The pooling calculation formula is: F pool_rad (t)=MaxPool(F conv_rad (t)) Among them, Fpoolrad(t) represents the feature map after pooling, and MaxPool represents the maximum pooling operation; The two-layer multi-layer perceptron is used to process meteorological data and extract meteorological features, which are expressed as follows: The first layer is fully connected: F fc1 (t)=ReLU(W fc1 ·W(t)+b fc1 ) Where W(t) represents the input meteorological data, W fc1 represents the weight, b fc1 Indicates bias, F fc1 (t) represents the output of the first fully connected layer; The second layer is fully connected: F met (t)=ReLU(W fc2 ·F fc1 (t)+b fc2 )) Among them, F fc1 (t) represents the output feature of the first layer, W fc2 represents the second layer weight, b fc2 Indicates bias, F met (t) represents the output of the second fully connected layer.

4. A real-time ship navigation route planning method based on augmented reality according to claim 3, It is characterized in that The attention mechanism is used to align the features of different modalities and fuse them to obtain fused features, and then a bidirectional long short-term memory network is used to perform temporal modeling on the fused features to generate the final environmental perception result, which is expressed as follows: Let the attention weight be α and the aligned feature be F aligned (t), then the attention weight is calculated as: α vid =softmax(W a1 ·F vid (t)) Among them, α vid represents the video attention weight, W a1 is the attention weight matrix of video features, and softmax is the softmax function; α AIS =softmax(W a2 ·F AIS (t)) Among them, α AIS represents the AIS attention weight, W a2 is the attention weight matrix of AIS features; α rad =softmax(W a3 ·F rad (t)) Among them, α rad represents the radar attention weight, W a3 is the attention weight matrix of radar features; α met =softmax(W a4 ·F met (t))) Among them, α met represents the meteorological attention weight, W a4 is the attention weight matrix of meteorological features; Feature alignment and fusion: F aligned (t)=a vid ·F vid (t)+a AIS ·F AIS (t)+a rad ·F rad (t)+a met ·F met (t) Among them, F vid (t) represents the video feature map, F AIS (t) represents the AIS feature, F rad (t) represents the radar characteristic graph, F met (t) represents meteorological characteristics, F aligned (t) is the aligned multimodal feature; A bidirectional long short-term memory network is used to perform temporal modeling on the fusion features, extract temporal dynamic information, and generate the final environmental perception result E(t), which is expressed as follows: E(t)=BiLSTM(F aligned (t)) Among them, BiLSTM represents the bidirectional long short-term memory network operation, and E(t) is the environmental perception result at time t.

5. The method for real-time ship navigation route planning based on augmented reality according to claim 1, It is characterized in that The structure of the generative adversarial network is constructed as follows: The input of the generator G is the noise vector z and the multimodal fusion feature F aligned (t), the output is the generated simulation scenario G(z, F aligned (t)), where the network structure of the generator G is a multi-layer perceptron network; The input of the discriminator D is the real scene X real and generate scenarios G(z, F aligned (t)), the output is the judgment result D(X real , G(z, F aligned (t))), where the network structure of the discriminator D is: convolutional neural network; Use the alternating training method to train the generator G and the discriminator D separately; the loss function of the generator G is expressed as follows: Among them, p z (z) represents the noise distribution, p data (F aligned ) represents the distribution of multimodal fusion features, z represents the noise vector, and E represents the expectation; The loss function of the discriminator D is expressed as follows Among them, p data (X real ) represents the distribution of the real scenario; According to the input of different noise vectors z and multimodal fusion features F aligned (t), generate multiple simulation scenarios G(z i , F aligned (t)).

6. A real-time ship navigation route planning method based on augmented reality according to claim 5, It is characterized in that Introduced a dynamic risk factor λ based on historical data dynamic and a nonlinear term γ based on the complexity of the environment env For each generated simulation scenario G(z i , F aligned (t)) to conduct a risk assessment, expressed as follows: R i =M risk (G(z i ,F aligned (t)))+λ dynamic ·HistoryFactor(G(z i ,F aligned (t)))+γ env ·EnvComplexity(G(z i ,F aligned (t))) Among them, M risk represents the basic risk assessment model; dynamic represents the historical dynamic risk factor; γ env represents the nonlinear term of environmental complexity; HistoryFactor represents the risk pattern weight in historical data; wherein, the HistoryFactor reflects the historical risk under similar navigation conditions, and the calculation requires the comparison between the historical database and the current scenario G(z i , F aligned (t)) are the best matching historical scenarios and are weighted according to the average risk value of these scenarios, expressed as follows: Where k represents the number of selected historical scenarios, w j represents the weight of the jth historical scenario that matches the current scenario, R hist (j) represents the risk score of historical scenario j; The risk scores of all generated scenarios are comprehensively analyzed to obtain the comprehensive risk score R under the overall navigation environment. total , which is expressed as follows: Among them, ω i represents the dynamic weighting factor, and N represents the number of generated scenarios.

7. The method for real-time ship navigation route planning based on augmented reality according to claim 1, It is characterized in that The S4 specifically includes: S401, define state s t , represents the environmental state of the ship at time t; define action a t , represents the navigation decision of the ship at time t; S402, reward function R(s t , a t ) is used to evaluate the effect of each state-action pair, where the reward function R(s t , a t ) including safety bonus R safety , Efficiency Reward R efficiency and risk penalty R risk ; S403, using the policy gradient method, design a policy network π(a t |s t ; θ), where θ is the parameter of the policy network, and the input of the policy network is the state s t , the output is action a t The probability distribution of S404、Design a value network V(s t ; φ), where φ is the parameter of the value network, used to estimate the state s t The value of the network, where the input of the value network is the state s t , the output is the value of the state V(s t ); S405, using the policy gradient method to optimize the policy network π (a t |s t ;θ) parameters θ, using the value network V(s t ;φ) optimizes the strategy networkπ(a t |s t ; θ) parameter φ; S406, using the optimized strategy network π(a t |s t ; θ) performs real-time path optimization and uses the real-time environment perception result E(t) and the current state s t , sample action a from the policy network t , adjust the ship's route.

8. A real-time ship navigation route planning method based on augmented reality according to claim 7, It is characterized in that The reward function R(s t , a t ) including safety bonus R safety , Efficiency Reward R efficiency and risk penalty R risk , which has the following representation: Safety Reward R safety (s t , a t ): Among them, d i represents the distance to the i-th surrounding ship, ∈ represents a small constant to prevent the denominator from being zero, and n represents the number of surrounding ships; Efficiency Reward R efficiency (s t , a t ): Among them, vel desired represents the expected speed, vel t Indicates actual speed; Risk penalty R risk (s t , a t ): R risk (s t ,a t )=R total (s t ) Among them, R total (s t ) represents the comprehensive risk score; The total reward function R(s t , a t ): R(s t ,a t )=λ 1 ·R safety (s t ,a t )+λ 2 ·R efficiency (s t ,a t )-l 3 ·R risk (s t ,a t ) Among them, λ 1 ,λ 2 ,λ 3 Represents the weight coefficient.

9. The method for real-time ship navigation route planning based on augmented reality according to claim 1, It is characterized in that In the real-time synchronization between AR display and navigation system, a joint optimization formula is introduced to dynamically adjust the visual effect of path display by combining environmental complexity and comprehensive risk score, which is expressed as follows: Among them, λ display1 and λ display2 They represent the adjustment parameters of path smoothness and information transparency, θ j represents the heading angle of the j-th waypoint on path P; Transparency (R i ) represents the risk area R i The transparency of the information is adjusted dynamically according to the risk level.

10. The method for real-time ship navigation route planning based on augmented reality according to claim 1, It is characterized in that For specific navigation scenarios, the risk accumulation term δ is designed risk , used to record the cumulative risk values ​​that occurred in similar environments in history, to help crew members predict potential dangerous areas in advance, as shown below: Among them, γ env is the cumulative risk factor.

Images