Deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion

Abstract

The invention relates to a deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion, and aims to solve the technical problem that the global convergence of a navigation result in the prior art is easily influenced by an environment state space. The invention provides a deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion. The deep reinforcement learning underwater autonomous navigation method comprises the following steps: step 1, constructing a training data set; 2, constructing a DQN decision model by fusing the dynamic geomagnetic map with deep reinforcement learning; 3, dynamically adjusting the exploration rate to update the moving direction of the aircraft until the aircraft reaches the target position; the method does not need a prior geomagnetic chart, is suitable for an unknown marine environment, and realizes reinforcement learning navigation and autonomous homing without the prior geomagnetic chart.

Description

Technical Field

[0001] The present invention relates to an autonomous navigation method for underwater vehicles, and in particular to a deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion. Background Art

[0002] Autonomous underwater vehicles (AUVs) require efficient autonomous navigation when conducting deep-sea exploration and resource surveys. Traditional navigation methods based on geomagnetic matching usually rely on prior geomagnetic maps. However, due to the limitations of actual measurements and the variability of the geomagnetic field in the marine environment, traditional methods have certain limitations in navigation accuracy and reliability.

[0003] To avoid the traditional methods' reliance on prior geomagnetic maps, existing technologies often employ autonomous underwater vehicles (AUVs) combined with deep reinforcement learning (DQN) algorithms to achieve autonomous path planning through interaction with the environment. In navigation path optimization tasks, the AUV continuously learns and improves the driving strategy from the current position to the target location, thereby approximating the optimal path. However, when the environmental state space is large or complex, the DQN algorithm alone is prone to falling into local minima, which affects the global convergence of the navigation results. Summary of the Invention

[0004] The purpose of the present invention is to solve the technical problem that the global convergence of existing navigation results is easily affected by the environmental state space, and to provide a deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion.

[0005] To achieve the above objectives, the technical solutions provided by the present invention are as follows:

[0006] A deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion is special in that it includes the following steps:

[0007] Step S1: Build a training dataset

[0008] S1.1. Define the compound reward function;

[0009] S1.2. Initialize the starting position parameters and target position parameters of the spacecraft, determine the state space data set based on the starting position parameters and target position parameters; determine the action space data set based on the spacecraft structure;

[0010] S1.3. Construct a training dataset based on the composite reward function, the state space dataset, and the action space dataset.

[0011] S1.1-S1.2 have no order and can be performed simultaneously or sequentially;

[0012] Step S2: Build a DQN decision model

[0013] S2.1. Obtain the geomagnetic parameters of the current location through a dynamic geomagnetic map construction method based on the fusion of the IGRF model and the CNN network;

[0014] S2.2. Construct an initial DQN decision model and initialize navigation parameters of the DQN decision model; the navigation parameters include an initial exploration rate;

[0015] S2.3. Based on the training data set obtained in S1.3, the geomagnetic parameters of the current position obtained in S2.1, and the initial exploration rate obtained in S2.2, the DQN decision model is iteratively trained until the DQN decision model meets the convergence conditions, thereby obtaining an optimized DQN decision model.

[0016] Step S3: Navigation

[0017] S3.1. Given the starting position parameters and target position parameters of the spacecraft, use the starting position parameters of the spacecraft as the current position parameters; given the initial exploration rate of the DQN decision model, use the initial exploration rate as the current exploration rate;

[0018] S3.2. Based on the current position parameters, target position parameters, current exploration rate, and the geomagnetic parameters of the current position obtained by the dynamic construction method of the geomagnetic map based on the fusion of the IGRF model and the CNN network, the optimized DQN decision model obtained in S2.3 is used to output the current navigation strategy. After the vehicle navigates to the next position according to the current navigation strategy, the current position parameters and current exploration rate are obtained.

[0019] S3.3. Calculate the distance from the current position of the aircraft to the target position. When the distance is less than a first threshold, update the current exploration rate by using discounted decay. When the distance is greater than or equal to the first threshold, maintain the current exploration rate.

[0020] S3.4. Calculate the difference between the geomagnetic parameters of the current position of the aircraft and the geomagnetic parameters of the target position. When the difference meets the second threshold, it is considered that the aircraft has reached the target position and the navigation ends. Otherwise, return to step S3.2 until the difference between the geomagnetic parameters of the current position of the aircraft and the geomagnetic parameters of the target position meets the second threshold and the navigation ends.

[0021] Furthermore, in step S2.1, the calculation steps for obtaining the geomagnetic parameters of the current location based on the dynamic construction method of the geomagnetic map fused with the IGRF model and the CNN network are as follows:

[0022] S2.1.1. Define a circular sampling area with a radius of r around the current position of the aircraft, set n sampling points at preset angular intervals, and uniformly collect geomagnetic data from the n sampling points, where 1 km ≤ r ≤ 5 km and n ≥ 1;

[0023] S2.1.2. Calculate the latitude, longitude, and altitude of the sampling point using a spherical coordinate system;

[0024] S2.1.3. Calculate the geomagnetic parameters B of the sampling points corresponding to the geomagnetic data according to the IGRF model IGRF ;

[0025] S2.1.4. The latitude, longitude and altitude of the sampling point after standardization according to the CNN model, as well as the geomagnetic parameter B of the sampling point IGRF Perform end-to-end learning and use mean square error (MSE) as the loss function to calculate the geomagnetic parameters of unknown areas.

[0026] S2.1.5, the magnetic parameter B of the sampling point IGRF and geomagnetic parameters of unknown regions According to the adaptive weight fusion, the final geomagnetic parameter estimation is formed, and the final geomagnetic parameter estimation is determined as the geomagnetic parameter B at the current position. final :

[0027]

[0028] Where α is the fusion coefficient.

[0029] Furthermore, after step S2.1.5, the following steps are also included:

[0030] S2.1.6. The geomagnetic parameter B of the current position obtained according to step S2.1.5 final , using the exponential moving average strategy, gradually construct a high-precision global geomagnetic field map. The process is as follows:

[0031] M t+1 [i, j] = β·M t [i,j]+(1-β)·B final

[0032] Where β is the forgetting factor, M is the global geomagnetic map matrix, (i, j) is the two-dimensional grid position coordinate, and M t [i, j] is the geomagnetic value of the two-dimensional grid (i, j) at the tth iteration, M t+1 [i, j] is the geomagnetic value of the two-dimensional grid (i, j) at the t+1th iteration.

[0033] Furthermore, in step S3.3, the calculation formula for updating the current exploration rate ε(t) by discount decay is as follows:

[0034] ε(t)=max(ε min +(ε initial -ε min )·γ t ,εclose )

[0035] Where, ε min is the minimum exploration rate, ε initial is the initial exploration rate, γ t is the exploration rate attenuation factor, ε close is the threshold close to the target exploration rate, ε close ≤0.01;

[0036] The exploration rate adjustment mechanism is as follows: the exploration rate decays exponentially, the initial exploration rate is greater than the minimum exploration rate, when the distance between the aircraft and the target position is less than the first threshold, the discount decay is increased to reduce the exploration rate to give priority to the learned strategy, when the distance between the aircraft and the target position is greater than or equal to the first threshold, the current exploration rate is maintained to give priority to the random exploration strategy.

[0037] Furthermore, in S1.1, the compound reward function R total Defined as:

[0038] R total =R direction +R distance +R efficiency

[0039] Where R direction is the direction reward, R distance is the distance reward, R efficiency The compound reward function comprehensively considers direction guidance, distance convergence and path efficiency to guide the vehicle to move along the optimal path, where the direction reward R direction The calculation formula is as follows:

[0040] R direction =λ1·cos(θ goal -θ)

[0041] Where λ1 is the direction reward weight, θ goal is the target direction, θ is the current heading angle of the vehicle;

[0042] Distance Reward R distance The calculation formula is as follows:

[0043] R distance =λ2·(d t -d t+1 )

[0044] Where λ2 is the distance reward weight, d t is the distance from the initial position of the spacecraft to the target position, d t+1 is the updated distance;

[0045] Path efficiency reward R efficiencyThe calculation formula is as follows:

[0046] R efficiency =-λ3·I((x t+1 ,y t+1 )∈V)

[0047] Where λ3 is the penalty reward weight, v is the set of visited locations, I is the indicator function, and x t+1 ,y t+1 is the current position coordinate of the aircraft.

[0048] Furthermore, the state space dataset in S1.2 includes the current position coordinates of the aircraft, the target position coordinates, the geomagnetic parameter values, and the Euclidean distance from the current position to the target position; the action space dataset includes 360 discrete steering angles of the aircraft, and each action corresponds to a steering increment of 1°.

[0049] Furthermore, the navigation parameters in S2.2 also include a preset step size, a learning rate, a discount factor, a minimum exploration rate, and a decay factor.

[0050] Beneficial effects of the present invention:

[0051] 1. This invention uses deep reinforcement learning to conduct underwater autonomous navigation based on dynamic geomagnetic map fusion. By dynamically adjusting the reinforcement learning strategy, the vehicle can autonomously achieve efficient homing navigation without a prior geomagnetic map. This solves the technical problem that the global convergence of existing navigation results is easily affected by the environmental state space.

[0052] 2. The present invention is based on a deep reinforcement learning underwater autonomous navigation method that integrates dynamic geomagnetic maps. It uses a dynamic geomagnetic map construction method based on the IGRF model and CNN network fusion. By collecting local geomagnetic data in real time, it provides real-time environmental status support for navigation strategies.

[0053] 3. This method, based on deep reinforcement learning and underwater autonomous navigation integrated with dynamic geomagnetic maps, defines a composite reward function that integrates direction reward, distance reward, and path efficiency reward functions. This solves the problems of traditional single reward functions in deep reinforcement learning, which are prone to falling into local optimality, slow convergence, and path oscillation during training. The method achieves improved training convergence speed, smoother and shorter paths, and significantly improved navigation success rates.

[0054] 4. The present invention is based on a deep reinforcement learning underwater autonomous navigation method fused with dynamic geomagnetic maps. It adopts a dynamic geomagnetic map construction method based on the fusion of the IGRF model and the CNN network. By collecting local geomagnetic data in real time and using the spatial feature learning ability of CNN to predict the geomagnetic distribution of unknown areas, a high-precision global geomagnetic field map is gradually constructed. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of the deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion of the present invention;

[0056] Figure 2 Schematic diagram of the DQN decision model structure in step S2 of an embodiment of a method for underwater autonomous navigation based on deep reinforcement learning and dynamic geomagnetic map fusion according to the present invention;

[0057] Figure 3 Schematic diagram of the action space in step S1 of an embodiment of the method for underwater autonomous navigation based on deep reinforcement learning and dynamic geomagnetic map fusion according to the present invention;

[0058] Figure 4 This is a diagram showing the autonomous navigation experiment results of an embodiment of the deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion according to the present invention;

[0059] Figure 5 This is a diagram showing the autonomous navigation results of an embodiment of the underwater autonomous navigation method based on deep reinforcement learning and dynamic geomagnetic map fusion in a single abnormal area of the present invention;

[0060] Figure 6 This is a diagram showing the autonomous navigation results of an embodiment of the underwater autonomous navigation method based on deep reinforcement learning and dynamic geomagnetic map fusion in multiple abnormal areas of the present invention;

[0061] Figure 7 In step S3 of the embodiment of the deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion of the present invention, a reward curve graph under different exploration rates is selected;

[0062] Figure 8 This is a comparison diagram of the paths of an embodiment of the deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion of the present invention and the paths of pure DQN, simulated annealing (SA) algorithm and genetic algorithm (EA) under the same navigation task;

[0063] Figure 9 This is a comparison chart of the path lengths of an embodiment of the deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion of the present invention, pure DQN, simulated annealing (SA) algorithm, and genetic algorithm (EA) under the same navigation task;

[0064] Figure 10 This is a graph showing the convergence of the objective function of the embodiment of the deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion in the navigation process of the present invention, pure DQN, simulated annealing (SA) algorithm, and genetic algorithm (EA);

[0065] Figure 11This is a comparison chart of rewards between an embodiment of the deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion and a pure DQN in the same navigation task;

[0066] Figure 12 This is a prediction error diagram of the geomagnetic map in a spatial area in an embodiment of the deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described below with reference to the accompanying drawings and examples.

[0068] This embodiment provides a deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion, such as Figure 1 As shown, the specific implementation steps are as follows:

[0069] Step S1: construct a training dataset based on the composite reward function, the state space dataset, and the action space dataset;

[0070] Step S1.1, define a composite reward function. During navigation, a vehicle must not only approach the target but also avoid falling into a local optimum in a complex environment. Therefore, a composite reward function is defined that comprehensively considers direction guidance, distance convergence, and path efficiency, thereby guiding the vehicle along the optimal path. This solves technical problems in reinforcement learning, such as the tendency of traditional single reward functions to fall into local optimums, slow convergence, path oscillation, and redundancy during training. This results in improved training convergence speed, smoother and shorter paths, and significantly improved navigation success rates. Define a composite reward function.

[0071] R total =R direction +R distance +R efficiency

[0072] Where R direction is the direction reward, R distance is the distance reward, R efficiency Reward for path efficiency;

[0073] The direction reward encourages the vehicle to move towards the target position. By calculating the angle between the current direction of the vehicle and the target direction, the direction reward can be obtained. The target direction θ goal It is defined as the great circle heading angle from the current position to the target position (i.e. the angle from the north direction to the target direction clockwise), calculated using the spherical trigonometry formula:

[0074] θ goal =arctan2(coslat g ·sin(lon g -lon t ),coslatt ·sinlat g -sinlat t ·coslat g ·cos(lon g -lon t ))

[0075] Where, (lat t ,lon t ) is the current latitude and longitude coordinates of the aircraft, (lat g ,lon g ) are the latitude and longitude coordinates of the target location;

[0076] The current direction is the current heading angle θ of the vehicle, and the direction reward is calculated by the cosine similarity of the angle between the two:

[0077] R direction =λ1·cos(θ goal -θ)

[0078] Where λ1 is the direction reward weight, θ goal is the target direction, θ is the current heading angle of the vehicle;

[0079] The distance reward is obtained by calculating the distance difference between the current position of the spacecraft and the target position. The reward value encourages the spacecraft to shorten the distance to the target position. The calculation formula is as follows:

[0080] R distance =λ2·(d t -d t+1 )

[0081] Where λ2 is the distance reward weight, d t is the Euclidean distance from the current position of the spacecraft to the target position, d t+1 is the updated distance;

[0082] The path efficiency reward takes into account the complexity of the path to reduce unnecessary detours. The calculation formula for the path efficiency reward is as follows:

[0083] R efficiency =-λ3·I((x t+1 ,y t+1 )∈V)

[0084] Where λ3 is the penalty reward weight, V is the set of visited locations, I is the indicator function, the condition is 1, otherwise it is 0, x t+1 ,y t+1 is the current position coordinate of the vehicle. The penalty term makes the agent tend to explore new areas and improve path efficiency;

[0085] Step S1.2, as Figure 3 As shown, (x t ,y t ) is the current position coordinate, (x g ,y g ) is the target position coordinate, initializing the starting position parameters and target position parameters of the spacecraft, and determining the state space data set according to the starting position parameters and the target position parameters;

[0086] The state space data set includes the current position coordinates of the spacecraft, the target position coordinates, the geomagnetic parameter values, and the Euclidean distance from the current position to the target position. It is defined as follows:

[0087] S=[(lat t ,lon t ),(lat g ,lon g ),(F,D,I),d t ]

[0088] Where F, D, and I are the total magnetic field strength (nT), magnetic declination (°), and magnetic inclination (°) at the current position of the spacecraft, respectively. t is the Euclidean distance from the current position of the aircraft to the target position, calculated using the longitude and latitude great circle distance formula:

[0089] d t =R·arccos(sinlat t ·sinlat g +coslat t ·coslat g ·cos(lon g -lon t ))

[0090] Where R is the average radius of the Earth, lat t ,lon t ,lat g ,lon g Need to convert to radians before substituting into calculations;

[0091] Step S1.3, determining an action space dataset; the action space dataset includes 360 discrete steering angles of the aircraft, and each action corresponds to a 1° steering increment;

[0092] Step S1.4: Construct a training dataset based on the composite reward function, the state space dataset, and the action space dataset.

[0093] Step S2: Figure 2As shown in the figure, a DQN decision model is constructed; to stabilize the deep reinforcement learning training process, a dual network architecture is adopted; the target network regularly copies the current network parameters to calculate the target Q value to avoid frequent fluctuations in the target value during training:

[0094] Q target =r+γmax a′ Q next (s′,a′,θ′ g )

[0095] Where r is the immediate reward, γ = 0.95 is the discount factor, which is used to balance the importance of current rewards and future rewards, s′ is the next state, a′ is the next action, and θ′ g is the target network parameter, and then the current network parameter is updated by minimizing the mean square error MSE loss function:

[0096] L=E[(Q target -Q pred (s,a,θ c )) 2 ]

[0097] Where Q pred is the Q value prediction of the current network for state and action, s is the current state, a is the current action, θ c The current network parameters are back-propagated through the Adam optimizer. This objective function enables the network to learn the optimal strategy considering long-term benefits in each iteration.

[0098] The specific construction steps are as follows:

[0099] Step S2.1, obtaining the geomagnetic parameters of the current location by a dynamic geomagnetic map construction method based on the fusion of the International Geomagnetic Reference Field Model (IGRF) and a convolutional neural network (CNN);

[0100] Step S2.1.1: Define a circular sampling area with a radius of 2 km around the current position of the aircraft, set 16 sampling points at 22.5° intervals, and evenly collect geomagnetic data from the 16 sampling points;

[0101] Step S2.1.2, the latitude, longitude and altitude of the sampling point are determined using a spherical coordinate system;

[0102] Step S2.1.3: Calculate the geomagnetic parameters B of the sampling points corresponding to the geomagnetic data according to the IGRF model. IGRF ;

[0103] Step S2.1.4: The latitude, longitude and altitude of the sampling point after standardization according to the CNN model, the geomagnetic parameter B of the sampling point IGRFPerform end-to-end learning to calculate the geomagnetic parameters of unknown areas by minimizing the mean square error (MSE) as the loss function

[0104] Step S2.1.5: Set the magnetic parameter B at the sampling point IGRF and geomagnetic parameters of unknown regions According to the adaptive weight fusion, the final geomagnetic parameter estimate is formed. The final geomagnetic parameter estimate is the geomagnetic parameter B at the current position. final :

[0105]

[0106] Where α is the fusion coefficient.

[0107] Step S2.2: Construct an initial DQN decision model and initialize the navigation parameters of the DQN decision model. The navigation parameters include a preset step size ΔL = 1 km, a learning rate α = 0.001, a discount factor γ = 0.95, and the discount factor is used to balance the importance of current rewards and future rewards. The exploration rate is initially set to 1, the exploration rate decay factor is dynamically adjusted by 0.998, and the minimum exploration rate is 0.01.

[0108] Step S2.3, iteratively train the DQN decision model based on the training data set obtained in step S1, the geomagnetic parameters of the current location obtained in step S2.1, and the initial exploration rate until the DQN decision model meets the convergence conditions, thereby obtaining an optimized DQN decision model;

[0109] Step S3: Based on the optimized DQN decision model, guide the vehicle to navigate according to the current strategy output by the DQN decision model;

[0110] Step S3.1: Given the starting position parameters and target position parameters of the spacecraft; using the starting position parameters of the spacecraft as the current position parameters; giving the initial exploration rate of the DQN decision model as 1; using the initial exploration rate as the current exploration rate;

[0111] Step S3.2: Based on the current position parameters, target position parameters, current exploration rate, and the geomagnetic parameters of the current position obtained by the dynamic construction method of the geomagnetic map based on the fusion of the IGRF model and the CNN network, the optimized DQN decision model outputs the current navigation strategy. After the vehicle navigates according to the current navigation strategy, the current position parameters and current exploration rate are obtained.

[0112] Step S3.3: Calculate the distance from the current position of the vehicle to the target position. When the distance is less than the first threshold, update the exploration rate by discount decay. When the distance is greater than or equal to the first threshold, maintain the current exploration rate. The exploration rate will decay exponentially as the navigation progresses to reduce random exploration and increase reliance on the learned strategy. The exploration rate update formula is:

[0113] ε(t)=max(ε min +(ε initial -ε min )·γ t ,ε close )

[0114] Where, ε min is the minimum exploration rate, ε initial is the initial exploration rate, γ t is the exploration rate attenuation factor, ε close is the threshold close to the target exploration rate, ε close ≤0.01;

[0115] The exploration rate adjustment mechanism is as follows: the exploration rate decays exponentially, and the initial exploration rate is greater than the minimum exploration rate. When the distance between the aircraft and the target position is less than a first threshold, the exploration rate is reduced by adding discount decay to give priority to the learned strategy. When the distance between the aircraft and the target position is greater than or equal to the first threshold, the current exploration rate is maintained to give priority to the random exploration strategy.

[0116] Step S3.4: Calculate the difference between the geomagnetic parameters of the current position of the aircraft and the geomagnetic parameters of the target position. When the difference meets the second threshold, it is considered that the aircraft has reached the target position and the navigation process ends. Otherwise, return to step S3.2 until the difference between the geomagnetic parameters of the current position of the aircraft and the geomagnetic parameters of the target position meets the second threshold, and the navigation process ends.

[0117] In this embodiment, to determine the decision output and movement direction of the vehicle, its objective function is calculated using the following steps:

[0118] Step S4.1: Define the geomagnetic field parameter set as a multidimensional vector:

[0119] B={B1,B2,···,B n}

[0120] Where B1, B2, ... B n is the parameter element of the magnetic field;

[0121] Step S4.2: The geomagnetic parameter B at the target location t The geomagnetic parameter B at the point k The difference between any corresponding parameter elements in is used as the objective function.

[0122]

[0123] Where, is the value of the i-th geomagnetic parameter at the target location, The value of the i-th geomagnetic parameter at the current moment;

[0124] Step S4.3: Obtain the geomagnetic parameter B at the target location t The geomagnetic parameter B at the point k The target search function corresponding to each parameter element in is normalized as follows:

[0125]

[0126] Where, is the geomagnetic parameter value at the starting position;

[0127] Step S4.5: The position of the aircraft is updated according to the selected action (steering angle). Assuming that the current latitude and longitude of the aircraft is (lat t ,lon t ), the selected action is angle θ″, then the new position is:

[0128] lat t+1 =lat t +ΔL·cos(θ″)

[0129] lon t+1 =lon t +ΔL·sin(θ″)

[0130] Where ΔL is the step length of the vehicle.

[0131] To achieve environmental perception and navigation coordination for underwater vehicles without prior maps, this paper also proposes a method for dynamically constructing geomagnetic maps based on the fusion of the International Geomagnetic Reference Field (IGRF) model and a convolutional neural network (CNN). This method collects local geomagnetic data in real time, uses the spatial feature learning capability of CNN to predict the geomagnetic distribution of unknown areas, and gradually constructs a high-precision global geomagnetic field map to provide real-time environmental status support for navigation strategies. The specific construction process is as follows:

[0132] Step S5.1, define a circular sampling area with a radius of r around the current position of the aircraft, set n sampling points at preset angle intervals, and uniformly collect geomagnetic data from the n sampling points, where 1 km ≤ r ≤ 5 km and n ≥ 1;

[0133] Step S5.2: The latitude, longitude and altitude of the sampling point are determined using a spherical coordinate system;

[0134] Step S5.3: Calculate the geomagnetic parameters B of the sampling points corresponding to the geomagnetic data according to the IGRF model. IGRF ;

[0135] Step S5.4: The latitude, longitude and altitude of the sampling point after standardization according to the CNN model, the geomagnetic parameter B of the sampling pointIGRF Perform end-to-end learning and use mean square error (MSE) as the loss function to calculate the geomagnetic parameters of unknown areas

[0136] Step S5.5: The magnetic parameter B of the sampling point IGRF and geomagnetic parameters of unknown regions According to the adaptive weight fusion, the final geomagnetic parameter estimate is formed. The final geomagnetic parameter estimate is the geomagnetic parameter B at the current position. final :

[0137]

[0138] Where α is the fusion coefficient.

[0139] Step S5.6: Based on the geomagnetic parameter B at the current location final , using the exponential moving average strategy, a high-precision global geomagnetic field map can be gradually constructed. The process is as follows:

[0140] M t+1 [i, j] = β·M t [i,j]+(1-β)·B final

[0141] Where β is the forgetting factor, M is the global geomagnetic map matrix, t is the time, (i, j) is the two-dimensional grid position coordinate, and M t [i, j] is the geomagnetic value of the two-dimensional grid (i, j) at the tth iteration, M t+1 [i, j] is the geomagnetic value of the two-dimensional grid (i, j) at the t+1th iteration. The global geomagnetic map matrix M is used to store the geomagnetic field intensity value (in nT) of each grid point in the environment. All elements of the matrix are empty at the beginning and are dynamically updated as the spacecraft moves.

[0142] like Figure 4 Figure 2 shows the results of a navigation simulation experiment using an embodiment of the present invention in a uniform geomagnetic field environment. The experimental environment is divided into a 100×100 two-dimensional grid area. The geomagnetic distribution in the simulated area is stable and has no significant anomalies. The geomagnetic parameters are generated using the IGRF model. The vehicle starts from the starting position and gradually explores the environment ahead with a step size of 1 km. In each iteration, the vehicle collects geomagnetic data within r=2 km and in n=16 directions (every 22.5°) around it. The CNN model is used to predict the geomagnetic field of unvisited areas, and the IGRF results are integrated to construct a dynamic geomagnetic map. During the navigation process, the DQN decision model comprehensively considers direction guidance, distance convergence, and repeated exploration penalties through a reward function, outputs the current optimal turning action, and successfully plans a path that gradually converges to the target point.

[0143] like Figure 5 and Figure 6 As shown, the simulation experiment results of the embodiment of the present invention in the area with geomagnetic anomaly interference are shown; Figure 5 For single anomaly area testing, the anomaly source is set to the left of the target point; Figure 6 For the multi-anomaly area test, multiple anomaly sources are distributed in the middle and right side of the path. In both cases, the vehicle can identify the high-error geomagnetic features corresponding to the anomaly areas in real time and correct the geomagnetic map through the CNN model. At the same time, the DQN strategy automatically adjusts the heading to avoid the anomaly areas and finally successfully approaches the target point. The results verify the adaptive robustness of the proposed method under non-ideal conditions.

[0144] like Figure 7 The figure shows the average reward change curve in step S3 of an embodiment of the present invention when different initial exploration rates (ε=0.5, 0.7, 0.8) are selected; each group of experiments runs 100 rounds under the same initial state, and the average reward value obtained in each round is plotted. It can be observed that when the dynamic adjustment mechanism is used, the learning efficiency of the DQN decision model is significantly improved and the convergence speed is accelerated, verifying the positive effect of the dynamic exploration rate adjustment mechanism on improving training efficiency and navigation performance.

[0145] To further verify the effectiveness of the method of the present invention, Figures 8 to 11 Demonstrates performance comparisons between the embodiments of the present invention and mainstream comparison algorithms in different experimental scenarios;

[0146] Figure 8 The following is a path comparison chart comparing the path results of our method with those of pure DQN, simulated annealing (SA), and genetic algorithms (EA) for the same navigation task. As can be seen from the chart, the path generated by our method is the shortest and smoothest, effectively avoiding areas of geomagnetic anomalies. Meanwhile, the pure DQN path fluctuates in some areas, the SA path fluctuates widely and lacks stability, and the EA path exhibits significant fallback and redundancy issues.

[0147] Figure 9 Furthermore, the path length was used as the evaluation index, and the average path lengths of the four algorithms were statistically analyzed in 10 groups of random tasks. The results showed that the method of the present invention had the shortest average path length in all tasks, which was significantly better than other methods.

[0148] Figure 10 The convergence curve of the objective function during the navigation process is shown. It can be seen that the objective function fluctuates greatly in the early exploration stage. As the iteration proceeds, the DQN decision model gradually converges to the optimal strategy, and the objective function value drops rapidly and stabilizes at a low level, indicating that the vehicle has successfully approached the target area.

[0149] Figure 11The changing trend of the average reward is given. The composite reward function structure designed by the present invention can effectively guide learning. The average reward value increases rapidly and remains stable during training, which is significantly better than other algorithms.

[0150] Figure 12 This is the prediction error curve of the geomagnetic map in the spatial area. It can be seen from the figure that as the training progresses, the RMSE gradually decreases, indicating that the difference between the model's geomagnetic prediction results and the actual observation values decreases, and the model fits better. This shows that the model gradually reduces the prediction error and improves the prediction accuracy during the learning process.

[0151] To further quantify the prediction performance, Table 1 compares the three key metrics of the proposed method for geomagnetic map prediction: RMSE, PSNR, and PPMCC, with those of kriging interpolation, bicubic interpolation, and gradient boosting regression. The results show that the proposed method achieves the lowest RMSE (4.1835), the highest PSNR (32.624), and a PPMCC (0.9925) close to 1, demonstrating its optimal accuracy and relevance, providing high-quality geomagnetic information support for navigation.

[0152]

[0153] Table 1

[0154] In summary, the adaptive deep reinforcement learning geomagnetic navigation method proposed in the present invention realizes efficient autonomous homing of underwater vehicles without the need for prior maps, solving the problem that existing technologies rely on prior geomagnetic maps and are prone to falling into local optimality.

[0155] Any content not described in detail in this specification belongs to the prior art known to those skilled in the art. The above examples are provided for illustrative purposes only and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. All equivalent substitutions and modifications that do not depart from the spirit and principles of the present invention are intended to be within the scope of the present invention.

Claims

1. A deep reinforcement learning underwater autonomous navigation method based on dynamic geomagnetic map fusion, characterized in that: The following steps are involved: Step S1: Build a training dataset S1.

1. Define the compound reward function; S1.

2. Initialize the starting position parameters and target position parameters of the spacecraft, determine the state space data set based on the starting position parameters and target position parameters; determine the action space data set based on the spacecraft structure; S1.

3. Construct a training dataset based on the composite reward function, the state space dataset, and the action space dataset. Step S2: Build a DQN decision model S2.

1. Obtain the geomagnetic parameters of the current location through a dynamic geomagnetic map construction method based on the fusion of the IGRF model and the CNN network; S2.

2. Construct an initial DQN decision model and initialize navigation parameters of the DQN decision model; the navigation parameters include an initial exploration rate; S2.

3. Based on the training data set obtained in S1.3, the geomagnetic parameters of the current position obtained in S2.1, and the initial exploration rate obtained in S2.2, the DQN decision model is iteratively trained until the DQN decision model meets the convergence conditions, thereby obtaining an optimized DQN decision model. Step S3: Navigation S3.

1. Given the starting position parameters and target position parameters of the spacecraft, use the starting position parameters of the spacecraft as the current position parameters; given the initial exploration rate of the DQN decision model, use the initial exploration rate as the current exploration rate; S3.

2. Based on the current position parameters, target position parameters, current exploration rate, and the geomagnetic parameters of the current position obtained by the dynamic construction method of the geomagnetic map based on the fusion of the IGRF model and the CNN network, the optimized DQN decision model obtained in S2.3 is used to output the current navigation strategy. After the vehicle navigates to the next position according to the current navigation strategy, the current position parameters and current exploration rate are obtained. S3.

3. Calculate the distance from the current position of the aircraft to the target position. When the distance is less than a first threshold, update the current exploration rate by using discounted decay. When the distance is greater than or equal to the first threshold, maintain the current exploration rate. S3.

4. Calculate the difference between the geomagnetic parameters of the current position of the aircraft and the geomagnetic parameters of the target position. When the difference meets the second threshold, it is considered that the aircraft has reached the target position and the navigation ends. Otherwise, return to step S3.2 until the difference between the geomagnetic parameters of the current position of the aircraft and the geomagnetic parameters of the target position meets the second threshold and the navigation ends.

2. The method for underwater autonomous navigation based on deep reinforcement learning and dynamic geomagnetic map fusion according to claim 1 is characterized in that: In step S2.1, the calculation steps for obtaining the geomagnetic parameters of the current location based on the dynamic construction method of the geomagnetic map fused with the IGRF model and the CNN network are as follows: S2.1.

1. Define a circular sampling area with a radius of r around the current position of the aircraft, set n sampling points at preset angular intervals, and uniformly collect geomagnetic data from the n sampling points, where 1 km ≤ r ≤ 5 km and n ≥ 1; S2.1.

2. Calculate the latitude, longitude, and altitude of the sampling point using a spherical coordinate system; S2.1.

3. Calculate the geomagnetic parameters B of the sampling points corresponding to the geomagnetic data according to the IGRF model IGRF ; S2.1.

4. The latitude, longitude and altitude of the sampling point after standardization according to the CNN model, as well as the geomagnetic parameter B of the sampling point IGRF Perform end-to-end learning and use mean square error (MSE) as the loss function to calculate the geomagnetic parameters of unknown areas. S2.1.5, the magnetic parameter B of the sampling point IGRF and geomagnetic parameters of unknown regions According to the adaptive weight fusion, the final geomagnetic parameter estimation is formed, and the final geomagnetic parameter estimation is determined as the geomagnetic parameter B at the current position. final : Where α is the fusion coefficient.

3. The method for underwater autonomous navigation based on deep reinforcement learning and dynamic geomagnetic map fusion according to claim 2 is characterized in that: After step S2.1.5, the following steps are also included: S2.1.

6. The geomagnetic parameter B of the current position obtained according to step S2.1.5 final , using the exponential moving average strategy, gradually construct a high-precision global geomagnetic field map. The process is as follows: M t+1 [i,j]=β·M t [i,j]+(1-β)·B final Where β is the forgetting factor, M is the global geomagnetic map matrix, (i, j) is the two-dimensional grid position coordinate, and M t [i, j] is the geomagnetic value of the two-dimensional grid (i, j) at the tth iteration, M t+1 [i, j] is the geomagnetic value of the two-dimensional grid (i, j) at the t+1th iteration.

4. The method for underwater autonomous navigation based on deep reinforcement learning and dynamic geomagnetic map fusion according to claim 3 is characterized by: In step S3.3, the calculation formula for updating the current exploration rate ε(t) by discount decay is as follows: ε(t)=max(ε min +(e initial -e min )·c t ,he close ) Where, ε min is the minimum exploration rate, ε initial is the initial exploration rate, γ t is the exploration rate attenuation factor, ε close is the threshold close to the target exploration rate, ε close ≤0.

01.

5. The method for underwater autonomous navigation based on deep reinforcement learning and dynamic geomagnetic map fusion according to claim 4 is characterized in that: In S1.1, the compound reward function R total Defined as: R total =R di re ction +R distance +R efficiency Where R direction is the direction reward, R distance is the distance reward, R efficiency Reward for path efficiency; Among them, the direction reward R direction The calculation formula is as follows: R direction =λ1·cos(θ goal -i) Where λ1 is the direction reward weight, θ goal is the target direction, θ is the current heading angle of the vehicle; Distance Reward R distance The calculation formula is as follows: R destance =λ2·(d t -d t+1 ) Where λ2 is the distance reward weight, d t is the distance from the initial position of the spacecraft to the target position, d t+1 is the updated distance; Path efficiency reward R efficiency The calculation formula is as follows: R efficiency =-λ3·I((x t+1 ,y t+1 )∈V) Where λ3 is the penalty reward weight, V is the set of visited locations, I is the indicator function, and x t+1 ,y t+1 is the current position coordinate of the aircraft.

6. The method for underwater autonomous navigation based on deep reinforcement learning and dynamic geomagnetic map fusion according to claim 5 is characterized by: The state space data set in S1.2 includes the current position coordinates of the spacecraft, the target position coordinates, the geomagnetic parameter values, and the Euclidean distance from the current position to the target position; The action space dataset consists of 360 discrete steering angles of the vehicle, with each action corresponding to a 1° steering increment.

7. The method for underwater autonomous navigation based on deep reinforcement learning and dynamic geomagnetic map fusion according to claim 6 is characterized in that: The navigation parameters in S2.2 also include preset step size, learning rate, discount factor, minimum exploration rate and decay factor.

Images