Amphibious vehicle path planning method and system based on fuzzy logic rule and reinforcement learning

Abstract

The invention relates to an amphibious vehicle path planning method and system based on fuzzy logic rules and reinforcement learning. The planning method comprises the following steps: collecting scene parameters by using a sensor according to an application scene of the amphibious vehicle, and taking the scene parameters as input parameters; performing fuzzification processing on the input parameters, performing reasoning by using a fuzzy rule base to obtain fuzzy output, and performing output defuzzification on the fuzzy output by using a gravity center method; defining a state space and an action space, resetting an environment, executing an action and calculating a reward, and finishing the step when a turn ending condition is met; an Actor-Critic neural network is initialized, and stable learning of a strategy is realized through experience playback and an updating mechanism of a target network; and training and optimizing the Actor-Critic neural network to obtain an optimal path planning strategy. According to the amphibious vehicle path planning method based on the fuzzy logic rule in combination with reinforcement learning, efficient solution of a path planning problem in a complex environment is realized through combination of a fuzzy logic system and a reinforcement learning algorithm, and the amphibious vehicle path planning method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to a path planning method and system for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning. Background Art

[0002] Currently, amphibious vehicles are increasingly used in complex environments. These vehicles need to switch freely between land and water, and navigate efficiently and safely in various terrain and water depth conditions. Traditional path planning methods usually rely on a lot of prior knowledge and manual intervention, and are difficult to adapt to complex and dynamically changing environments. In addition, amphibious vehicles may encounter various obstacles, slope changes, and water flow interference during driving, which puts higher requirements on path planning.

[0003] Existing path planning methods mainly focus on navigation in a single environment and lack effective response strategies in multiple complex environments. For example, although the rule-based path planning method is simple and direct, it is often not ideal when faced with uncertainty and dynamic changes in the environment. Although the path planning methods based on graph search algorithms such as the A* algorithm and the Dijkstra algorithm can find the optimal path, they have high computational complexity and are difficult to apply in real time in dynamic environments.

[0004] In recent years, the application of reinforcement learning methods in path planning has gradually attracted attention. Reinforcement learning can gradually optimize strategies through interaction with the environment, thereby achieving autonomous navigation in complex environments. However, traditional reinforcement learning methods are prone to slow convergence and long training time when faced with large state spaces. In addition, in a changing environment, the reinforcement learning algorithm has limited ability to handle noise and uncertainty, resulting in insufficient stability and reliability of the planned path.

[0005] In response to the above problems, fuzzy logic systems have been introduced into path planning due to their ability to handle uncertainty and fuzzy information. Fuzzy logic systems can effectively convert complex continuous input data into fuzzy values, reduce the dimension of the state space, thereby accelerating the convergence of the algorithm and improving the efficiency and robustness of path planning. However, a single fuzzy logic system still has some shortcomings when facing a complex environment, such as difficulty in handling dynamically changing environments and complex multi-variable inputs. Summary of the invention

[0006] In view of some or all of the problems in the prior art, the present invention provides a path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning. The path planning method combines the advantages of the fuzzy logic system and the reinforcement learning algorithm, preprocesses the input data through the fuzzy logic system, converts the complex continuous input data into fuzzy values, and then optimizes it through the Deep Deterministic Policy Gradient (DDPG) algorithm to solve the optimal path planning strategy. The path planning method includes the following steps:

[0007] According to the application scenario of the amphibious vehicle, using sensors to collect scenario parameters, and using the scenario parameters as input parameters;

[0008] The input parameters are fuzzified, reasoning is performed using a fuzzy rule base to obtain a fuzzy output, and the fuzzy output is defuzzified using a centroid method;

[0009] Define the state space and action space, reset the environment, execute actions and calculate rewards, and complete this step when the round end condition is met;

[0010] Initialize the Actor-Critic neural network and achieve stable learning of the strategy through experience replay and target network update mechanism; and

[0011] Train and optimize the Actor-Critic neural network to obtain the best path planning strategy.

[0012] Furthermore, the scene parameters include: speed V, acceleration A, water depth D, slope S, nearest obstacle position O, throttle position T rt , Steering wheel angle S a , braking force B, target position T p 、Target heading T h .

[0013] Further, the input parameters are fuzzified, reasoning is performed using a fuzzy rule base to obtain a fuzzy output, and the fuzzy output is defuzzified using a centroid method, including:

[0014] Convert the input parameters into fuzzy sets and describe the input parameters using membership functions;

[0015] Using the rules of the fuzzy rule base, reasoning on the input parameters to obtain a fuzzy output;

[0016] The fuzzy output includes a driving risk index, a path complexity index, a collision risk index, and an energy efficiency index;

[0017] The rules of the fuzzy rule base corresponding to the driving risk index are: if the speed is fast and the acceleration is high, the driving risk index is high; if the speed is medium and the acceleration is medium, the driving risk index is medium; if the speed is slow or the acceleration is low, the driving risk index is low; if the accelerator position is heavy and the steering wheel angle is left turn or right turn, the driving risk index is high; if the braking force is heavy braking, the driving risk index is high;

[0018] The rules of the fuzzy rule base corresponding to the path complexity index are: if the water depth is deep or the slope is steep, the path complexity index is high; if the water depth is medium or the slope is medium, the path complexity index is medium; if the water depth is shallow or the slope is gentle, the path complexity index is low;

[0019] The rules of the fuzzy rule base corresponding to the collision risk index are: if the nearest obstacle position is close and the speed is fast, the collision risk index is high; if the nearest obstacle position is medium and the speed is medium, the collision risk index is medium; if the nearest obstacle position is far and the speed is slow, the collision risk index is low; if the speed is medium and the acceleration is high and the nearest obstacle position is medium, the collision risk index is high; if the braking force is medium braking and the nearest obstacle position is close, the collision risk index is medium;

[0020] The rules of the fuzzy rule base corresponding to the energy efficiency index are: if the speed is fast, the target position is far, and the target heading is straight, the energy efficiency index is low; if the speed is medium, the target position is medium, and the target heading is straight, the energy efficiency index is medium; if the speed is slow, the target position is close, and the target heading is straight, the energy efficiency index is high; if the target heading is left yaw or right yaw, the energy efficiency index is medium.

[0021] The defuzzification formula is:

[0022]

[0023] Among them, I crisp is the index value after defuzzification, μ j (I) is the membership degree of the jth rule to index I; I j is the output value of index I corresponding to the jth rule, and the values ​​corresponding to low, medium and high indexes are 0, 0.5 and 1 respectively;

[0024] The index I is,

[0025] I∈{DRI,PCI,CRI,EEI}

[0026] Among them, DRI is the driving risk index, PCI is the path complexity index, CRI is the collision risk index, and EEI is the energy efficiency index;

[0027] Through the defuzzification formula, the defuzzified index value DRI is obtained crisp 、PCI crisp 、CRI crisp EEI crisp .

[0028] Further, defining the state space and action space, resetting the environment, executing actions and calculating rewards include:

[0029] Define the state space s as,

[0030] s={x,y,z,T,DRI crisp ,PCI crisp ,CRI crisp ,EEI crisp}

[0031] Among them, DRI crisp 、PCI crisp 、CRI crisp EEI crisp is the defuzzified index value, x, y, z are the position coordinates of the amphibious vehicle, and T is the terrain type;

[0032] Normalize the parameters contained in the state space s to obtain the specific state input s of the reinforcement learning algorithm t ,

[0033] s t ={x norm ,y norm ,z norm ,T norm ,DRI norm ,PCI norm ,CRI norm ,EEI norm}

[0034] Among them, x norm ,y norm 、z norm , T norm 、DRI norm 、PCI norm 、CRI norm EEI norm They are x, y, z, T, and DRI respectively. crisp 、PCI crisp 、CRI crisp EEI crisp The normalized value of ;

[0035] Define the action space a t for,

[0036]

[0037] in, is the throttle position, is the steering wheel angle, u B For braking force;

[0038] Initialize the environment state, set the initial position and various initial state parameters of the amphibious vehicle;

[0039] Update the environment state according to the current action space,

[0040] s t+1 =f(s t ,a t )

[0041] Among them, s t+1 is the updated state space, f is the environment state update function, a t is the action space;

[0042] Calculate the reward function R t ,

[0043] R t =R safe +R path +R energy +R goal

[0044] R safe , R path , R energy , R goal They are safety reward, path complexity reward, energy efficiency reward and goal completion reward.

[0045] Furthermore, the round is a complete interactive process in which the amphibious vehicle starts from the initial state in the environment and performs a series of actions until the termination condition is met; each round may include multiple time steps, and the amphibious vehicle completes a specific task through a series of state transitions and action selections;

[0046] The end of round conditions include:

[0047] When the amphibious vehicle successfully reaches the target location, the round ends.

[0048] T p ≤∈

[0049] Among them, ∈ is the tolerance distance of the target position;

[0050] When the nearest obstacle position O does not exceed the tolerance distance Omin , that is, a collision occurs and the round ends;

[0051] The time step number reaches the preset maximum time step number T max , the round ends.

[0052] Furthermore, the Actor-Critic neural network is initialized, and the stable learning of the strategy is achieved through experience replay and target network update mechanism, including:

[0053] Initialize the Actor network μ(s|θ μ ), μ(s|θ μ ) is used to calculate the current state s t Output control action a t ;

[0054] Initialize the Critic network Q(s,a|θ Q ), Q(s,a|θ Q ) is used to evaluate the state-action pair Q(s t ,a t )’s value;

[0055] Initialize the target network μ′(s|θ μ′ ) and Q′(s,a|θ Q′ ), the parameters are initialized to be the same as the main network, that is, θ μ′ =θ μ ,θ Q′ =θ Q ;

[0056] Initialize experience replay pool

[0057] From the current state t , through the Actor network μ(s t |θ μ ) Calculate action a t , and add noise Explore.

[0058]

[0059] Execute action a t , get the next state s t+1 、Instant Rewards t , termination sign done;

[0060] The experience (s t ,a t ,r t ,s t+1 , done) stored in the experience replay pool

[0061] Update the Critic network by minimizing the mean square error loss function.

[0062]

[0063] Where N is the size of the batch data, that is, the number of experiences sampled from the experience replay pool; y i is the target value of the Critic network,

[0064] y i =r i +γ1(1-done i )Q′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ )

[0065] Among them, γ1 is the discount factor, (s i ,a i ,r i ,s i+1 ,done i ) is a small batch of data sampled from the experience replay pool;

[0066] Update the Actor network using policy gradients:

[0067]

[0068] Where N is the size of the batch data, that is, the number of experiences sampled from the experience replay pool; The objective function J is the Actor network parameter θ μ The gradient of indicates the optimization direction of the Actor network; is the gradient of the action in the Critic network, representing the state-action pair (s i ,a)’s action value function Q(s i ,a) derivative of action a; is the strategy function μ(s) of the Actor network i |θ μ ) for network parameters θ μ The gradient of the Actor network for a given state s i The rate of change of action that should be chosen;

[0069] Update the target network parameters through the soft update mechanism,

[0070] θ μ′ ←τθ μ +(1-τ)θ μ′

[0071] θ Q′ ←τθ Q +(1-τ)θ Q′

[0072] Among them, τ is the soft update coefficient.

[0073] Furthermore, the Actor-Critic neural network is trained and optimized to obtain the best path planning strategy including:

[0074] Set the initial state s0 to initialize an empty experience replay pool Initialize neural network parameters;

[0075] In each training round, the state update, experience replay and target network update of multiple time steps are executed cyclically; the Actor network is updated according to the current state s t Output control action a t , the Critic network evaluates the value of each state-action pair and updates it by minimizing the loss function;

[0076] During the training process, define the cumulative reward R t ,

[0077]

[0078] Among them, T is the end time of the current round, γ2 is the discount factor, r k is the immediate reward at the kth time step;

[0079] The conditions for stopping training are:

[0080] When the number of training rounds reaches the maximum value E max When , the training stops;

[0081] Amphibious vehicles accumulate rewards R in a continuous round t Exceeding the predetermined threshold R target When , the training stops;

[0082] When training stops, the optimal path planning strategy is obtained.

[0083] The present invention also provides a path planning system for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning for the path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning, the system comprising the following modules:

[0084] An input parameter collection module is configured to use sensors to collect scenario parameters according to the application scenario of the amphibious vehicle and use the scenario parameters as input parameters;

[0085] A fuzzy logic system building module is configured to fuzzify the input parameters, perform reasoning using a fuzzy rule base to obtain a fuzzy output, and perform output defuzzification on the fuzzy output using a centroid method;

[0086] A reinforcement learning environment establishment module is configured to define the state space and action space, reset the environment, execute actions and calculate rewards, and complete this step when the round end condition is met;

[0087] The Deep Deterministic Policy Gradient algorithm module is configured to initialize the Actor-Critic neural network and achieve stable learning of the policy through experience replay and target network update mechanism; and

[0088] The training and optimization module is configured to train and optimize the Actor-Critic neural network to obtain the best path planning strategy.

[0089] The present invention also provides a computer system, comprising:

[0090] a processor configured to execute machine-readable instructions;

[0091] A graphics card with an artificial intelligence chip configured to train a path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning; and

[0092] The memory is configured to store machine-readable instructions, which, when executed by the processor and / or the graphics card, perform the steps of the path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning.

[0093] The present invention also provides a computer-readable storage medium having machine-readable instructions stored thereon, and when the machine-readable instructions are executed by a processor, the steps of the path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning are executed.

[0094] Compared with the prior art, the present invention has the following advantages:

[0095] 1. The path planning method for amphibious vehicles based on fuzzy logic rules combined with reinforcement learning proposed in the present invention is based on fuzzy logic rules and DDPG algorithm, combined with fuzzy logic system to pre-process input data, effectively process continuous input data in complex environments, and ensure efficient and safe navigation of amphibious vehicles in different terrains and waters.

[0096] 2. The present invention proposes a path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning. The established fuzzy logic system can quickly respond to environmental changes and provide real-time control instructions through fuzzy processing and rule reasoning, thereby improving the flexibility and adaptability of path planning.

[0097] 2. The present invention proposes a path planning method for amphibious vehicles based on fuzzy logic rules combined with reinforcement learning. The DDPG algorithm stabilizes the reinforcement learning process through experience replay and soft update of the target network, thereby improving the learning efficiency and the robustness of the model.

[0098] 4. The path planning method for amphibious vehicles based on fuzzy logic rules combined with reinforcement learning proposed in the present invention realizes efficient solution to path planning problems in complex environments through the combination of fuzzy logic system and reinforcement learning algorithm, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] To further illustrate the above and other advantages and features of various embodiments of the present invention, a more specific description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings only depict typical embodiments of the present invention and are therefore not to be considered as limiting the scope thereof. In the accompanying drawings, for clarity, identical or corresponding parts will be represented by identical or similar reference numerals.

[0100] Figure 1 A schematic flow chart of a path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to an embodiment of the present invention is shown;

[0101] Figure 2 A schematic diagram showing the training network structure of the DDPG algorithm according to an embodiment of the present invention; and

[0102] Figure 3 A schematic diagram of a path planning system for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0103] In the following description, the present invention is described with reference to various embodiments. However, those skilled in the art will recognize that various embodiments may be implemented without one or more specific details or with other replacement and / or additional methods or components. In other cases, well-known structures or operations are not shown or described in detail to avoid obscuring the inventive point of the present invention. Similarly, for the purpose of explanation, specific numbers and configurations are set forth to provide a comprehensive understanding of embodiments of the present invention. However, the present invention is not limited to these specific details.

[0104] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in this specification does not necessarily all refer to the same embodiment.

[0105] It should be noted that the embodiments of the present invention describe the method steps in a specific order, but this is only for the purpose of illustrating the specific embodiment, rather than limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0106] In the present invention, each module of the system according to the present invention can be implemented using software, hardware, firmware or a combination thereof. When the module is implemented using software, the function of the module can be implemented by a computer program flow, for example, the module can be implemented by a code segment (such as a code segment of a language such as C, C++) stored in a storage device (such as a hard disk, a memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When the module is implemented using hardware, the function of the module can be implemented by setting a corresponding hardware structure, for example, the function of the module can be implemented by hardware programming a programmable device such as a field programmable gate array (FPGA), or the function of the module can be implemented by designing an application-specific integrated circuit (ASIC) including electronic devices such as a plurality of transistors, resistors and capacitors. When the module is implemented using firmware, the function of the module can be written into a read-only memory such as an EPROM or EEPROM of the device in the form of a program code, and the corresponding function of the module can be implemented when the program code is executed by a processor. In addition, certain functions of the module may need to be implemented by separate hardware or by collaboration with the hardware, for example, the detection function is implemented by corresponding sensors (such as proximity sensors, acceleration sensors, gyroscopes, etc.), the signal transmission function is implemented by corresponding communication devices (such as Bluetooth devices, infrared communication devices, baseband communication devices, Wi-Fi communication devices, etc.), the output function is implemented by corresponding output devices (such as displays, speakers, etc.), and so on.

[0107] Figure 1 The following is a flow chart showing a path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to an embodiment of the present invention. Figure 1 , the path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning proposed by the present invention is described. In one embodiment of the present invention, the path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning can be executed by a computer. Figure 1 As shown, the path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning includes the following steps:

[0108] Firstly, according to the application scenario of the amphibious vehicle, sensors are used to collect scenario parameters, and the scenario parameters are used as input parameters.

[0109] The scene parameters include: speed V, acceleration A, water depth D, slope S, nearest obstacle position O, throttle position T rt , Steering wheel angle S a , braking force B, target position T p 、Target heading T h .

[0110] Next, the input parameters are fuzzified, reasoning is performed using a fuzzy rule base to obtain a fuzzy output, and the fuzzy output is defuzzified using a centroid method. This step is to establish a fuzzy logic system.

[0111] The input parameters are converted into fuzzy sets, and membership functions are used to describe the input parameters.

[0112] The fuzzy set of speed (Speed, V) is slow, medium, and fast. The membership function of speed V is,

[0113]

[0114] Among them, μ Slow (V), μ Medium (V), μ Fast (V) are the membership functions when the speed V is slow, medium and fast respectively.

[0115] The fuzzy set of acceleration (Acceleration, A) is low acceleration (Low), medium acceleration (Medium), high acceleration (High), and the membership function of acceleration A is,

[0116]

[0117] Among them, μ Low (A), μ Medium (A), μ Fast (A) are the membership functions of acceleration A for low acceleration, medium acceleration, and high acceleration, respectively.

[0118] The fuzzy set of water depth (D) is shallow, medium, and deep. The membership function of water depth D is:

[0119]

[0120]

[0121] Among them, μ Shallow (D), μ Medium (D), μ Deep(D) are the membership functions for water depths D of shallow, medium, and deep, respectively.

[0122] The fuzzy set of slope (Slope, S) is flat, moderate, and steep. The membership function of slope S is:

[0123]

[0124] Among them, μ Flat (S), μ Moderate (S), μ Steep (S) are the membership functions for slopes S as gentle, medium and steep slopes respectively.

[0125] The fuzzy set of the nearest obstacle position (Obstacle Position, O) is Near, Medium, and Far. The membership function of the nearest obstacle position O is,

[0126]

[0127] Among them, μ Near (O), μ Medium (O), μ Far (O) are the membership functions when the nearest obstacle position O is near, medium, and far, respectively.

[0128] Throttle Position, T rt ) is a fuzzy set of light accelerator (Light), medium accelerator (Medium), and heavy accelerator (Heavy). The throttle position T rt The membership function of is,

[0129]

[0130] Among them, μ Light (T rt ), μ Medium (T rt ), μ Heavy (T rt ) are the throttle positions T rt is the membership function of light accelerator, medium accelerator and heavy accelerator.

[0131] Steering Angle, S a ) is a fuzzy set of Left, Straight, and Right. The steering wheel angle S a The membership function of is,

[0132]

[0133]

[0134] Among them, μ Left (S a ), μ Straight (S a ), μ Right (S a ) are steering wheel angles S a is the membership function of left turn, straight go and right turn.

[0135] The fuzzy set of Brake Force (B) is Light, Medium, and Hard. The membership function of Brake Force B is:

[0136]

[0137] Among them, μ Light (B), μ Medium (B), μ Hard (B) are the membership functions of the braking force B for light braking, medium braking, and heavy braking, respectively.

[0138] Target Position, T p ) is a fuzzy set of near, medium, and far. The target position T p The membership function of is,

[0139]

[0140] Among them, μ Near (T p ), μ Medium (T p ), μ Far (T p ) are the target positions T p is the membership function for near, medium distance, and far distance.

[0141] Target Heading (T h ) is the fuzzy set of Left Deviation, Straight, and Right Deviation. The target heading T h The membership function of is,

[0142]

[0143] Among them, μ LeftDeviation (Th ), μ Straight (T h ), μ RightDeviation (T h ) are the target heading T h is the membership function of left yaw, straight travel, and right yaw.

[0144] The input parameters are inferred using the rules of the fuzzy rule base to obtain the fuzzy output. The rule form of the rule base is "IF condition THEN action".

[0145] The fuzzy output includes a driving risk index (DRI), a path complexity index (PCI), a collision risk index (CRI) and an energy efficiency index (EEI).

[0146] The rules of the fuzzy rule base corresponding to the driving risk index are:

[0147] Rule 1: If the speed is fast and the acceleration is high, the driving risk index is high, that is, Rule 1: IFμ Fast (V)ANDμ High (A) THENμ High (DRI).

[0148] Rule 2: If the speed is medium and the acceleration is medium, the driving risk index is medium, that is, Rule 2: IFμ Medium (V)ANDμ Medium (A) THENμ Medium (DRI).

[0149] Rule 3: If the speed is slow or the acceleration is low, the driving risk index is low, that is, Rule 3: IFμ Slow (V)ORμ Low (A) THENμ Low (DRI).

[0150] Rule 4: If the accelerator pedal is pressed hard and the steering wheel angle is left or right, the driving risk index is high, that is, Rule 4:IFμ Heavy (T rt )AND(μ Left (S a )ORμ Right (S a ))THENμ High (DRI).

[0151] Rule 5: If the braking force is heavy, the driving risk index is high, that is,

[0152] Rule 5:IFμ Hard (B)THENμ High (DRI).

[0153] The rules of the fuzzy rule base corresponding to the path complexity index are:

[0154] Rule 6: If the water depth is deep or the slope is steep, the path complexity index is high, i.e.

[0155] Rule 6:IFμ Deep (D)ORμ Steep (S)THENμ High (PCI).

[0156] Rule 7: If the water depth is medium or the slope is medium, the path complexity index is medium, that is, Rule 7:IFμ Medium (D)ORμ Moderate (S)THENμ Medium (PCI).

[0157] Rule 8: If the water depth is shallow or the slope is gentle, the path complexity index is low, that is,

[0158] Rule 8:IFμ Shallow (D)ORμ Flat (S)THENμ Low (PCI).

[0159] The rules of the fuzzy rule base corresponding to the collision risk index are:

[0160] Rule 9: If the nearest obstacle is close and the speed is fast, the collision risk index is high, that is, Rule 9: IFμ Near (O)ANDμ Fast (V)THENμ High (CRI).

[0161] Rule 10: If the nearest obstacle position is medium and the speed is medium, then the collision risk index is medium, that is, Rule 10: IFμ Medium (O)ANDμ Medium (V)THENμ Medium (CRI).

[0162] Rule 11: If the nearest obstacle is far away and the speed is slow, the collision risk index is low, that is, Rule 11: IFμ Far (O)ANDμ Slow(V)THENμ Low (CRI).

[0163] Rule 12: If the speed is medium and the acceleration is high and the position of the nearest obstacle is medium, then the collision risk index is high, i.e.

[0164] Rule 12:IFμ Medium (V)ANDμ High (A) ANDμ Medium (O)THENμ High (CRI).

[0165] Rule 13: If the braking force is medium braking and the nearest obstacle position is close, the collision risk index is medium, that is,

[0166] Rule 13:IFμ Medium (B)ANDμ Near (O)THENμ Medium (CRI).

[0167] The rules of the fuzzy rule base corresponding to the energy efficiency index are:

[0168] Rule 14: If the speed is fast and the target position is far and the target heading is straight,

[0169] The energy efficiency index is low, that is

[0170] Rule 14:IFμ Fast (V)ANDμ Far (T p )ANDμ Straight (T h )THENμ Low (EEI).

[0171] Rule 15: If the speed is medium, the target position is medium, and the target heading is straight, then the energy efficiency index is medium, i.e.

[0172] Rule 15:IFμ Medium (V)ANDμ Medium (T p )ANDμ Straight (T h )THENμ Medium (EEI) Rule 16, if the speed is slow and the target position is close and the target heading is straight,

[0173] The energy efficiency index is high, that is

[0174] Rule 16:IFμ Slow (V)ANDμ Near (Tp )ANDμ Straight (T h )THENμ High (EEI).

[0175] Rule 17: If the target heading is left or right, the energy efficiency index is medium.

[0176] Right now

[0177] Rule 17:IF(μ Left Deviation (T h )ORμ Right Deviation (T h ))THENμ Medium (EEI).

[0178] In order to convert the fuzzy output (driving risk index, path complexity index, collision risk index, energy efficiency index) into the specific state input required for reinforcement learning, the centroid method is used for defuzzification. The centroid method calculates the centroid position of the fuzzy set to obtain a specific and accurate control value. The defuzzification formula is:

[0179]

[0180] Among them, I crisp is the index value after defuzzification, μ j (I) is the membership degree of the jth rule to index I; I j is the output value of index I corresponding to the jth rule, and the values ​​corresponding to low, medium and high indexes are 0, 0.5 and 1 respectively;

[0181] The index I is,

[0182] I∈{DRI,PCI,CRI,EEI}

[0183] Among them, DRI is the driving risk index, PCI is the path complexity index, CRI is the collision risk index, and EEI is the energy efficiency index;

[0184] Through the defuzzification formula, the defuzzified index value DRI is obtained crisp 、PCI crisp 、CRI crisp EEI crisp .

[0185] Next, define the state space and action space, reset the environment, execute actions and calculate rewards, and complete this step when the round end condition is met. This step is to establish a reinforcement learning environment.

[0186] The state space is characterized by the fuzzy inference system, the state dimension is reduced, and the convergence speed of the reinforcement learning algorithm is accelerated. The state space s is defined as:

[0187] s={x,y,z,T,DRI crisp ,PCI crisp ,CRI crisp ,EEI crisp}

[0188] Among them, DRI crisp 、PCI crisp 、CRI crisp EEI crisp is the defuzzified index value, x, y, z are the position coordinates of the amphibious vehicle, and T is the terrain type.

[0189] In order to ensure that the exponential values ​​of the above states are suitable as state inputs for reinforcement learning, they are standardized and normalized. The specific methods are as follows:

[0190] First, standardization and normalization of continuous variables:

[0191] The continuous variables in the state space include the position coordinates (x, y, z) of the amphibious vehicle and various index values ​​(DRI crisp 、PCI crisp 、CRI crisp EEI crisp ). These variables are standardized and normalized using Min-MaxNormalization:

[0192] Normalize each continuous variable v to the interval [0,1], the formula is as follows:

[0193]

[0194] Among them: v is the original variable value, v min and v max are the minimum and maximum values ​​of the variable in the training data, respectively.

[0195] For encoding and normalization of categorical variables:

[0196] The categorical variables in the state space include the terrain type T, which is converted into numerical form through the categorical conversion method and normalized:

[0197] Assume that the terrain type T has three categories: Land, Water, and Muddy. Use the numerical mapping method to map the terrain type to the following values:

[0198]

[0199] The standardization and normalization of the comprehensive state space are used to obtain the specific state input s of the reinforcement learning algorithm. t ,

[0200] s t ={x norm ,y norm ,z norm ,T norm ,DRI norm ,PCI norm ,CRI norm ,EEI norm}

[0201] Among them, x norm ,y norm 、z norm , T norm 、DRI norm 、PCI norm 、CRI norm EEI norm They are x, y, z, T, and DRI respectively. crisp 、PCI crisp 、CRI crisp EEI crisp The normalized value of .

[0202] Define the action space a t for,

[0203]

[0204] in, is the throttle position, is the steering wheel angle, u B For braking force.

[0205] Reset environment: Initialize the environment state, set the initial position and initial state parameters of the amphibious vehicle.

[0206] Update the environment state according to the current action space,

[0207] s t+1 =f(s t ,a t )

[0208] Among them, s t+1 is the updated state space, f is the environment state update function, a t is the action space. The actions include throttle Steering wheel angle Braking force B .

[0209] Function f represents the given current state s t and action a t In this paper, by introducing physical state updates and exponential reasoning of fuzzy logic, how the environment transitions to the next state includes the vehicle's motion dynamics, sensor data, terrain changes, and fuzzy logic reasoning output.

[0210] The specific state update function equation is:

[0211] First, the location update:

[0212]

[0213] Among them, θ t is the steering angle of the vehicle, v t For speed, is the sensor noise; Δt is the time step, which represents the time interval between each state update.

[0214] Secondly, the fuzzy logic reasoning is updated:

[0215] The fuzzy logic reasoning system updates various indexes through the fuzzy rule base according to the current state and action. The risk index update formula after defuzzification is:

[0216] Driving Risk Index Update:

[0217]

[0218] Path complexity index update:

[0219]

[0220] Collision Risk Index Update:

[0221]

[0222] Energy Efficiency Index Update:

[0223]

[0224] Terrain Type T norm Environmental data detected by sensors is updated based on changes in vehicle position and path selection.

[0225] Calculate the reward value based on the output of the fuzzy logic system and the updated environment state. Calculate the reward function R t ,

[0226] R t =R safe +R path +R energy +R goal

[0227] R safe , R path , R energy , R goal They are safety reward, path complexity reward, energy efficiency reward and goal completion reward.

[0228] Safety rewards are calculated based on the Driving Risk Index (DRI) and the Crash Risk Index (CRI) to penalize high-risk states and ensure safe driving.

[0229] R safe = -α1·DRI norm -α2·CRI norm

[0230] Among them, α1 and α2 are weight coefficients, which are used to balance the impact of driving risk and collision risk respectively. The path complexity reward is calculated by the path complexity index (PCI), which penalizes complex paths and encourages amphibious vehicles to choose simpler and more efficient paths.

[0231] R path =-β·PCI norm

[0232] Among them, β is the weight coefficient, which is used to control the influence of path complexity.

[0233] Energy efficiency rewards are calculated through the Energy Efficiency Index (EEI), which encourages energy-efficient driving and optimizes the vehicle's energy consumption performance.

[0234] R energy =γ·EEI norm

[0235] Among them, γ is the weight coefficient, which is used to encourage energy-efficient driving strategies.

[0236] The goal completion reward is based on the vehicle's position with the goal T p distance, rewarding amphibious vehicles for reaching the target location as quickly as possible.

[0237]

[0238] Among them, δ is the target reward coefficient.

[0239] The episode is a complete interactive process in which the amphibious vehicle starts from the initial state in the environment and performs a series of actions until the termination condition is met. Each episode can contain multiple time steps. The amphibious vehicle completes a specific task through a series of state transitions and action selections. The termination condition defines when an episode ends, which is used to ensure the rationality and effectiveness of the amphibious vehicle training process.

[0240] The end of round conditions include:

[0241] When the amphibious vehicle successfully reaches the target location, the round ends.

[0242] T p ≤∈

[0243] Among them, ∈ is the tolerance distance of the target position;

[0244] When the nearest obstacle position O does not exceed the tolerance distance O min , that is, O≤O min , a collision occurs and the round ends;

[0245] The time step number reaches the preset maximum time step number T max , the round is forced to end to avoid unlimited training cycles.

[0246] Taking the above conditions into consideration, when any condition is met, the current round ends and enters the next round.

[0247] Next, the Actor-Critic neural network is initialized to achieve stable learning of the strategy through experience replay and target network update mechanism.

[0248] The DDPG algorithm is suitable for continuous action spaces and combines deterministic policy gradients with the value evaluation of the Critic network. The algorithm uses the Actor-Critic neural network architecture and achieves stable learning of the strategy through experience replay and target network update mechanisms.

[0249] Initialize the Actor network μ(s|θ μ ), μ(s|θ μ ) is used to calculate the current state s t Output control action a t .

[0250] Initialize the Critic network Q(s,a|θ Q ), Q(s,a|θ Q ) is used to evaluate the state-action pair Q(s t ,a t )’s value.

[0251] Initialize the target network μ′(s|θ μ′ ) and Q′(s,a|θ Q′ ) is updated in a smooth process, and the parameters are initialized to be the same as the main network, that is, θ μ′ =θ μ ,θ Q′ =θ Q .

[0252] Initialize experience replay pool Used to store experience data.

[0253] From the current state t , through the Actor network μ(s t |θ μ ) Calculate action a t , and add noise Explore.

[0254]

[0255] Execute action a t , get the next state s t+1 、Instant Rewards t , termination sign done.

[0256] The experience (s t ,a t ,r t ,s t+1 , done) stored in the experience replay pool

[0257] Mini-batches of data (s) sampled from the experience replay pool i ,a i ,r i ,s i+1 ,done i ), calculate the target value y of the Critic network i ,

[0258] y i =r i +γ1(1-done i )Q′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ )

[0259] Among them, γ1 is the discount factor.

[0260] Update the Critic network by minimizing the mean square error loss function.

[0261]

[0262] Where N is the size of the batch data, that is, the number of experiences sampled from the experience replay pool.

[0263] Update the Actor network using policy gradients:

[0264]

[0265] Where N is the size of the batch data, that is, the number of experiences sampled from the experience replay pool; The objective function J is the Actor network parameter θ μ The gradient of indicates the optimization direction of the Actor network; is the gradient of the action in the Critic network, representing the state-action pair (s i ,a)’s action value function Q(s i ,a) derivative of action a; is the strategy function μ(s) of the Actor network i |θ μ ) for network parameters θ μ The gradient of the Actor network for a given state s i The rate of change of action that should be chosen.

[0266] Update the target network parameters through the soft update mechanism,

[0267] θ μ′ ←τθ μ +(1-τ)θ μ′

[0268] θ Q′ ←τθ Q +(1-τ)θ Q′

[0269] Among them, τ is the soft update coefficient.

[0270] Finally, the Actor-Critic neural network is trained and optimized to obtain the optimal path planning strategy.

[0271] During training, the amphibious vehicle learns the optimal policy through continuous interaction with the environment. Figure 2 FIG. 4 shows a schematic diagram of a training network structure of a DDPG algorithm according to an embodiment of the present invention. Figure 2 As shown, the specific training and optimization steps are as follows:

[0272] Reset the environment, set the initial state s0, and initialize an empty experience replay pool Initialize the neural network parameters.

[0273] In each training round, the state update, experience replay and target network update of multiple time steps are executed cyclically. Specifically, at each time step, the current scene parameters are fuzzified by the fuzzy inference system to generate fuzzy outputs, and the new index values ​​are obtained by the centroid method. These index values ​​are used to update the current state space. Next, experience replay and target network update are performed. For the current state s t, using the Actor network μ(s t |θ μ ) Generate action a t , execute action a in the current environment t , record the new state s t+1 、Instant Rewards t ; The current round of experience data (s t ,a t ,r t ,s t+1 , done) stored in the experience replay pool In each network update, the experience replay pool Randomly sample a mini-batch (s i ,a i ,r i ,s i+1 ,done i ) Update the Critic and Actor networks; perform a soft update of the target network. The Critic network evaluates the value of each state-action pair and updates it by minimizing the loss function.

[0274] During the training process of the DDPG algorithm, the optimization goal is to maximize the cumulative reward R of the amphibious vehicle in the continuous action space. t , thereby learning the optimal strategy.

[0275] During the training process, define the cumulative reward R t ,

[0276]

[0277] Among them, T is the end time of the current round, γ2 is the discount factor, r k is the immediate reward at the kth time step;

[0278] The training process continues until the training stop condition is met.

[0279] The conditions for stopping training are:

[0280] When the number of training rounds reaches the maximum value E max The training stops when .

[0281] Amphibious vehicles accumulate rewards R in a continuous round t Exceeding the predetermined threshold R target When , the algorithm is considered to have converged and the training is terminated. Through the above process, when the training stops, the optimal path planning strategy is finally obtained.

[0282] The path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning proposed in the present invention is based on fuzzy logic rules and DDPG algorithm, combined with fuzzy logic system to pre-process input data, effectively process continuous input data in complex environments, and ensure efficient and safe navigation of amphibious vehicles in different terrains and waters; the established fuzzy logic system can quickly respond to environmental changes and provide real-time control instructions through fuzzification processing and rule reasoning, thereby improving the flexibility and adaptability of path planning; the DDPG algorithm stabilizes the reinforcement learning process through experience replay and soft update of the target network, thereby improving the learning efficiency and the robustness of the model; through the combination of fuzzy logic system and reinforcement learning algorithm, efficient solution to path planning problems in complex environments is achieved, and it has broad application prospects.

[0283] In one embodiment of the present invention, the present invention also provides a path planning system for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning for use in the path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning as described above. Figure 3 FIG. 1 is a schematic diagram of a path planning system for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to an embodiment of the present invention. Figure 3 As shown, the system includes the following modules:

[0284] The present invention also provides a path planning system for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning for the path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning, the system comprising the following modules:

[0285] An input parameter collection module is configured to use sensors to collect scenario parameters according to the application scenario of the amphibious vehicle and use the scenario parameters as input parameters;

[0286] A fuzzy logic system building module is configured to fuzzify the input parameters, perform reasoning using a fuzzy rule base to obtain a fuzzy output, and perform output defuzzification on the fuzzy output using a centroid method;

[0287] A reinforcement learning environment establishment module is configured to define the state space and action space, reset the environment, execute actions and calculate rewards, and complete this step when the round end condition is met;

[0288] The Deep Deterministic Policy Gradient algorithm module is configured to initialize the Actor-Critic neural network and achieve stable learning of the policy through experience replay and target network update mechanism; and

[0289] The training and optimization module is configured to train and optimize the Actor-Critic neural network to obtain the best path planning strategy.

[0290] In one embodiment of the present invention, the present invention also provides a computer system, which includes a processor, a graphics card with an artificial intelligence chip and a memory, the memory is configured to store machine-readable instructions, the graphics card is configured to train the path planning method of the amphibious vehicle based on fuzzy logic rules combined with reinforcement learning, and the processor is configured to execute the machine-readable instructions. When the processor and / or the graphics card execute the machine-readable instructions, the following processing steps are implemented: according to the application scenario of the amphibious vehicle, the scene parameters are collected using sensors, and the scene parameters are used as input parameters; the input parameters are fuzzified, and the fuzzy rule library is used for reasoning to obtain fuzzy output, and the fuzzy output is defuzzified using the center of gravity method; the state space and action space are defined, the environment is reset, the action is executed and the reward is calculated, and this step is completed when the round end condition is met; the Actor-Critic neural network is initialized, and the stable learning of the strategy is realized through the experience playback and the update mechanism of the target network; the Actor-Critic neural network is trained and optimized to obtain the best path planning strategy.

[0291] The graphics card may preferably have a GPU computing power higher than 5.0. Since the amount of data to be trained is large, providing a graphics card configuration can significantly increase the training speed.

[0292] The memory includes: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store machine-readable instructions.

[0293] It can be understood that in addition to the memory and processor mentioned above, the above-mentioned computer system also includes other software and hardware components not listed in this specification. The specific components can be determined according to the model of specific data processing equipment in different application scenarios, and this specification will not list them one by one in detail.

[0294] In one embodiment of the present invention, the present invention also provides a computer-readable storage medium having machine-readable instructions stored thereon, wherein the machine-readable instructions implement the following processing steps when executed by a processor: according to the application scenario of the amphibious vehicle, using sensors to collect scene parameters, and using the scene parameters as input parameters; fuzzifying the input parameters, using a fuzzy rule library for reasoning to obtain a fuzzy output, and using a centroid method to perform output defuzzification on the fuzzy output; defining a state space and an action space, resetting the environment, executing actions and calculating rewards, and completing this step when the round end condition is met; initializing the Actor-Critic neural network, and realizing stable learning of the strategy through experience replay and the update mechanism of the target network; training and optimizing the Actor-Critic neural network to obtain the optimal path planning strategy.

[0295] Although various embodiments of the present invention are described above, it should be understood that they are presented as examples only and not as limitations. It is obvious to those skilled in the relevant art that various combinations, modifications and changes can be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined according to the technical solutions of the present invention and their equivalents.

Claims

1. A path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning, characterized in that: The steps include: According to the application scenario of the amphibious vehicle, using sensors to collect scenario parameters, and using the scenario parameters as input parameters; The input parameters are fuzzified, reasoning is performed using a fuzzy rule base to obtain a fuzzy output, and the fuzzy output is defuzzified using a centroid method; Define the state space and action space, reset the environment, execute actions and calculate rewards, and complete this step when the round end condition is met; Initialize the Actor-Critic neural network and achieve stable learning of the strategy through experience replay and target network update mechanism; as well as Train and optimize the Actor-Critic neural network to obtain the best path planning strategy.

2. The path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to claim 1 is characterized in that: The scene parameters include: speed V, acceleration A, water depth D, slope S, nearest obstacle position O, throttle position T rt , Steering wheel angle S a , braking force B, target position T p 、Target heading T h .

3. The path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to claim 1 is characterized in that: The input parameters are fuzzified, reasoning is performed using a fuzzy rule base to obtain a fuzzy output, and the fuzzy output is defuzzified using a centroid method, including: Convert the input parameters into fuzzy sets and describe the input parameters using membership functions; Using the rules of the fuzzy rule base, reasoning on the input parameters to obtain a fuzzy output; The fuzzy output includes a driving risk index, a path complexity index, a collision risk index, and an energy efficiency index; The rules of the fuzzy rule base corresponding to the driving risk index are: if the speed is fast and the acceleration is high, the driving risk index is high; if the speed is medium and the acceleration is medium, the driving risk index is medium; if the speed is slow or the acceleration is low, the driving risk index is low; if the accelerator position is heavy and the steering wheel angle is left turn or right turn, the driving risk index is high; if the braking force is heavy braking, the driving risk index is high; The rules of the fuzzy rule base corresponding to the path complexity index are: if the water depth is deep or the slope is steep, the path complexity index is high; if the water depth is medium or the slope is medium, the path complexity index is medium; if the water depth is shallow or the slope is gentle, the path complexity index is low; The rules of the fuzzy rule base corresponding to the collision risk index are: if the nearest obstacle position is close and the speed is fast, the collision risk index is high; if the nearest obstacle position is medium and the speed is medium, the collision risk index is medium; if the nearest obstacle position is far and the speed is slow, the collision risk index is low; if the speed is medium and the acceleration is high and the nearest obstacle position is medium, the collision risk index is high; if the braking force is medium braking and the nearest obstacle position is close, the collision risk index is medium; The rules of the fuzzy rule base corresponding to the energy efficiency index are: if the speed is fast, the target position is far, and the target heading is straight, the energy efficiency index is low; if the speed is medium, the target position is medium, and the target heading is straight, the energy efficiency index is medium; if the speed is slow, the target position is close, and the target heading is straight, the energy efficiency index is high; if the target heading is left yaw or right yaw, the energy efficiency index is medium. The defuzzification formula is: Among them, I crisp is the index value after defuzzification, μ j (I) is the membership degree of the jth rule to index I; I j is the output value of index I corresponding to the jth rule, and the values ​​corresponding to low, medium and high indexes are 0, 0.5 and 1 respectively; The index I is, I∈{DRI,PCI,CRI,EEI} Among them, DRI is the driving risk index, PCI is the path complexity index, CRI is the collision risk index, and EEI is the energy efficiency index; Through the defuzzification formula, the defuzzified index value DRI is obtained crisp 、PCI crisp 、CRI crisp EEI crisp .

4. The path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to any one of claims 1 to 3, characterized in that: Defining the state space and action space, resetting the environment, executing actions and calculating rewards involves: Define the state space s as, S={x,y,z,T,DRI crisp ,PCI crisp ,CRI crisp ,EEI crisp } Among them, DRI crisp 、PCI crisp 、CRI crisp EEI crisp is the defuzzified index value, x, y, z are the position coordinates of the amphibious vehicle, and T is the terrain type; Normalize the parameters contained in the state space s to obtain the specific state input s of the reinforcement learning algorithm t , s t ={x norm ,y norm ,z norm ,T norm ,DRI norm ,PCI norm ,CRI norm ,EEI norm } Among them, x norm ,y norm 、z norm , T norm 、DRI norm 、PCI norm 、CRI norm EEI norm They are x, y, z, T, and DRI respectively. crisp 、PCI crisp 、CRI crisp EEI crisp The normalized value of ; Define the action space a t for, in, is the throttle position, is the steering wheel angle, u B For braking force; Initialize the environment state, set the initial position and various initial state parameters of the amphibious vehicle; Update the environment state according to the current action space, s t+1 =f(s t ,a t ) Among them, s t+1 is the updated state space, f is the environment state update function, a t is the action space; Calculate the reward function R t , R t =R safe +R path +R energy +R goal R safe , R path , R energy , R goal They are safety reward, path complexity reward, energy efficiency reward and goal completion reward.

5. The path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to claim 1 is characterized in that: The round is a complete interactive process in which the amphibious vehicle starts from the initial state in the environment and performs a series of actions until the termination condition is met; each round can include multiple time steps, and the amphibious vehicle completes a specific task through a series of state transitions and action selections; The end of round conditions include: When the amphibious vehicle successfully reaches the target location, the round ends. T p ≤∈ Among them, ∈ is the tolerance distance of the target position; When the nearest obstacle position O does not exceed the tolerance distance O min , that is, a collision occurs and the round ends; The time step number reaches the preset maximum time step number T max , the round ends.

6. The path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to claim 1 is characterized in that: Initialize the Actor-Critic neural network and achieve stable learning of the strategy through experience replay and target network update mechanism, including: Initialize the Actor network μ(s|θ μ ), μ(s|θ μ ) is used to calculate the current state s t Output control action a t ; Initialize the Critic network Q(s,a|θ Q ), Q(s,a|θ Q ) is used to evaluate the state-action pair Q(s t ,a t )’s value; Initialize the target network μ′(s|θ μ′ ) and Q′(s,a|θ Q′ ), the parameters are initialized to be the same as the main network, that is, θ μ′ =θ μ ,θ Q′ =θ Q ; Initialize experience replay pool From the current state t , through the Actor network μ(s t |θ μ ) Calculate action a t , and add noise Explore. Execute action a t , get the next state s t+1 、Instant Rewards t , termination sign done; The experience (s t ,a t ,r t ,s t+1 , done) stored in the experience replay pool Update the Critic network by minimizing the mean square error loss function. Where N is the size of the batch data, that is, the number of experiences sampled from the experience replay pool; y i is the target value of the Critic network, y i =r i +γ1(1-done i )Q′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ) Among them, γ1 is the discount factor, (s i ,a i ,r i ,s i+1 ,done i ) is a small batch of data sampled from the experience replay pool; Update the Actor network using policy gradients: Where N is the size of the batch data, that is, the number of experiences sampled from the experience replay pool; The objective function J is the Actor network parameter θ μ The gradient of indicates the optimization direction of the Actor network; is the gradient of the action in the Critic network, representing the state-action pair (s i ,a)’s action value function Q(s i ,a) derivative of action a; is the strategy function μ(s) of the Actor network i |θ μ ) for network parameters θ μ The gradient of the Actor network for a given state s i The rate of change of action that should be chosen; Update the target network parameters through the soft update mechanism, i μ′ ←tth μ +(1-τ)θ μ′ i Q′ ←tth Q +(1-τ)θ Q′ Among them, τ is the soft update coefficient.

7. The path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning according to any one of claims 1 to 6, characterized in that: Training and optimizing the Actor-Critic neural network to obtain the best path planning strategy includes: Set the initial state s0 to initialize an empty experience replay pool Initialize neural network parameters; In each training round, the state update, experience replay and target network update of multiple time steps are executed cyclically; the Actor network is updated according to the current state s t Output control action a t , the Critic network evaluates the value of each state-action pair and updates it by minimizing the loss function; During the training process, define the cumulative reward R t , Among them, T is the end time of the current round, γ2 is the discount factor, r k is the immediate reward at the kth time step; The conditions for stopping training are: When the number of training rounds reaches the maximum value E max When , the training stops; Amphibious vehicles accumulate rewards R in a continuous round t Exceeding the predetermined threshold R target When , the training stops; When training stops, the optimal path planning strategy is obtained.

8. A planning system for the path planning method of an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: Includes the following modules: An input parameter collection module is configured to use sensors to collect scenario parameters according to the application scenario of the amphibious vehicle and use the scenario parameters as input parameters; A fuzzy logic system building module is configured to fuzzify the input parameters, perform reasoning using a fuzzy rule base to obtain a fuzzy output, and perform output defuzzification on the fuzzy output using a centroid method; A reinforcement learning environment establishment module is configured to define the state space and action space, reset the environment, execute actions and calculate rewards, and complete this step when the round end condition is met; The deep deterministic policy gradient algorithm module is configured to initialize the Actor-Critic neural network and achieve stable learning of the policy through experience replay and target network update mechanism; as well as The training and optimization module is configured to train and optimize the Actor-Critic neural network to obtain the best path planning strategy.

9. A computer system, characterized in that: include: a processor configured to execute machine-readable instructions; A graphics card with an artificial intelligence chip configured to train a path planning method for an amphibious vehicle based on fuzzy logic rules combined with reinforcement learning; as well as A memory configured to store machine-readable instructions, wherein the machine-readable instructions, when executed by a processor and / or a graphics card, perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Machine-readable instructions are stored thereon, and when the machine-readable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are performed.

Images