Method for controlling differential steering driving system of climbing cleaning vehicle based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based control method for a differential steering driving system of a climbing cleaning vehicle, and belongs to the technical field of hull cleaning vehicle control methods. Comprising the following steps: model establishment: modeling a differential steering driving control process of the wall-climbing cleaning vehicle into a Markov decision process; defining a state space and an action space; constructing a state transition model of the differential steering driving system; acquiring current state data, and generating a steering wheel speed and steering time distribution strategy through the model; executing the steering action strategy, and updating the state data after action execution; rewards are returned according to the actual execution effect to judge whether the action strategy is good or bad; performing model optimization through an experience playback pool; and resetting the scene to continue training until the model converges, and finally obtaining the trained differential steering driving system model. According to the method, the complexity of path planning and control algorithms is reduced, the requirement for maximum cleaning of the coverage area is met, and the difficulty of the path planning and control algorithms is simplified.

Description

Technical Field

[0001] The present invention belongs to the technical field of ship hull cleaning vehicle control methods, and more specifically, relates to a control method for a differential steering drive system of a crawling cleaning vehicle based on reinforcement learning. Background Art

[0002] The inner wall of bulk carrier holds is often rusted and contaminated, making cleaning difficult and requiring strict inspection. Especially when loading grain after loading oily cargoes, cargo ship companies need to invest a lot of resources in cleaning to avoid delays. The outer wall of the hull is also subject to biofouling, which not only affects the speed and increases fuel consumption, but is also the main carrier of aquatic species transfer.

[0003] At present, manual cleaning or semi-automatic equipment is mostly used to clean the outer wall of the hull. Not only is it time-consuming, costly, and labor-intensive, but there are also safety risks. Therefore, cargo hold cleaning robots and underwater cleaning robots are gradually being used to complete the cleaning task. The cleaning robot is usually based on a trolley structure, fixed to the hull wall using the principle of magnetic adsorption, and cleaned with a cleaning brush or jet disk. However, these devices usually require professional personnel to operate, and their autonomous navigation capabilities and intelligence levels still need to be improved. Not only does it cause the inner and outer wall surfaces of the hull to be complex, but it is also difficult for the cleaning trolley to effectively cover the entire cleaning area through a small radius turn, resulting in omissions and repeated cleaning. In addition, the high control level required of the operator affects the hull cleaning effect and cleaning efficiency. Summary of the invention

[0004] The present invention aims at solving the technical problems existing in the prior art and provides a control method for a differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning.

[0005] To solve the above technical problems, the present invention comprises the following steps:

[0006] S1. Model establishment: the differential steering drive control process of the wall-climbing cleaning vehicle is modeled as a Markov decision process;

[0007] S2, complete the definition of state space and action space;

[0008] S3, constructing a state transfer model of the differential steering drive system;

[0009] S4, obtaining current state data, and generating a steering wheel speed and steering time allocation strategy through a model;

[0010] S5, execute the steering action strategy and update the status data after the action is executed;

[0011] S6. Return rewards based on actual execution results to judge the quality of the action strategy;

[0012] S7, model optimization through experience replay pool;

[0013] S8, the wall-climbing cleaning vehicle continues to move forward and presets the cleaning distance;

[0014] S9. Repeat S4-S8. After the training is completed, reset the scene and continue training until the model converges. Finally, a trained differential steering drive system model is obtained.

[0015] Preferably, in S1, a tuple [S, A, R, P] is used to represent the Markov decision process of the differential steering drive of the wall-climbing cleaning vehicle, and an initial action strategy π is given, where:

[0016] S represents the state space, that is, the state information of the environment where the agent is located at time t;

[0017] A represents the action space, that is, at time t, the environment state is S t When , the agent selects actions according to strategy π, where strategy π refers to the mapping from state to action;

[0018] R represents the reward, that is, at time t, the environment state is S t When the agent takes action A t Enter the next state S t+1 Rewards obtained when

[0019] P represents the state transition probability, which means that the agent is given a previous state S at time t. t ∈S and action A t ∈A, the probability of occurrence of random variables sˊ and r.

[0020] Preferably, S2 comprises:

[0021] S2.1. Construct a global coordinate system with the initial wall climbing position of the car as the origin;

[0022] S2.2, the current state data of the wall-climbing cleaning vehicle includes: the coordinates (x, y, z) of the wall-climbing cleaning vehicle in the global coordinate system, the speed v of the wall-climbing cleaning vehicle in the geodetic coordinate system, the roll angle Roll, the pitch angle Pitch, the heading angle Yaw, and the preset turning and lateral displacement distance d of the wall-climbing cleaning vehicle;

[0023] S2.3, action output is: Action = [leftS, rightS, turnT];

[0024] Among them, leftS represents the left wheel speed of the wall-climbing cleaning vehicle when turning, radian / second, rightS represents the right wheel speed of the wall-climbing cleaning vehicle when turning, radian / second, and turnT represents the turning time of the wall-climbing cleaning vehicle, seconds.

[0025] Preferably, the minimum, maximum and step sizes of the left and right wheel speeds and turning times of the wall-climbing cleaning vehicle are set to convert continuous values ​​into discrete points. At this time, the action space can be understood as a three-dimensional discrete point action space, and each point in the three-dimensional action space corresponds to different left and right wheel speeds and turning times.

[0026] Preferably, in S3, a control model based on a deep Q network DQN framework is constructed, and the DQN framework includes:

[0027] The policy network estimates the Q value of each action in a given state, that is, the action-value function Q(s,a) is used to evaluate the expected value of the reward after performing action a in state s;

[0028] Target network, a target network with the same structural parameters as the online network generates the target Q value.

[0029] Preferably, in S4, the state data is transmitted as input to the policy network, and the policy network outputs the expected value of the reward obtained when taking different actions in the current state.

[0030] Preferably, in S5, an ∈-greedy algorithm is introduced, including:

[0031] S5.1, randomly select an action with probability ∈, that is, a random steering wheel speed and steering time;

[0032] S5.2, select the steering wheel speed and steering time strategy with the largest Q value as the action output of the current strategy with probability 1-∈;

[0033] S5.3. Execute actions and update status data S t+1 .

[0034] Preferably, S6 includes:

[0035] Reward R(s,a)=R(str)+R(dev)+R(ec)

[0036] Among them, R(str) is the straight-line walking penalty, which is used to measure whether the car can travel in a predetermined straight line after turning;

[0037] The judgment index is the change in cleaning height or depth z after turning and walking a set cleaning distance;

[0038] R(dev) is the lateral displacement penalty, which is used to measure whether the car has moved laterally by a set distance relative to its position before turning after turning.

[0039] The height / depth of the trolley before turning is z, the height / depth of the trolley after turning is z′, and the position offset before and after turning is z′-z;

[0040] The two coordinate systems corresponding to the Euler angles are determined by the roll angle, pitch angle and heading angle of the wall-climbing cleaning vehicle before and after turning relative to the global coordinate system, the projections of the two coordinate systems on the corresponding planes are calculated, and the projection vectors are normalized to solve the angle θ between the two vectors before and after turning and the vehicle body relative to the turning wall;

[0041] The judgment index is the difference between the front and rear position offset z′-z and the preset steering lateral displacement d under the action of the angle θ between the front and rear body and the steering wall;

[0042] R(ec) is the energy loss penalty, which is used to measure whether the car slips and generates unnecessary energy consumption during the steering process;

[0043] The judgment index is the lateral slip angle, which is the angular difference between the actual driving direction and the direction of the tire;

[0044] The lateral slip angle is defined as:

[0045]

[0046] Among them, v lateral is the lateral velocity, equivalent to the velocity of the wall-climbing cleaning vehicle in the geodetic coordinate system;

[0047] v longitudinal is the longitudinal velocity, calculated using the acceleration and angular velocity data provided by the inertial unit.

[0048] Preferably, S7 includes:

[0049] S7.1. Perform data sampling and convert the interaction data [S, A, R, S t+1 ] is stored in the experience replay pool. When the experience replay pool has accumulated enough samples, data in the experience replay pool is randomly extracted for training;

[0050] S7.2. Introduce priority experience replay to improve sample utilization and calculate the sampling probability of each sample in the experience replay pool. The sampling probability is calculated as follows:

[0051]

[0052] Among them, D j is the priority of the jth sample, D k is the priority of the kth sample, and α is the priority degree;

[0053] S7.3, predict the current sample through the current value network, calculate the loss between the current value network and the target value network, and use gradient descent to minimize the loss to update the current value network parameters. After a certain number of iterations, synchronize the parameters of the current value network to the target value network;

[0054] The value function update formula is:

[0055]

[0056] Among them, α is the learning rate, γ is the reward discount factor, and w is the parameter of the corresponding network.

[0057] Preferably, S9 comprises:

[0058] As training progresses, the cumulative reward of each round of training gradually increases, indicating that the model strategy is constantly improving and eventually tends to a stable value, indicating that the performance of the model in the current training environment has reached a balance point;

[0059] The loss function used to measure the mean square error between the current value network and the target value network should gradually decrease, reflecting the convergence of the model during the optimization process. The loss stabilizes at a certain value, indicating that the model's predictions are consistent with the Q-value target Q-value.

[0060] When both indicators show a stable state, it can be considered that the current training has converged and the model has achieved relatively ideal performance.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention adopts a reinforcement learning method to optimize the steering speed and steering time strategy of the four-wheel wall-climbing cleaning vehicle. It can autonomously adapt to task requirements to perform lateral displacement with a small turning radius, and convert the problems of where to go and how to go that need to be solved in path planning into what kind of posture to adjust to meet the requirements of maximizing the cleaning coverage area.

[0063] The present invention constructs a state transfer model of the differential steering drive system, collects state data of the wall-climbing vehicle, and transmits it to the model, generates a corresponding steering action strategy, and simultaneously performs model optimization and strategy evaluation, thereby reducing the complexity of path planning and control algorithms. The reinforcement learning algorithm enables the vehicle to continuously learn and adjust its own behavior strategy through interaction with the environment. It can learn how to take actions in the environment to maximize expected returns through continuous trial and error during the interaction between the intelligent agent and the environment, find the best wall-climbing path and action, and meet the requirement of maximizing the cleaning area by adjusting to the best posture, thereby effectively improving the intelligence of the wall-climbing cleaning vehicle and improving the cleaning efficiency and cleaning effect of the wall-climbing cleaning vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0065] Figure 1 A schematic diagram of a training process of an embodiment of a control method of the present invention;

[0066] Figure 2 Schematic diagram of the algorithm framework of the control method embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] See also Figure 1 The embodiment of the present invention provides a control method for a differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning, comprising the following steps:

[0069] S1. Model establishment: the differential steering drive control process of the wall-climbing cleaning vehicle is modeled as a Markov decision process;

[0070] S2, complete the definition of state space and action space;

[0071] S3, constructing a state transfer model of the differential steering drive system;

[0072] S4, obtaining current state data, and generating a steering wheel speed and steering time allocation strategy through a model;

[0073] S5, execute the steering action strategy and update the status data after the action is executed;

[0074] S6. Return rewards based on actual execution results to judge the quality of the action strategy;

[0075] S7, model optimization through experience replay pool;

[0076] S8, the wall-climbing cleaning vehicle continues to move forward and presets the cleaning distance;

[0077] S9. Repeat S4-S8. After the training is completed, reset the scene and continue training until the model converges.

[0078] The present invention adopts a reinforcement learning method to optimize the steering speed and steering time strategy of the four-wheel wall-climbing cleaning vehicle. It can autonomously adapt to task requirements to perform lateral displacement with a small turning radius, and convert the problems of where to go and how to go that need to be solved in path planning into what kind of posture to adjust to meet the requirements of maximizing the cleaning coverage area.

[0079] The present invention constructs a state transfer model of the differential steering drive system, collects state data of the wall-climbing vehicle, and transmits it to the model, generates a corresponding steering action strategy, and simultaneously performs model optimization and strategy evaluation, thereby reducing the complexity of path planning and control algorithms. The reinforcement learning algorithm enables the vehicle to continuously learn and adjust its own behavior strategy through interaction with the environment. It can learn how to take actions in the environment to maximize expected returns through continuous trial and error during the interaction between the intelligent agent and the environment, find the best wall-climbing path and action, and meet the requirement of maximizing the cleaning area by adjusting to the best posture, thereby effectively improving the intelligence of the wall-climbing cleaning vehicle and improving the cleaning efficiency and cleaning effect of the wall-climbing cleaning vehicle.

[0080] Specifically, the embodiment of the present invention provides a control method for a differential steering drive system of a wall-climbing cleaning vehicle based on reinforcement learning. By adjusting to the optimal posture to meet the demand for maximizing the cleaning area, the wall-climbing vehicle can cover more areas and improve the cleaning efficiency within a unit time. Assuming that the current cleaning method is to achieve horizontal cleaning of different cleaning heights or depths by steering, the embodiment of the present invention is also applicable to longitudinal cleaning. On this basis, the control method includes the following steps:

[0081] S1. Model establishment: The differential steering drive control process of the wall-climbing cleaning vehicle is modeled as a Markov decision process.

[0082] Specifically, the Markov decision process of the differential steering drive of the wall-climbing cleaning vehicle is represented by a tuple [S, A, R, P], and the initial action strategy π is given, where:

[0083] S represents the state space, that is, the state information of the environment where the agent is located at time t;

[0084] A represents the action space, that is, at time t, the environment state is S t When , the agent selects actions according to strategy π, where strategy π refers to the mapping from state to action;

[0085] R represents the reward, that is, at time t, the environment state is S t When the agent takes action A t Enter the next state S t+1 Rewards obtained when

[0086] P represents the state transition probability, which means that the agent is given a previous state S at time t. t ∈S and action A t ∈A, the probability of occurrence of random variables s′ and r.

[0087] S2. Complete the definition of state space and action space.

[0088] Specifically include:

[0089] S2.1. First, take the initial wall climbing position of the car as the origin and construct the global coordinate system;

[0090] S2.2. On this basis, the current state data of the wall-climbing cleaning vehicle include: the coordinates (x, y, z) of the wall-climbing cleaning vehicle in the global coordinate system, the speed v of the wall-climbing cleaning vehicle in the geodetic coordinate system, the roll angle Roll, the pitch angle Pitch, the heading angle Yaw, and the preset turning and lateral displacement distance d of the wall-climbing cleaning vehicle;

[0091] S2.3, action output is: Action = [leftS, rightS, turnT];

[0092] Wherein, leftS represents the left wheel speed of the wall-climbing cleaning vehicle when turning, in radians per second;

[0093] rightS represents the right wheel speed of the wall-climbing cleaning vehicle when turning, in radians per second;

[0094] turnT represents the turning time of the wall-climbing cleaning vehicle, in seconds.

[0095] Furthermore, in order to simplify the amount of calculation, the minimum, maximum and step sizes of the left and right wheel speeds and turning times of the wall-climbing cleaning vehicle are set to convert continuous values ​​into discrete points. At this time, the action space can be understood as a three-dimensional discrete point action space, and each point in the three-dimensional action space corresponds to different left and right wheel speeds and turning times.

[0096] S3. Construct a state transfer model of the differential steering drive system.

[0097] Construct a control model based on the deep Q network DQN framework, such as Figure 2 As shown in Figure 2, the DQN framework includes the following two key components:

[0098] 1. Policy network (Q network), estimates the Q value of each action in a given state, that is, the action-value function Q(s,a) is used to evaluate the expected value of the reward after performing action a in state s.

[0099] Second, a target network with the same structural parameters as the online network generates a target Q value.

[0100] S4. Obtain current state data and generate steering wheel speed and steering time allocation strategy through the model.

[0101] The state data is transmitted as input to the policy network, and the policy network outputs the expected value of the reward obtained when taking different actions in the current state.

[0102] S5. Execute the steering action strategy and update the status data after the action is executed.

[0103] Introduce the ∈-greedy algorithm, including:

[0104] S5.1, randomly select an action with probability ∈, that is, a random steering wheel speed and steering time;

[0105] S5.2, select the steering wheel speed and steering time strategy with the largest Q value as the action output of the current strategy with probability 1-∈;

[0106] S5.3. Execute actions and update status data S t+1 .

[0107] S6. Return rewards based on the actual execution results to judge the quality of the action strategy.

[0108] Specifically include:

[0109] Reward R(s,a)=R(str)+R(dev)+R(ec)

[0110] Among them, R(str) is the straight-line walking penalty, which is used to measure whether the car can travel in a predetermined straight line after turning.

[0111] The judgment index is the change in cleaning height or depth z after turning and walking a set cleaning distance.

[0112] R(dev) is the lateral displacement penalty, which is used to measure whether the car moves laterally by a set distance relative to its pre-turn position after turning.

[0113] The height / depth of the trolley before turning is z, and the height / depth of the trolley after turning is z′. The position offset before and after turning is z′-z.

[0114] Furthermore, in this embodiment, since the hull has a certain curvature, the two coordinate systems corresponding to the Euler angles are determined by the roll angle, pitch angle and heading angle of the wall-climbing cleaning vehicle before and after turning relative to the global coordinate system, the projections of the two coordinate systems on the corresponding plane are calculated, and the projection vectors are normalized to solve the angle θ between the two vectors before and after turning and the body relative to the turning wall.

[0115] The judgment index is the difference between the front and rear position offset z′-z and the preset steering lateral displacement d under the action of the angle θ between the front and rear vehicle body and the steering wall.

[0116] R(ec) is the energy loss penalty, which is used to measure whether the car slips and generates unnecessary energy consumption during the steering process.

[0117] The judgment indicator is the lateral slip angle, which is the angular difference between the actual driving direction and the direction of the tire.

[0118] The lateral slip angle is defined as:

[0119]

[0120] Among them, v lateral is the lateral velocity, equivalent to the velocity of the wall-climbing cleaning vehicle in the geodetic coordinate system;

[0121] v longitudinal is the longitudinal velocity, calculated using the acceleration and angular velocity data provided by the inertial unit.

[0122] S7. Model optimization through experience replay pool.

[0123] like Figure 2 As shown, specifically including:

[0124] S7.1. First, perform data sampling and convert the interaction data [S, A, R, S t+1 ] is stored in the experience replay pool. When the experience replay pool has accumulated enough samples, data in the experience replay pool is randomly extracted for training.

[0125] S7.2. At the same time, priority experience replay is introduced to improve the utilization rate of samples. The sampling probability of each sample in the experience replay pool is calculated. The calculation method of the sampling probability is:

[0126]

[0127] Among them, D j is the priority of the jth sample, D k is the priority of the kth sample, α is the priority level; when α is 0, it is uniform sampling.

[0128] S7.3. Predict the current sample through the current value (strategy) network, calculate the loss between the current value network and the target value network, and use gradient descent to minimize the loss to update the current value network parameters. After a certain number of iterations, synchronize the parameters of the current value network to the target value network.

[0129] The value function update formula is:

[0130]

[0131] Among them, α is the learning rate, γ is the reward discount factor, and w is the parameter of the corresponding network.

[0132] S8. The wall-climbing cleaning vehicle continues to move forward and presets the cleaning distance.

[0133] S9. Repeat S4-S8. After the training is completed, reset the scene and continue training until the model converges. Finally, a trained differential steering drive system model is obtained.

[0134] Specifically include:

[0135] As the training progresses, the cumulative reward of each round of training gradually increases, indicating that the model strategy is constantly improving and eventually tends to a stable value, indicating that the performance of the model in the current training environment has reached a balance point.

[0136] The loss function used to measure the mean square error between the current value network and the target value network should gradually decrease, reflecting the convergence of the model during the optimization process, and the loss stabilizes at a certain value, indicating that the model's predictions are becoming consistent with the Q-value target Q-value.

[0137] When both of the above indicators show a stable state, it can be considered that the current training has converged and the model has achieved relatively ideal performance.

[0138] The embodiment of the present invention proposes a control method for a differential steering drive system of a wall-climbing cleaning vehicle based on reinforcement learning to solve the technical problem that it is difficult to construct an accurate dynamic model and design an effective path planning design algorithm in the existing hull outer wall cleaning equipment. The method mainly includes the following steps: modeling process: modeling the differential steering drive control process of the wall-climbing cleaning vehicle as a Markov decision process, defining the state space and action space, and constructing a state transition model of the differential steering drive system; strategy generation: collecting the state data of the wall-climbing cleaning vehicle and passing it to the model to generate a corresponding steering action strategy. At the same time, returning a reward value according to the execution result of the strategy to evaluate the pros and cons of the strategy; model optimization: storing the state, action and reward data in an experience replay pool, training and improving the model through an optimization algorithm, and finally obtaining a differential steering drive system model that outputs the optimal strategy.

[0139] The present invention adopts a reinforcement learning method to optimize the steering speed and steering time strategy of the four-wheel wall-climbing cleaning vehicle, realizes the optimization control that autonomously adapts to the task requirements, can autonomously adapt to the task requirements to perform lateral displacement with a small steering radius, and converts the problem of where to go and how to go to solve the path planning into what kind of posture to adjust to meet the maximum cleaning coverage area requirements, simplifies the difficulty of path planning and control algorithms, and can effectively deal with the impact of factors such as gravity, speed interference and surface material on system performance. In addition, the present invention has a low degree of dependence on sensor accuracy, takes slippage into consideration during the training process, improves the stability of the four-wheel wall-climbing cleaning vehicle during the steering process, reduces energy consumption and reduces the wear of the magnetic wheel anti-skid cover.

[0140] The present invention constructs a state transfer model of the differential steering drive system, collects state data of the wall-climbing vehicle, and transmits it to the model, generates a corresponding steering action strategy, and simultaneously performs model optimization and strategy evaluation, thereby reducing the complexity of path planning and control algorithms. The reinforcement learning algorithm enables the vehicle to continuously learn and adjust its own behavior strategy through interaction with the environment. It can learn how to take actions in the environment to maximize expected returns and find the best wall-climbing path and action through continuous trial and error during the interaction between the intelligent agent and the environment. It can adapt to different surfaces and gravity conditions and cope with speed interference or changes in surface friction. It can adjust to the optimal posture to meet the demand of maximizing the cleaning area, effectively improve the intelligence level of the wall-climbing cleaning vehicle, improve the cleaning efficiency and cleaning effect of the wall-climbing cleaning vehicle, and is suitable for efficient and intelligent cleaning of the inner and outer walls of the cabin.

[0141] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0142] In addition, in the description of the present application, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0143] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A control method for a differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning, characterized in that: The following steps are involved: S1. Model establishment: the differential steering drive control process of the wall-climbing cleaning vehicle is modeled as a Markov decision process; S2, complete the definition of state space and action space; S3, constructing a state transfer model of the differential steering drive system; S4, obtaining current state data, and generating a steering wheel speed and steering time allocation strategy through a model; S5, execute the steering action strategy and update the status data after the action is executed; S6. Return rewards based on actual execution results to judge the quality of the action strategy; S7, model optimization through experience replay pool; S8, the wall-climbing cleaning vehicle continues to move forward and presets the cleaning distance; S9. Repeat S4-S8. After the training is completed, reset the scene and continue training until the model converges. Finally, a trained differential steering drive system model is obtained.

2. The control method of the differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning according to claim 1 is characterized in that: In S1, a tuple [S, A, R, P] is used to represent the Markov decision process of the differential steering drive of the wall-climbing cleaning vehicle, and an initial action strategy π is given, where: S represents the state space, that is, the state information of the environment where the agent is located at time t; A represents the action space, that is, at time t, the environment state is S t When , the agent selects actions according to strategy π, where strategy π refers to the mapping from state to action; R represents the reward, that is, at time t, the environment state is S t When the agent takes action A t Enter the next state S t+1 Rewards obtained when P represents the state transition probability, which means that the agent is given a previous state S at time t. t ∈S and action A t ∈A, the probability of occurrence of random variables sˊ and r.

3. The control method of the differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning according to claim 1 is characterized in that S2 include: S2.

1. Construct a global coordinate system with the initial wall climbing position of the car as the origin; S2.2, the current state data of the wall-climbing cleaning vehicle includes: the coordinates (x, y, z) of the wall-climbing cleaning vehicle in the global coordinate system, the speed v of the wall-climbing cleaning vehicle in the geodetic coordinate system, the roll angle Roll, the pitch angle Pitch, the heading angle Yaw, and the preset turning and lateral displacement distance d of the wall-climbing cleaning vehicle; S2.3, action output is: Action = [leftS, rightS, turnT]; Among them, leftS represents the left wheel speed of the wall-climbing cleaning vehicle when turning, radian / second, rightS represents the right wheel speed of the wall-climbing cleaning vehicle when turning, radian / second, and turnT represents the turning time of the wall-climbing cleaning vehicle, seconds.

4. The control method of the differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning according to claim 3 is characterized in that: The minimum, maximum and step sizes of the left and right wheel speeds and turning time of the wall-climbing cleaning vehicle are set to convert continuous values ​​into discrete points. At this time, the action space can be understood as a three-dimensional discrete point action space, and each point in the three-dimensional action space corresponds to different left and right wheel speeds and turning times.

5. The control method of the differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning according to claim 1 is characterized in that: In S3, a control model based on the deep Q network DQN framework is constructed. The DQN framework includes: The policy network estimates the Q value of each action in a given state, that is, the action-value function Q(s,a) is used to evaluate the expected value of the reward after performing action a in state s; Target network, a target network with the same structural parameters as the online network generates the target Q value.

6. The control method of the differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning according to claim 1 is characterized in that: In S4, the state data is transmitted as input to the policy network, and the policy network outputs the expected value of the reward obtained when taking different actions in the current state.

7. The control method of the differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning according to claim 1 is characterized in that: In S5, the ∈-greedy algorithm is introduced, including: S5.1, randomly select an action with probability ∈, that is, a random steering wheel speed and steering time; S5.2, select the steering wheel speed and steering time strategy with the largest Q value as the action output of the current strategy with probability 1-∈; S5.

3. Execute actions and update status data S t+1 .

8. The control method of the differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning according to claim 1 is characterized in that S6 include: Reward R(s,a)=R(str)+R(dev)+R(ec) Among them, R(str) is the straight-line walking penalty, which is used to measure whether the car can travel in a predetermined straight line after turning; The judgment index is the change in cleaning height or depth z after turning and walking a set cleaning distance; R(dev) is the lateral displacement penalty, which is used to measure whether the car has moved laterally by a set distance relative to its position before turning after turning. The height / depth of the trolley before turning is z, the height / depth of the trolley after turning is z′, and the position offset before and after turning is z′-z; The two coordinate systems corresponding to the Euler angles are determined by the roll angle, pitch angle and heading angle of the wall-climbing cleaning vehicle before and after turning relative to the global coordinate system, the projections of the two coordinate systems on the corresponding planes are calculated, and the projection vectors are normalized to solve the angle θ between the two vectors before and after turning and the vehicle body relative to the turning wall; The judgment index is the difference between the front and rear position offset z′-z and the preset steering lateral displacement d under the action of the angle θ between the front and rear body and the steering wall; R(ec) is the energy loss penalty, which is used to measure whether the car slips and generates unnecessary energy consumption during the steering process; The judgment index is the lateral slip angle, which is the angular difference between the actual driving direction and the direction of the tire; The lateral slip angle is defined as: Among them, v lateral is the lateral velocity, equivalent to the velocity of the wall-climbing cleaning vehicle in the geodetic coordinate system; v longitudinal is the longitudinal velocity, calculated using the acceleration and angular velocity data provided by the inertial unit.

9. The control method of the differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning according to claim 1 is characterized in that: S7 includes: S7.

1. Perform data sampling and convert the interaction data [S, A, R, S t+1 ] is stored in the experience replay pool. When the experience replay pool has accumulated enough samples, data in the experience replay pool is randomly extracted for training; S7.

2. Introduce priority experience replay to improve sample utilization and calculate the sampling probability of each sample in the experience replay pool. The sampling probability is calculated as follows: Among them, D j is the priority of the jth sample, D k is the priority of the kth sample, and α is the priority degree; S7.3, predict the current sample through the current value network, calculate the loss between the current value network and the target value network, and use gradient descent to minimize the loss to update the current value network parameters. After a certain number of iterations, synchronize the parameters of the current value network to the target value network; The value function update formula is: Among them, α is the learning rate, γ is the reward discount factor, and w is the parameter of the corresponding network.

10. The control method of the differential steering drive system of a crawling and cleaning vehicle based on reinforcement learning according to claim 1 is characterized in that: S9 includes: As training progresses, the cumulative reward of each round of training gradually increases, indicating that the model strategy is constantly improving and eventually tends to a stable value, indicating that the performance of the model in the current training environment has reached a balance point; The loss function used to measure the mean square error between the current value network and the target value network should gradually decrease, reflecting the convergence of the model during the optimization process. The loss stabilizes at a certain value, indicating that the model's predictions are consistent with the Q-value target Q-value. When both indicators show a stable state, it can be considered that the current training has converged and the model has achieved relatively ideal performance.

Images