Multi-agent weeding operation planning method for non-flat photovoltaic field

Abstract

The invention discloses a multi-agent weeding operation planning method for a non-flat photovoltaic field. The method comprises the steps that parameter configuration is conducted on agents, and observation information is collected; designing a window type hierarchical collaborative A * algorithm to carry out path planning on multiple agents; designing a trajectory optimization scheme based on a deep neural Q network DQN; applying a cluster-based obstacle avoidance rule to multi-agent obstacle avoidance; integrating trajectory planning and obstacle avoidance of the agents by using a model prediction control algorithm, and performing prediction to complete multi-agent weeding operation planning. According to the method, three algorithms of model predictive control, window type collaborative A * and deep neural Q network are fused. According to the method, the problems of path conflict, dynamic environment adaptation and cooperation efficiency of multiple agents in a complex terrain are effectively solved, the success rate and efficiency of multi-agent weeding operation are remarkably improved, and the cost is reduced.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent path planning, and in particular relates to a multi-agent weeding operation planning method for non-flat photovoltaic sites. Background Technology

[0002] Photovoltaic sites are large in scale, covering a wide area, and equipped with a large number of photovoltaic panels. These panels are closely arranged, forming a unique grid-like spatial structure, with certain spacing between them to facilitate inspection and operation. However, the terrain is mostly uneven, including various undulating slopes and irregular landforms, which complicates the planning of movement and operations within the site. Furthermore, in uneven photovoltaic sites, weed growth can affect the light-gathering efficiency of the photovoltaic panels, reducing power generation, thus requiring weeding operations.

[0003] However, current path planning methods, such as the traditional A* algorithm and Dijkstra's algorithm, are mainly suitable for single planning problems in flat environments. They have limitations in multi-agent scenarios on non-flat photovoltaic sites, making it difficult for multiple agents to cooperate, easily leading to path conflicts between agents, and lacking adaptability to non-flat terrain, failing to adequately consider the influence of site slope, topography, and other factors on the kinematic and dynamic characteristics of agents. Traditional integrated methods for autonomous vehicle path planning and obstacle avoidance include the Dijkstra algorithm and the ORCA algorithm. The Dijkstra algorithm is only suitable for static environments; when encountering complex environments, such as dynamic obstacles, it has poor adaptability, cannot effectively avoid obstacles, and has low path planning efficiency in large maps. Summary of the Invention

[0004] Objective: This invention aims to provide a multi-agent weeding operation planning method for non-flat photovoltaic sites. It employs a model predictive control (MMC) algorithm, a window-based hierarchical collaborative A* algorithm, and a deep neural Q-network algorithm. The MMC algorithm utilizes the system's dynamic model to solve the finite-time optimal control problem online at each control moment, predicting future trajectories in real time based on the agent's kinematic model and current state. The window-based hierarchical collaborative A* algorithm decomposes multi-agent path planning into short-term and long-term tasks, efficiently handling path coordination and conflicts between agents by maintaining a dynamic window, thus improving overall planning efficiency. The deep neural Q-network algorithm learns through agent-environment interaction, using agent position and surrounding environment as input states and action decisions as output actions. It evaluates decision effectiveness using a reward function. After extensive training, the agent can learn the optimal strategy in complex, non-flat photovoltaic site environments, providing reasonable action decisions for different scenarios while considering the long-term benefits of multi-agent collaboration.

[0005] Technical solution: The present invention provides a multi-agent weeding operation planning method for non-flat photovoltaic sites, comprising the following steps:

[0006] Step 1: Configure the parameters of the agent and collect observation information;

[0007] Step 2: Design a window-based hierarchical collaborative A* algorithm for path planning among multiple agents;

[0008] Step 3: Design a trajectory optimization scheme based on deep neural Q-network (DQN);

[0009] Step 4: Apply cluster-based obstacle avoidance rules to multi-agent obstacle avoidance;

[0010] Step 5: Integrate the trajectory planning and obstacle avoidance of the agents using model predictive control algorithms, and make predictions to complete the multi-agent weeding operation planning.

[0011] Further, step 1 specifically involves: denoting the total cost of reaching the target from the starting point as J(t,x,y), and denoting the cost as J; assuming 4-connectivity exists in the space, the agent moves to various positions by taking action a from time t=0 to t=T; and incurring a cost j during state transitions. a And incurs time cost j between time intervals. t Time step and movement cost between units in space j s The space is defined as a discretized three-dimensional grid in spacetime; each cell represents the position (x, y) at time t; the coordinates of each agent are (x, y), and the coordinates of the obstacles are... Assuming that both the obstacle and the target move at a constant speed, use dynamic programming to calculate the cost for each cell.

[0012] t∈{0,Δt,2Δt,...,T}

[0013] x∈{x min ,x min +Δx,...,x max}

[0014] y∈{y min ,y min +Δy,...,y max}

[0015] (x,y) t+1 ←(x,y) t +a

[0016] a∈{(-Δx,0),(Δx,0),(0,-Δy),(0,Δy),(0,0)}

[0017]

[0018] J(t,x,y)=min a (J(t+1,x,y)+j a )

[0019] Where Δt refers to the time interval between t = 0 and t = T; x min The x-coordinate refers to the x-coordinate at time t = 0, and Δx refers to the distance the agent moves in the x-direction during the time interval Δt. max It refers to the x-coordinate at time t = T; y-coordinate min The y-coordinate refers to the position of the agent at time t = 0, and Δy refers to the distance the agent moves in the y-direction during the time interval Δt. max This refers to the y-coordinate at time t = T; the coordinates of the obstacle. middle These refer to the first derivatives in the x and y directions, respectively, i.e., velocity; R refers to the additional safety distance between the artificially set obstacle and the intelligent agent.

[0020] Furthermore, step 2 specifically includes the following steps:

[0021] Step 2.1: Use a linear quadratic regulator to track the desired state and obtain the feedback gain matrix;

[0022] First, define the state matrix A, the input matrix B, the state weight matrix Q, and the control input weight matrix R; Q and R are positive definite matrices.

[0023] Then, the positive definite cost function matrix P is calculated using the Riccati equation:

[0024] A T P+PA-PBR -1 B T P+Q=0

[0025] Finally, the feedback gain matrix K is calculated:

[0026] K = R -1 B T P

[0027] Step 2.2: A single agent plans a path using a window-based hierarchical collaborative A* algorithm and a feedback gain matrix;

[0028] The optimal path for a single agent is planned using a window-based hierarchical collaborative A* algorithm, given a point (t). n The next point on the optimal path is (t, x, y). n+1 ,x * ,y * );

[0029]

[0030] Among them, t n+1 Indicates the next time period; x * ,y * This indicates that at time t n+1 The optimal position coordinates at time; argmin represents finding the variable value that minimizes the objective function; J(t) n+1 (x′, y′) represents the time interval t. n+1 The cost of position (x′, y′) at time t; (x′, y′) represents the cost at time t. n+1 When selecting a position coordinate, (x, y) represents the current position coordinate;

[0031] Step 2.3: Randomly assign agent levels and use the window-based hierarchical collaborative A* algorithm to perform path planning for multiple agents in sequence. Periodically and randomly change the agent level. The agent with the highest level plans the path first and adds it to the relevant cost. The paths planned by subsequent agents will not conflict with those of agents with higher levels.

[0032] Furthermore, step 3 specifically includes the following steps:

[0033] Step 3.1: Based on the Bellman equation, derive the expected cumulative reward for each state-action pair to obtain the optimal policy;

[0034] Step 3.2: Deep neural Q-networks approximate the state-action function by training deep neural networks to estimate the expected cumulative reward;

[0035] Step 3.3: Use two reward schemes, independent reward and shared reward, to reward and punish agents. Independent reward means that each agent receives its own reward, while shared reward means that all agents receive the same reward, which is composed of the average of the independent rewards.

[0036] Furthermore, step 3.1 specifically involves: the optimality criterion being determined by the defined reward function R; each state-action pair derives a value Q(s,a), where the value function represents the cumulative reward the agent expects to obtain when taking action a in state S; and the value of the Q function for the optimal policy π is given by the Bellman equation.

[0037]

[0038] Where Q(s,a) represents the state-action value function, which represents the cumulative reward expected by the agent after performing action a in state s; r(s,a) represents the reward function, which represents the immediate reward obtained by the agent when performing action a in state s; γ represents the discount factor, which ranges from [0,1] and is used to measure the present value of future rewards. The closer the value is to 1, the more the agent values ​​future rewards; the closer the value is to 0, the more the agent values ​​immediate rewards; p(s′|s,a) represents the state transition probability, which represents the probability of transitioning to state s′ after performing action a in state s; max a′∈A Q(s′,a′) represents the maximum Q value that the agent can obtain in the next state s′, which is the maximum expected cumulative reward that the agent can obtain in state s′ under the optimal policy; a′ is the action taken in state s′.

[0039] For a given Q function, the optimal strategy is as follows:

[0040]

[0041] Furthermore, step 3.2 specifically involves: the deep neural Q-network approximating the function to estimate the Q-value by training a deep neural network, using the following loss function:

[0042]

[0043] in Let e ​​represent the expectation of all states, actions, rewards, and the next state, i.e., the average of all samples. The Bellman error e is:

[0044] e = r(s,a) + γmax a′ Q(s′,a′;θ - )-Q(s,a,θ)

[0045] Where, γmax a Q(s′,a′;θ - ) indicates the use of target network parameters θ - The present value of the future reward obtained by multiplying the maximum Q value of the next state s′ by the discount factor γ, θ - The parameters of the target network are used to estimate the target Q-value and are periodically updated from the parameters θ of the current network to stabilize the training process; Q(s,a,θ) represents the Q-value of the current state s and action a calculated using the current network parameters θ; θ represents the parameters of the current Q network, used to estimate the Q-value in the current state.

[0046] For the last s t+1 Calculate B t =r t For other s t+1 Calculate Bt =r t +γmax a′ Q(s t+1 ,a t+1 ;θ - ); where r t Indicates that in state s t Take action a t The reward received later; B t Indicates that in state s t Take action a t The expected cumulative reward; the maximum value Q represents the agent's reward in state s. t+1 The best expected cumulative reward that can be obtained is a t+1 In state s t+1 The actions taken below.

[0047] Furthermore, step 4 specifically includes the following steps:

[0048] Step 4.1: Set the position and speed of the agent relative to surrounding obstacles, and set the safety radius;

[0049] Step 4.2: The agent generates a virtual agent and sets its state, considering the radius of the agent and obstacles as neighbors;

[0050] Step 4.3: When the agent encounters an obstacle with a radius that is too small, that is, when the distance between the agent and the obstacle is less than the sum of the radii of the neighboring obstacles, send the desired acceleration command to the agent to initiate an obstacle avoidance response.

[0051] Furthermore, step 4.2 specifically involves: Let q i and p i Let q be the position and velocity of agent i, respectively. k and p k Let R be the position and velocity of the obstacle surrounding agent i, and let R be the radius. k Agent i generates a virtual agent, whose state is: On the boundary of the obstacle or agent k, and maintaining contact with that agent, let D... a With D o The radius is considered as the nearest radius for the agent and the obstacle, respectively.

[0052]

[0053] μ is a scaling factor that ensures the virtual agent lies on the boundary; P is a projection matrix with a direction from q k Pointing to q i The direction of the outward normal of the unit normal vector;

[0054]

[0055] ‖·‖ represents the norm, a function used to measure the length or size of a vector;

[0056] When a virtual agent is generated on the environment boundary, the virtual agent is located at the point on the environment boundary closest to agent i, q k It is any point on the boundary of the environment, and I refers to the identity matrix;

[0057]

[0058] For agent i, the set of nearby virtual agents generated by other agents is (the set of virtual agents in the vicinity generated by other agents). The set of nearby virtual agents generated by obstacles is The repulsive force exerted by nearby virtual agents on agent i is This force is sent to the agent as a desired acceleration command;

[0059]

[0060] and and The relevant terms will exclude agent i from nearby virtual agents, while... and The related terms will produce a velocity damping effect, φ β It is a repulsive force function. It is a normal vector that has been normalized by σ, b i,k It is a heterogeneous adjacency term between intelligent agent i and its virtual intelligent agent k. and These are positive constants used to adjust the repulsive force and velocity damping between agents. Since obstacles behave differently from agents, different scaling constants are used for the repulsive force generated by obstacles. and

[0061] Furthermore, step 5 specifically includes the following steps:

[0062] Step 5.1: Set the time step and use the environment model to predict the state within the future time step;

[0063] Step 5.2: Use model predictive control algorithms to integrate path planning and obstacle avoidance;

[0064] Step 5.3: Each agent activates only one component at a time.

[0065] Furthermore, in step 5.2, the model predictive control algorithm specifically comprises:

[0066] Input: Cost J, agent, target and obstacle locations;

[0067] Output: Control input set {u1, u2, ... u N};

[0068] Within the time range T p In Chinese, it is used to predict future states;

[0069] Agent i cycles through numbers from 1 to N:

[0070] If the obstacle avoidance planner is active

[0071] Apply obstacle avoidance rules to u i superior;

[0072] In other cases, add agent i to the priority sequence. middle;

[0073] End the loop;

[0074] All non-sequence The predicted state of the agent in the process is added to J;

[0075] For in sequence Multi-agent path planning is performed on the intelligent agents in the system.

[0076] In priority sequence Agent i in the loop begins:

[0077] Use LQR to control and trace path p i And calculate u i ;

[0078] End the loop;

[0079] The result is {u1, u2, ... u}. N}

[0080] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: This invention proposes a multi-agent weeding operation planning method for non-flat photovoltaic sites, integrating three algorithms: Model Predictive Control (MMCC), Window-based Collaborative A*, and Deep Neural Q-Network. This method effectively solves the problems of path conflict, dynamic environment adaptation, and collaborative efficiency of multi-agent operations in complex terrain, significantly improving the success rate and efficiency of multi-agent weeding operations while reducing costs. This invention utilizes the Model Predictive Control algorithm, which can effectively avoid dynamic obstacles. This method is robust and can handle nonlinear models and real-time disturbances. The ORCA algorithm constructs velocity obstacles for each autonomous vehicle and then searches for collision-free velocity vectors in the velocity space. To avoid local minima, this algorithm does not perform long-term planning. Therefore, when there are many static obstacles, the obstacle avoidance accuracy of this algorithm is low. This invention, utilizing the Model Predictive Control algorithm, can maintain a high accuracy even with many static obstacles. When the ORCA algorithm encounters dynamic obstacles, its processing speed is slow. The Model Predictive Control algorithm can solve problems quickly and perform accurate obstacle avoidance. Attached Figure Description

[0081] Figure 1 This is a flowchart of the present invention;

[0082] Figure 2 This is the network architecture of the deep neural network trained by the deep neural Q-network algorithm of the present invention;

[0083] Figure 3 This is a comparison diagram between two different reward schemes in this invention;

[0084] Figure 4 This is the multi-agent path planning diagram in this invention. Detailed Implementation

[0085] The technical solution of the present invention will be further described below with reference to the accompanying drawings. Figure 1 As shown, this invention provides an improved weeding technology for intelligent agent swarms based on model predictive control algorithms and deep neural Q-networks, including the following steps:

[0086] (1) Set the parameters of the intelligent agent system

[0087] The total cost of reaching the goal from the starting point is denoted as J(t,x,y), and will be expressed as J. Assuming 4-way connectivity exists in the space, the agent moves to various locations by taking action a from time t=0 to t=T. Cost j is incurred during state transitions. a Furthermore, time costs j will be incurred between time intervals. t The time step and the movement cost j between units in space. sSpace is defined as a discretized three-dimensional grid in spacetime. Each cell represents the position (x, y) at time t. The coordinates of each agent are (x, y), and the coordinates of obstacles are... Assuming that both obstacles and targets move at a constant speed, dynamic programming can be used to calculate the cost for each cell.

[0088] t∈{0,Δt,2Δt,...,T}(1)

[0089] x∈{x min ,x min +Δx,...,x max}(2)

[0090] y∈{y min ,y min +Δy,...,y max}(3)

[0091] (x,y) t+1 ←(x,y) t +a(4)

[0092] a∈{(-Δx,0),(Δx,0),(0,-Δy),(0,Δy),(0,0)}(5)

[0093]

[0094] J(t,x,y)=min a (J(t+1,x,y)+j a (7)

[0095] Where Δt refers to the time interval between t = 0 and t = T. min The x-coordinate refers to the x-coordinate at time t = 0, and Δx refers to the distance the agent moves in the x-direction during the time interval Δt. max This refers to the x-coordinate at time t = T. y min The y-coordinate refers to the position of the agent at time t = 0, and Δy refers to the distance the agent moves in the y-direction during the time interval Δt. max This refers to the y-coordinate at time t = T. (The coordinates of the obstacle are also mentioned.) middle These refer to the first derivatives in the x and y directions, respectively, i.e., velocities. R refers to the additional safe distance between the artificially placed obstacle and the agent.

[0096] (2) Path planning for multi-agent systems based on window-based hierarchical collaborative A* algorithm

[0097] The optimal path for a single agent is planned using a window-based hierarchical collaborative A* algorithm. Given a point (t) nThe next point on the optimal path is (t, x, y). n+1 ,x * ,y * ).

[0098]

[0099] t n+1 Indicates the next time period. x * ,y * This indicates that at time t n+1 The optimal position coordinates at time. argmin represents finding the variable values ​​that minimize the objective function. J(t) n+1 (x′, y′) represents the time interval t. n+1 The cost at position (x′, y′) at time t. (x′, y′) represents the cost at time t. n+1 When selecting a position coordinate, (x,y) represents the current position coordinate.

[0100] To follow the path points generated by the planner, a smooth trajectory is generated using cubic spline interpolation, connecting these path points. By evaluating the cubic spline and its derivative over time t, the desired state of its position, velocity, and acceleration can be calculated.

[0101] (2.1) Using a linear quadratic regulator to track the desired state

[0102] First, define the state matrix A, the input matrix B, the state weight matrix Q, and the control input weight matrix R. Generally, Q and R are positive definite matrices.

[0103] Then, the positive definite cost function matrix P is calculated using the Riccati equation:

[0104] A T P+PA-PBR -1 B T P+Q=0 (9)

[0105] Finally, the feedback gain matrix K is calculated:

[0106] K = R -1 B T P (10)

[0107] (2.2) Path planning based on a single agent

[0108] This invention starts with a single-agent solution and then uses the single-agent solution as a heuristic method for multi-agent planning, in which agents plan sequentially and update the constraints of the search problem for subsequent agents.

[0109] To address the single-agent path planning problem, a window-based hierarchical cooperative A* algorithm is proposed, the specific process of which is as follows:

[0110]

[0111] Through x t+1 =Ax t +Bu t Calculate the next point x t+1 Thus, the path p is obtained.

[0112] Where p, These refer to the path, and the velocity and acceleration at each point along that path. p y p These refer to the path, velocity, and acceleration in the x and y coordinates, respectively.

[0113] (2.3) Multi-agent-based path planning

[0114] To account for multiple agents, these agents are assigned ranks, and path planning is performed sequentially. The highest-ranking agent plans the path first and adds it to the set J, with subsequent agents following suit. Therefore, the paths planned by later agents will not conflict with those planned by higher-ranking agents. To reduce computation time, agents only plan up to the time window t. w The path within. Time window t w This refers to the future timeframe that the agent considers when planning a path. By limiting the time window, computational complexity can be reduced, and the agent can be ensured to plan paths within a reasonable timeframe.

[0115] The agent's hierarchy is randomly assigned and periodically changed to minimize deadlock. To address the multi-agent path planning problem, a multi-agent window-based hierarchical cooperative A* algorithm is proposed, the specific process of which is as follows:

[0116]

[0117]

[0118] (3) Find the optimal strategy π

[0119] The optimality criterion is determined by the defined reward function R. Each state-action pair yields a value Q(s,a). This value function represents the cumulative reward the agent expects to receive when taking action a in state S. The value of the Q function for the optimal policy π is given by the Bellman equation:

[0120]

[0121] Where Q(s,a) represents the state-action value function, indicating the cumulative reward the agent expects to receive after performing action a in state s. r(s,a) represents the reward function, indicating the immediate reward the agent receives when performing action a in state s. γ represents the discount factor, ranging from [0,1], used to measure the present value of future rewards. The closer the value is to 1, the more the agent values ​​future rewards; the closer the value is to 0, the more the agent focuses on immediate rewards. p(s′|s,a) represents the state transition probability, indicating the probability of transitioning to state s′ after performing action a in state s. a′∈A Q(s′,a′) represents the maximum Q value that the agent can obtain in the next state s′, which is the maximum expected cumulative reward that the agent can obtain in state s′ under the optimal policy. a′ is the possible action to take in state s′.

[0122] For a given Q function, the optimal strategy is as follows:

[0123]

[0124] Deep neural Q-networks approximate this function by training a deep neural network to estimate the Q-value. The training uses the following loss function:

[0125]

[0126] in Let e ​​represent the expectation over all possible states, actions, rewards, and the next state; that is, the average of all possible samples. The Bellman error e is:

[0127] e = r(s,a) + γmax a′ Q(s′,a′;θ - )-Q(s,a,θ) (14)

[0128] Where γmax a Q(s′,a′;θ - ) indicates the use of target network parameters θ - The present value of the future reward is obtained by multiplying the maximum Q value of the next state s′ by the discount factor γ. θ - The parameters of the target network, θ, are used to estimate the target Q-value and are periodically updated from the current network's parameters θ to stabilize the training process. Q(s,a,θ) represents the Q-value of the current state s and action a calculated using the current network parameters θ. θ represents the parameters of the current Q-network, used to estimate the Q-value in the current state.

[0129] To achieve the optimal policy π, a deep neural Q-network algorithm is proposed, the specific process of which is as follows:

[0130]

[0131]

[0132] For the last s t+1 Calculate B t =r t For other s t+1 Calculate B t =r t +γmax a′ Q(s t+1 ,a t+1 ;θ - ).

[0133] Where r t Indicates that in state s t Take action a t The reward received later. B t Indicates that in state s t Take action a t The expected cumulative reward. This maximum value Q represents the agent's reward in state s. t+1 The best expected cumulative reward that can be obtained. t+1 In state s t+1 The following actions may be taken.

[0134] Some traditional methods, such as the A* algorithm and Dijkstra's algorithm, assume a static environment. Once the environment changes, such as the appearance of new obstacles or a change in the target's position, path planning needs to be re-executed, making it unable to react quickly in real-time to dynamic environments. This invention uses a deep neural Q-network algorithm. Unlike traditional methods, this method learns through continuous interaction with the environment via a deep neural network, enabling it to adapt to dynamically changing environments. For example, during the agent's driving process, when facing other dynamic obstacles, the deep neural Q-network algorithm can automatically adjust its strategy based on the new environmental state, selecting the optimal obstacle avoidance and path planning scheme without requiring manual path replanning.

[0135] (4) Collect observation information

[0136] Each agent collects feature vectors representing its own local state and that of another agent, along with a local observation image of the grassland from the agent's perspective. Each agent's feature vector is itself composed of four continuous variable vectors: all turning angles {φ}. j |j∈1,2,...,N}=φ, the distance {ρ} between j∈1,2,...,N} and other agents. ij ∣j∈1,2,...,N∧j≠i}=ρ i The relative heading angle {θ} between the agent and other agents ij ∣j∈1,2,...,N∧j≠i}=θ iAnd the relative heading angles {ψ} of other intelligent agents ij ∣j∈1,2,...,N∧j≠i}=ψ i .

[0137] (5) Use two reward schemes, independent reward and shared reward, to reward and punish agents.

[0138] The reward for each agent i consists of four different parts:

[0139]

[0140]

[0141] Independent rewards mean that each agent receives its own reward. Shared rewards mean that all agents receive the same reward, which is the average of the independent rewards. Choosing this scheme means that agents should be severely penalized for straying too far from the weeding area, or for making excessively large turns or getting too close to other agents.

[0142] Where r1 refers to the distance from the weeding area, which is proportional to the distance to the nearest cell that needs weeding;

[0143] r2 refers to the number of nearby dry cell batteries, which is directly proportional to the number of dry cell batteries within the radius.

[0144] r3 refers to the high turning angle, which is proportional to the square of the turning angle;

[0145] r4 refers to the proximity to other agents, which is the sum of the contributions of each other agent, and its saturation value is λ4.

[0146] The tuning parameters have been set to the following values: λ1=10, λ2=1, λ3=100, λ4=1000, r0=10, c=100.

[0147] F t (s) represents the state at time t where position s in the environment is excessively tall with weeds. This function describes the changes in weed height, helping the agent understand the current weed distribution in the environment. In the reward function of reinforcement learning, F... t (s) is used to calculate rewards or penalties related to weeds. Specifically: if F t If (s) = 1, it means that the weeds are too tall at position s at time t. If F t If (s) = 0, it means that the weeds are not tall at position s at time t. s This represents the distance between the agent and position s, which is the Euclidean distance between the agent's current position and position s.

[0148] (6) Perform obstacle avoidance operations.

[0149] Let q i and p i Let q be the position and velocity of agent i, respectively. k and p k Let R be the position and velocity of the obstacle surrounding agent i, and let R be the radius. k Agent i generates a virtual agent, whose state is... On the boundary of the obstacle or agent k, maintain a certain distance from the agent to avoid collision. Let D a With D o The radius is considered as the nearest radius for the agent and the obstacle, respectively.

[0150]

[0151]

[0152] μ is a scaling factor that ensures the virtual agent lies on the boundary. P is a projection matrix with direction from q. k Pointing to q i The direction of the outward normal of the unit normal vector.

[0153]

[0154] ‖·‖ represents the norm, a function used to measure the length or size of a vector, and is generally used to calculate the Euclidean distance.

[0155] Virtual agents can also be generated on the environment boundary, in which case they are represented as hyperplanes. In this case, the relevant virtual agent is located at the point on the environment boundary closest to agent i. k It refers to any point on the boundary of the environment. I refers to the identity matrix.

[0156]

[0157] For agent i, the set of nearby virtual agents generated by other agents is (the set of virtual agents in the vicinity generated by other agents). The set of nearby virtual agents generated by obstacles is The repulsive force exerted by nearby virtual agents on agent i is This force is sent to the agent as a command for the desired acceleration.

[0158]

[0159] and and The relevant terms will exclude agent i from nearby virtual agents, while... and The related terms will produce a velocity damping effect. φ β It is a function of repulsive force. It is a normal vector that has been normalized by σ. i,k It is a heterogeneous adjacency term between intelligent agent i and its virtual intelligent agent k. and These are positive constants used to adjust the repulsive force and velocity damping between agents, respectively. Since obstacles behave differently from agents, different scaling constants are used for the repulsive force generated by obstacles. and

[0160] (7) Integrate path planning and obstacle avoidance

[0161] The model employs a constant velocity model for both the target and obstacles. If the agent does not encounter an obstacle within the predicted time, it continues to optimize the path. Otherwise, the agent follows the trajectory generated by the planner. To prevent the path generated by the planner from getting too close to obstacles and triggering collision avoidance rules, an additional safety distance R is added to the obstacles. + .

[0162] To integrate path planning and obstacle avoidance, a model predictive control algorithm is proposed, the specific process of which is as follows:

[0163]

[0164] Traditional methods for integrating path planning and obstacle avoidance technologies for autonomous vehicles include Dijkstra's algorithm and ORCA algorithm. Dijkstra's algorithm is only suitable for static environments; it has poor adaptability and cannot effectively avoid obstacles in complex environments, such as dynamic obstacles, and its path planning efficiency is low in large maps. This invention utilizes a model predictive control algorithm, which can effectively avoid dynamic obstacles as well. This method is robust and can handle nonlinear models and real-time disturbances. The ORCA algorithm constructs velocity obstacles for each autonomous vehicle and then searches for collision-free velocity vectors in the velocity space. To avoid local minima, this algorithm does not perform long-term planning. Therefore, when there are many static obstacles, its obstacle avoidance accuracy is low. This invention, utilizing a model predictive control algorithm, can maintain high accuracy even with many static obstacles. When ORCA encounters dynamic obstacles, its processing speed is slow. The model predictive control algorithm can solve the problem quickly and perform accurate obstacle avoidance.

[0165] Figure 2The architecture of a deep Q-network is shown. The neural network processes the two types of observations separately. Feature data passes through a series of fully connected layers with ReLU activation functions. The image, on the other hand, first passes through a series of convolutional and max-pooling layers to efficiently reduce its dimensionality before entering its own fully connected layer. The outputs of the two networks are then concatenated and passed through two additional fully connected layers.

[0166] Figure 3 This section presents a comparison of different reward schemes. The primary performance metric for this model is cumulative reward, which is the sum of the immediate rewards received by all agents in a single round. Both reward configurations allow agents to learn and average out their penalties. When an agent approaches a weeding area, it will drive towards it and move closer, but due to penalties for excessive turning angles, it cannot make small circles within the same area. Instead, the agent will cross the weeding area while maintaining a certain distance from other agents. When the area of ​​interest is too small, the penalty for approaching agents outweighs other factors, causing one agent to remain stationary and make small circles.

[0167] Figure 4 A visualization of multi-agent path planning is presented. This diagram illustrates the visualization effect of hierarchical planning. Each subsequent agent uses the paths of previous agents as constraints when planning its own path. The path generated by the planner focuses on reaching the goal. If an agent's path has no obstacles or obstructions from other agents, the planner is not used. Paths generated by control policies should be added to replace the planner's paths.

Claims

1. A multi-agent weeding operation planning method for non-flat photovoltaic sites, characterized in that, Includes the following steps: Step 1: Configure the parameters of the agent and collect observation information; Step 2: Design a window-based hierarchical collaborative A* algorithm for path planning among multiple agents; Step 3: Design a trajectory optimization scheme based on deep neural Q-network (DQN); Step 4: Apply cluster-based obstacle avoidance rules to multi-agent obstacle avoidance; Step 5: Integrate the trajectory planning and obstacle avoidance of the agents using model predictive control algorithms, and make predictions to complete the multi-agent weeding operation planning.

2. The multi-agent weeding operation planning method for non-flat photovoltaic sites according to claim 1, characterized in that, Step 1 is as follows: The total cost of reaching the target from the starting point is denoted as J(t,x,y), and is further denoted as J; Assume there is 4-way connectivity in the space, and the agent moves to various positions by taking action a from time t=0 to t=T; a cost j is generated during state transitions. a And incurs time cost j between time intervals. t Time step and movement cost between units in space j s The space is defined as a discretized three-dimensional grid in spacetime; each cell represents the position (x, y) at time t; the coordinates of each agent are (x, y), and the coordinates of the obstacles are... Assuming that both the obstacle and the target move at a constant speed, use dynamic programming to calculate the cost for each cell. t∈{0,Δt,2Δt,…,T} x∈{x min ,x min +Δx,…,x max } y∈{y min ,y min +Δy,…,y max } (x,y) t+1 ←(x,y) t +a a∈{(-Δx,0),(Δx,0),(0,-Δy),(0,Δy),(0,0)} J(t,x,y)=min a (J(t+1,x,y)+j a ) Where Δt refers to the time interval between t = 0 and t = T; x min The x-coordinate refers to the x-coordinate at time t = 0, and Δx refers to the distance the agent moves in the x-direction during the time interval Δt. max It refers to the x-coordinate at time t = T; y-coordinate min The y-coordinate refers to the position of the agent at time t = 0, and Δy refers to the distance the agent moves in the y-direction during the time interval Δt. max This refers to the y-coordinate at time t = T; the coordinates of the obstacle. middle These refer to the first derivatives in the x and y directions, respectively, i.e., velocity; R refers to the additional safety distance between the artificially set obstacle and the intelligent agent.

3. The multi-agent weeding operation planning method for non-flat photovoltaic sites according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Use a linear quadratic regulator to track the desired state and obtain the feedback gain matrix; First, define the state matrix A, the input matrix B, the state weight matrix Q, and the control input weight matrix R; Q and R are positive definite matrices. Then, the positive definite cost function matrix P is calculated using the Riccati equation: A T P+PA-PBR -1 B T P+Q=0 Finally, the feedback gain matrix K is calculated: K=R -1 B T P Step 2.2: A single agent plans a path using a window-based hierarchical collaborative A* algorithm and a feedback gain matrix; The optimal path for a single agent is planned using a window-based hierarchical collaborative A* algorithm, given a point (t). n The next point on the optimal path is (t, x, y). n+1 ,x * ,y * ); Among them, t n+1 Indicates the next time period; x * ,y * This indicates that at time t n+1 The optimal position coordinates at time; argmin represents finding the variable value that minimizes the objective function; J(t) n+1 (x′, y′) represents the time interval t. n+1 The cost of position (x′, y′) at time t; (x′, y′) represents the cost at time t. n+1 When selecting a position coordinate, (x, y) represents the current position coordinate; Step 2.3: Randomly assign agent levels and use the window-based hierarchical collaborative A* algorithm to perform path planning for multiple agents in sequence. Periodically and randomly change the agent level. The agent with the highest level plans the path first and adds it to the relevant cost. The paths planned by subsequent agents will not conflict with those of agents with higher levels.

4. The multi-agent weeding operation planning method for non-flat photovoltaic sites according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: Based on the Bellman equation, derive the expected cumulative reward for each state-action pair to obtain the optimal policy; Step 3.2: Deep neural Q-networks approximate the state-action function by training deep neural networks to estimate the expected cumulative reward; Step 3.3: Use two reward schemes, independent reward and shared reward, to reward and punish agents. Independent reward means that each agent receives its own reward, while shared reward means that all agents receive the same reward, which is composed of the average of the independent rewards.

5. The multi-agent weeding operation planning method for non-flat photovoltaic sites according to claim 4, characterized in that, Step 3.1 specifically involves: the optimality criterion is determined by the defined reward function R. For each state-action pair, a value Q(s,a) is derived. The value function represents the cumulative reward that the agent expects to obtain when taking action a in state S. The value of the Q function of the optimal policy π is given by the Bellman equation. Where Q(s,a) represents the state-action value function, which represents the cumulative reward expected by the agent after performing action a in state s; r(s,a) represents the reward function, which represents the immediate reward obtained by the agent when performing action a in state s; γ represents the discount factor, which ranges from [0,1] and is used to measure the present value of future rewards. The closer the value is to 1, the more the agent values ​​future rewards; the closer the value is to 0, the more the agent values ​​immediate rewards; p(s′|s,a) represents the state transition probability, which represents the probability of transitioning to state s′ after performing action a in state s; max a′∈A Q(s′,a′) represents the maximum Q value that the agent can obtain in the next state s′, which is the maximum expected cumulative reward that the agent can obtain in state s′ under the optimal policy; a′ is the action taken in state s′. For a given Q function, the optimal strategy is as follows:

6. The multi-agent weeding operation planning method for non-flat photovoltaic sites according to claim 4, characterized in that, Step 3.2 specifically involves: The deep neural Q-network approximates the function and estimates the Q-value by training a deep neural network. The training uses the following loss function: in Let e ​​represent the expectation of all states, actions, rewards, and the next state, i.e., the average of all samples. The Bellman error e is: e=r(s,a)+γmax a ′Q(s′,a′;θ - )-Q(s,a,θ) Where, γmax a Q(s′,a′;θ - ) indicates the use of target network parameters θ - The present value of the future reward obtained by multiplying the maximum Q value of the next state s′ by the discount factor γ, θ - The parameters of the target network are used to estimate the target Q-value and are periodically updated from the parameters θ of the current network to stabilize the training process; Q(s,a,θ) represents the Q-value of the current state s and action a calculated using the current network parameters θ; θ represents the parameters of the current Q network, used to estimate the Q-value in the current state. For the last s t+1 Calculate B t =r t For other s t+1 Calculate B t =r t +γmax a′ Q(s t+1 ,a t+1 ;θ - ); where r t Indicates that in state s t Take action a t The reward received later; B t Indicates that in state s t Take action a t The expected cumulative reward; the maximum value Q represents the agent's reward in state s. t+1 The best expected cumulative reward that can be obtained is a t+1 In state s t+1 The actions taken below.

7. The multi-agent weeding operation planning method for non-flat photovoltaic sites according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4.1: Set the position and speed of the agent relative to surrounding obstacles, and set the safety radius; Step 4.2: The agent generates a virtual agent and sets its state, considering the radius of the agent and obstacles as neighbors; Step 4.3: When the agent encounters an obstacle with a radius that is too small, that is, when the distance between the agent and the obstacle is less than the sum of the radii of the neighboring obstacles, send the desired acceleration command to the agent to initiate an obstacle avoidance response.

8. A multi-agent weeding operation planning method for non-flat photovoltaic sites according to claim 7, characterized in that, Step 4.2 specifically involves: Let q i and p i Let q be the position and velocity of agent i, respectively. k and p k Let R be the position and velocity of the obstacle surrounding agent i, and let R be the radius. k Agent i generates a virtual agent, whose state is: On the boundary of the obstacle or agent k, and maintaining contact with that agent, let D... a With D o The radius is considered as the nearest neighbor for the agent and the obstacle, respectively. μ is a scaling factor that ensures the virtual agent lies on the boundary; P is a projection matrix with a direction from q k Pointing to q i The direction of the outward normal of the unit normal vector; ‖·‖ represents the norm, a function used to measure the length or size of a vector; When a virtual agent is generated on the environment boundary, the virtual agent is located at the point on the environment boundary closest to agent i, q k It is any point on the boundary of the environment, and I refers to the identity matrix; For agent i, the set of nearby virtual agents generated by other agents is . The set of nearby virtual agents generated by obstacles is The repulsive force exerted by nearby virtual agents on agent i is This force is sent to the agent as a desired acceleration command; and and The relevant terms will exclude agent i from nearby virtual agents, while... and The related terms will produce a velocity damping effect, φ β It is a repulsive force function. It is a normal vector that has been normalized by σ, b i,k It is a heterogeneous adjacency term between intelligent agent i and its virtual intelligent agent k. and These are positive constants used to adjust the repulsive force and velocity damping between agents. Since obstacles behave differently from agents, different scaling constants are used for the repulsive force generated by obstacles. and 9. The multi-agent weeding operation planning method for non-flat photovoltaic sites according to claim 1, characterized in that, Step 5 specifically includes the following steps: Step 5.1: Set the time step and use the environment model to predict the state within the future time step; Step 5.2: Use model predictive control algorithms to integrate path planning and obstacle avoidance; Step 5.3: Each agent activates only one component at a time.

10. A multi-agent weeding operation planning method for non-flat photovoltaic sites according to claim 9, characterized in that, In step 5.2, the model predictive control algorithm is specifically as follows: Input: Cost J, agent, target and obstacle locations; Output: Control input set {u1, u2, ... u N }; Within the time range T p In Chinese, it is used to predict future states; Agent i cycles through numbers from 1 to N: If the obstacle avoidance planner is active Apply obstacle avoidance rules to u i superior; In other cases, add agent i to the priority sequence. middle; End the loop; All non-sequence The predicted state of the agent in the process is added to J; For in sequence Multi-agent path planning is performed on the intelligent agents in the system. In priority sequence Agent i in the loop begins: Use LQR to control and trace path p i And calculate u i ; End the loop; The result is {u1, u2, ... u}. N }

Images