Unmanned aerial vehicle agricultural bird repelling method and system based on topological sorting reward mechanism

Abstract

The application discloses a topological ordering reward mechanism-based unmanned aerial vehicle agricultural bird driving method and system, which comprises the following steps: constructing a bird recognizer, collecting three-dimensional target data of birds, and performing three-dimensional target tracking; automatically generating a reward automaton according to the recognition result, and constructing a connection mechanism of the reward automaton and a bottom algorithm, and performing path planning by using a reinforcement learning method; the reinforcement learning method based on the topological ordering reward mechanism is used for unmanned aerial vehicle path planning problems, and the unmanned aerial vehicle based on the topological ordering reward mechanism is used for agricultural bird driving work. The application improves the parallelism and resource utilization of the system through the hierarchical reinforcement learning idea based on the dependency relationship between tasks in the topological ordering, and applies the generated top-level abstract task representation to the control of the bottom-level continuous system, and adaptively controls the dynamic environment, and solves the actual problems in the current agricultural bird driving field, such as poor bird damage control, poor task coordination, easy path repetition and coverage, high energy consumption and low efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of agricultural bird repelling technology, and particularly relates to an unmanned aerial vehicle agricultural bird repelling method and system based on a topological sorting reward mechanism. BACKGROUND

[0002] In the field of agricultural bird repelling, net catching has a significant effect, but causes damage to the ecology. The method of repelling birds by using dummies is too simple and single. As a new type of bird repelling technology, sound waves have been widely used in airports, high-voltage power lines and some orchards abroad, but their application in China is limited. Existing ultrasonic bird repellers have fixed frequencies, which leads to the easy adaptation of birds. Laser bird repellers irradiate the protection area by high-brightness laser beams, but long-term use will make birds adapt, and the effect is not good. Electronic bird repellers use electric current to excite air explosion, but long-term use will also lead to the enhancement of bird adaptability and the weakening of the effect. Bird repellents usually use chemical agents diluted with water for spraying, which can slowly and persistently release odors affecting the central nervous system of birds, but this method will cause residue problems. Ultrasonic bird repellers use ultrasonic pulses to interfere with, stimulate and damage the nervous and physiological systems of birds. Its working principle is to make the tympanic membrane of birds vibrate at an amplitude of 40nm-80nm through 20kHz-80kHz ultrasonic frequency. In order to improve the bird prevention effect, a new type of bird repeller composed of sound, light and ultrasonic bird repellers can be used. Birds will gradually adapt to the bird repelling device, which will lead to a decrease in the bird repelling effect. In addition, the habits of birds are full of unknowns, and it is impossible to accurately understand their feeding time. Bird repelling devices need human participation for driving operation, which cannot achieve the goal of efficient and economic bird repelling. Therefore, it is urgent to introduce intelligent equipment to replace labor in agricultural production activities. A real-time bird repelling system based on reinforcement learning is necessary, which uses target detection technology to monitor the bird invasion in the field and combines with unmanned aerial vehicle technology to dynamically repel birds. Through the combination of the two technologies, the problem of agricultural bird damage can be efficiently and labor-savingly solved. SUMMARY

[0003] In view of the above problems, the present application is proposed.

[0004] Therefore, the technical problem solved by the present application is to solve the actual problems of poor scalability, easy to fall into local optimum, sparse reward and the like existing in the current multi-agent multi-task continuous control.

[0005] To solve the above technical problems, the present application provides the following technical scheme: an unmanned aerial vehicle agricultural bird repelling method based on a topological sorting reward mechanism, comprising:

[0006] The bird recognizer is constructed, three-dimensional target data of birds is collected, and three-dimensional target tracking is performed; a reward automaton is automatically generated according to a recognition result, a connection mechanism of the reward automaton and a bottom algorithm is constructed, and a reinforcement learning method is used for path planning; the reinforcement learning method based on the reward mechanism of the topological sorting is used for the unmanned aerial vehicle path planning problem, and the unmanned aerial vehicle agricultural bird driving method based on the reward mechanism of the topological sorting is established; the unmanned aerial vehicle based on the reward mechanism of the topological sorting is used for agricultural bird driving work.

[0007] As a preferred scheme of the unmanned aerial vehicle agricultural bird driving method based on the reward mechanism of the topological sorting, the bird recognizer comprises: a bird classifier is constructed using a deep learning method; image data sets of different types of birds are collected and labeled; a convolutional neural network is used to train the image data to construct an accurate bird classifier; the classifier monitors birds in a farmland in real time through a camera, judges sensitive data domains of a target picture to be detected, judges the next frame picture if there is no sensitive data domain, uses a YOLO algorithm to detect a target region displayed by the picture if the picture has a sensitive data domain, collects boundary box information of all small regions in the picture, classifies and judges the detected bird target through a convolutional neural network, then judges the whole picture, and finally outputs the boundary box and object class information of the target contained in the picture.

[0008] As a preferred scheme of the unmanned aerial vehicle agricultural bird driving method based on the reward mechanism of the topological sorting, the target tracking comprises: a camera is used for target monitoring, and internal and external parameters of the camera are obtained; the internal parameters comprise focal length and principal point coordinates of the camera, and the external parameters comprise position and posture of the camera; target detection is performed on the picture, and all b-type bird targets appearing satisfy b∈B, B is a set of all birds; an image block around each target is extracted as a feature representation of the target according to the position of the boundary box of each target; a convolutional neural network method is used to obtain the target feature; in the next time step, the YOLO algorithm is used for target detection, and a similarity measurement method IoU is used to match the target of the current frame with the target of the previous frame.

[0009] According to the target matching result, the target tracking is realized by updating the position and state of the target; a continuous tracking algorithm Kalman filter is used to predict the activity path of the bird and track the movement of the bird target; the tracking algorithm predicts and estimates the position of the next frame of the target according to the target detection result of the current frame and the tracking result of the previous frame, and realizes continuous tracking of the target.

[0010] As a preferred scheme of the unmanned aerial vehicle agricultural bird repelling method based on the topological sorting reward mechanism, the target three-dimensional information acquisition includes using a double camera parallax effect to calculate three-dimensional coordinates of the birds, and using a triangulation method; the same bird target is detected in images of the left camera and the right camera at the same time, and corresponding pixel coordinates are obtained; the parallax M between the corresponding pixels is calculated through an image processing algorithm; the parallax is obtained by calculating the difference between the horizontal pixel coordinates:

[0011]

[0012] the bird target pixel coordinates in the left camera are the bird target pixel coordinates in the right camera; according to the camera parameters and the geometric relationship, the relationship between the parallax and the actual depth is obtained; the relationship can be obtained through the baseline distance N of the camera, the focal length L and the parallax M:

[0013] J=(N*L) / M

[0014] the depth value J of the target is calculated using the obtained baseline length and the viewpoint distance, and the calculated parallax value; in the coordinate system with the camera as the coordinate origin, the three-dimensional coordinates of the target are (X n ,Y n ,Z n ), the corresponding pixel coordinates of the target in the two camera images are and for the detected target, it is represented as:

[0015]

[0016] the obtained three-dimensional coordinates are converted to a specific coordinate system, and the obtained three-dimensional coordinates are reserved in the sensor as state space variables; a Kalman filter is used to track the target and perform noise reduction processing to obtain an optimal estimate of the target position, including an estimate of the current target position and an estimate of the future position;

[0017] then, an update step is performed, and the measurement model equation of the update is:

[0018] D(k)=q(k)-H*W(k)

[0019] wherein D(k) is a measurement residual, q(k) is a measurement value at the current time, and H is a measurement matrix;

[0020] the updated Kalman gain equation is:

[0021] K(k)=P(k)*H^T*(H*P(k)*H^T+R)^-1

[0022] wherein K(k) is a Kalman gain, and R is a covariance matrix of measurement noise.

[0023] The updated state estimation value update equation is:

[0024] W(k) = W(k) + K(k) * D(k).

[0025] The updated covariance matrix update equation is:

[0026] P(k) = (I - K(k) * H) * P(k)

[0027] wherein I is an identity matrix; the above-mentioned prediction and update steps are repeated, in the update step, the state vector W(k) is corrected according to the measurement model and the Kalman gain model, and the contained velocity component is retained in the sensor as a state space variable.

[0028] As a preferred scheme of the unmanned aerial vehicle agricultural bird repelling method based on the topology sorting reward mechanism, the reinforcement learning path planning method comprises the following steps: automatically generating a reward automaton according to the recognition result, constructing a connection mechanism of the reward automaton and a bottom algorithm; explicitly defining the dependency relationship between each task, defining a task hierarchy, constructing a state space and an action space, and defining the dependency relationship of different birds and a current state according to different birds and the current state; according to the sorting result of the task, assigning a corresponding reward or weight to each task, and defining the state and action space according to the specific requirements: the state represents the current state of the system or the observation result of the environment, and the action represents the operation or strategy that can be executed by the reward automaton; the current state and the executed action are mapped to a real reward value by using the reward mechanism, and the result generated by the automatic reward machine is directly used here.

[0029] A top-level reward function is designed, and the whole project is divided into multiple stages and tasks into multiple subtasks; there is a dependency relationship between each subtask, and the dependency relationship can be represented as a directed acyclic graph DAG, wherein each node represents a subtask, and the edge represents the dependency relationship between tasks; the DAG of the task is topologically sorted, and the classic topological sorting algorithm depth-first search DFS is used to realize the topological sorting, and the distribution parameters of the reward are adjusted according to the properties of the project and the performance of the team; the distribution is based on balancing the priority of the task, the size of the reward and the relationship between different tasks, and the constraints are defined according to the execution time and specific conditions.

[0030] As a preferred scheme of the unmanned aerial vehicle agricultural bird repelling method based on the topological sorting reward mechanism, the reward automaton is generated according to the identified bird information; the reward automaton provides a reward or punishment signal for the unmanned aerial vehicle according to a specific behavior rule to guide the behavior of the unmanned aerial vehicle; the reward signal is combined with the path planning algorithm of the unmanned aerial vehicle, an interface is established to connect the state and action space of the reward automaton with the bottom algorithm; the three-dimensional space position result is output, and the target observation result, the unmanned aerial vehicle movement and the influence of bird repelling are unified into the three-dimensional space scale; the environment is observed and perceived, and the reward is generated:

[0031] MDP M=(S,s0,A,P,γ)

[0032] Wherein, S represents a finite state set, s0∈S represents an initial state, A represents a finite action set, P∈S×A→Dist(S) represents a transition probability distribution, γ represents a discount factor; R∈S×A×S→R represents a reward assigned to the transition between states; for each step, a policy derived from the DQN procedure is followed, which is based on the current state S and derived using the function, Q represents;

[0033] Q(<s,e>,a;θ,α,β)

[0034] Wherein, Q∈S×A→R, <s,e> represents a state, a represents a finite action set, θ represents a network parameter of a convolution layer, and α and β represent network parameters of a fully connected layer; further actions are guided by the policy derived from Q(<s,e>,a;θ,α,β) and the selection of actions from the action space a, the next state (s',e') is observed, it is judged whether the action meets the policy goal, and the calculation of the reward is performed, and the dynamic state transition and learning strategy are progressively established; the reward machine is used to expand the MDP to reward the implementation of each action, and the reward function δ r and the potential function The corresponding reward is calculated at a given state and action (e,a); set M and N share the same label function L∈S×A×S→F, F is a local action set in a given reward machine and learning strategy; if (e,L(s,a,s'))∈dom(δ e ), then the reward machine is transferred from the state e to L(s,a,s'), otherwise it remains in the state e; wherein the expanded transition probability is the same as the transition probability in the original MDP. Given the reward machine N:

[0035]

[0036] Wherein, E represents a finite state set, E0∈S represents an initial state, δ eE x F -> E represents a transition function between states, and δ r E x F -> R is a reward function of state and transition function, is a potential energy function of state, and M = (S, s0, A, P, γ) is an MDP, wherein M and N share a label function L, and T E S is a set of accepted states;The extended MDP is defined as The extended MDP is mapped from the original MDP, and there is a mapping relationship between the states, so Because in the state conversion process, the transition probability is defined based on the state set S and the action set A, and the transition probability is represented as:

[0037]

[0038] If the next state of the reward machine is accepted, the reward function δ r is updated, if the next state is not accepted, the reward function δ r and the potential energy function are updated simultaneously, and the update formula is represented as:

[0039]

[0040] In order to ensure that the bottom strategy and the top strategy have a proper communication and coordination mechanism, the task can be successfully executed, and the bottom strategy and the top strategy need to be iteratively trained and optimized, and the path planning is continuously optimized.

[0041] As a preferred scheme of the unmanned aerial vehicle agricultural bird repelling method based on the topological sorting reward mechanism provided by the application, wherein: the reward mechanism reinforcement learning planning unmanned aerial vehicle path based on topological sorting includes,

[0042] Modeling the problem, defining the hierarchy of the task, decomposing the path planning task into different subtasks;Clarify the classification and attributes of the starting point and the target point in the target space;Convert the problem form into a graph, the node is a discrete coordinate point, in motion, and the edge represents the path;Define and build a complete and clear state space and action space;

[0043] Establish the topological structure, design the top-level reward function, and generate the reward signal;Optimize the path by the topological sorting method, and each subtask is represented as path planning from the current position to a certain intermediate point;The reward function should encourage the unmanned aerial vehicle to select the path consistent with the topological structure;The potential energy function is introduced to set the reward mechanism, when the unmanned aerial vehicle selects the path according to the topological structure, positive reward is given;When the unmanned aerial vehicle violates the topological structure, negative reward is given, and unreasonable path selection is punished;

[0044] In each subtask, the underlying strategy and the corresponding environment are defined to guide the reinforcement learning algorithm to select the optimal action in each subtask; the DQN algorithm is used to learn the path planning strategy, so that the UAV can select the appropriate action to drive away the birds according to the current state; each subtask is executed using the underlying strategy, and a top-level strategy is learned, which determines when to switch to a different subtask; a proper interface or interaction mode is established to ensure proper communication and coordination mechanism between the underlying strategy and the top-level strategy, to achieve smooth execution of the task, and iteratively train and optimize the underlying strategy and the top-level strategy, debug and evaluate the path planning system.

[0045] An agricultural bird repelling system for UAV based on the topological sorting reward mechanism using any of the methods of the present application, wherein:

[0046] The bird tracking and monitoring module uses the monitoring unit to identify and classify the bird target and transmits it to the comprehensive calculation module to obtain the three-dimensional information of the bird; the path planning module uses the reinforcement learning method to establish a UAV path planning model and optimizes it in combination with the topological sorting reward mechanism; the information receiving and sending module receives the information input by each module and sends it according to the demand, and coordinates the operation of each module; the comprehensive calculation module comprehensively calculates the information collected by the monitoring module and the model established by the path planning module, and feeds back through the information receiving and sending module;

[0047] A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the method of any one of the present application.

[0048] A computer readable storage medium having a computer program stored thereon, characterized in that the computer program is executed by a processor to implement the steps of the method of any one of the present application.

[0049] The beneficial effects of the present application: the agricultural bird repelling method for UAV based on the topological sorting reward mechanism provided by the present application, BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor. Among them:

[0051] Figure 1 The overall flowchart of the agricultural bird repelling method for UAV based on the topological sorting reward mechanism provided by the first embodiment of the present application;

[0052] Figure 2 The overall block diagram of the unmanned aerial vehicle agricultural bird repelling method based on the topological sorting reward mechanism provided for the first embodiment of the present application is shown in the figure;

[0053] Figure 3 The state transition diagram of the control strategy in the unmanned aerial vehicle agricultural bird repelling method based on the topological sorting reward mechanism provided for the first embodiment of the present application is shown in the figure;

[0054] Figure 4 The three-dimensional target recognition and tracking principle diagram of the unmanned aerial vehicle agricultural bird repelling method based on the topological sorting reward mechanism provided for the first embodiment of the present application is shown in the figure;

[0055] Figure 5 The diagram for calculating the three-dimensional coordinates of the target using the triangle principle of the unmanned aerial vehicle agricultural bird repelling method based on the topological sorting reward mechanism provided for the first embodiment of the present application is shown in the figure;

[0056] Figure 6 The performance comparison diagram of different methods of the unmanned aerial vehicle agricultural bird repelling method based on the topological sorting reward mechanism provided for the second embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0058] Embodiment 1

[0059] Reference Figures 1-5 For an embodiment of the present application, an unmanned aerial vehicle agricultural bird repelling method based on a topological sorting reward mechanism is provided, comprising:

[0060] S1: Construct a bird recognizer, collect three-dimensional target data of birds, and perform three-dimensional target tracking.

[0061] Further, the bird recognizer: a bird classifier is constructed using a deep learning method. First, a large number of image data sets of different types of birds are collected and labeled. Then, a convolutional neural network (CNN) is used to train the image data to construct an accurate bird classifier. The classifier can monitor birds in farmland in real time through a camera.

[0062] Further, bird tracking and three-dimensional target acquisition:

[0063] Step 1, as Figure 4Target monitoring: Use two cameras, left camera (Camera L ) and right camera (Camera R ), their relative position and angle are known. Calibration: First, the binocular camera needs to be calibrated to obtain the internal and external parameters of the camera. The internal parameters include the focal length of the camera, the principal point coordinates (p), etc. The external parameters include the position and attitude of the camera. Take the binocular camera to monitor and track the environment. Through the left and right cameras with infrared modules, the environment is monitored and the collected pictures are transmitted.

[0064] Step 2, target detection: First, the transmitted target picture is judged by the left camera whether there is sensitive data field to be detected. If there is no sensitive data field, the next frame of picture is judged; if the picture contains sensitive data field, the target detection work is carried out on the target area displayed by the frame picture. The frame picture is detected by YOLO algorithm for image detection. The algorithm divides the target picture into multiple grid regions of the same size, collects the boundary box information of all small regions in the picture, and classifies and judges the detected bird target through convolutional neural network, then judges the whole picture, and finally outputs the boundary box (bx, by, bw, bh) and object class (class_label) information contained in the frame picture.

[0065] Step 3, feature extraction: target detection of the frame picture may appear multiple different types of bird targets, and the b type bird target appears satisfies b∈B, B is the set of all birds. The bounding box position (bx, by, bw, bh) of each target is extracted around the target to obtain a certain size of image block as the feature representation of the target. Convolutional neural network (CNN) method can be used to obtain the target feature. Target matching: in the next time step, continue to use YOLO algorithm for target detection, and use similarity measurement method IoU (intersection over union) to match the target of the current frame with the target of the previous frame.

[0066] Step 4, target tracking: according to the target matching result, the target tracking is realized by updating the position and state of the target. The continuous tracking algorithm Kalman filter is used to predict the activity path of the bird and track the movement of the bird target. The tracking algorithm will predict and estimate the next frame of the target position according to the target detection result of the current frame and the tracking result of the previous frame, so as to realize the continuous tracking of the target.

[0067] Step 5, target three-dimensional information acquisition: using the parallax effect of double cameras to calculate the three-dimensional coordinates of birds, using the method of triangulation. The three-dimensional coordinate acquisition steps are as follows: the same bird target is detected in the images of the left camera and the right camera at the same time, and the corresponding pixel coordinates are obtained. Assuming that the pixel coordinates of the bird target in the left camera are The pixel coordinates of the bird target in the right camera are Parallax calculation: two images obtained by binocular cameras can calculate the parallax (M) between corresponding pixels through image processing algorithm. Parallax represents the pixel difference of corresponding points of the same target in two images, which can be used to estimate the target distance. Parallax can be obtained by calculating the difference between horizontal pixel coordinates: Depth calculation: according to the camera parameters and geometric relationship, the relationship between parallax and actual depth can be obtained. This relationship can be obtained by the baseline distance (N) of the camera, the focal length (L) and the parallax (M): J = (N*L) / M. Using the obtained baseline length and viewpoint distance, and the calculated parallax value, the depth value (J) of the target is calculated. Triangulation: calculate the three-dimensional coordinates of the target by using the principle of triangulation. In the coordinate system with the left camera as the coordinate origin, assuming that the three-dimensional coordinates of the target are (X n ,Y n ,Z n ), the corresponding pixel coordinates of the target in the two camera images are and , then the three-dimensional coordinates of the target can be calculated by using the principle of triangle as shown in the figure. Figure 5

[0068] Further, according to the similarity of triangle:

[0069]

[0070] Therefore, for the detected target, there is:

[0071] Z n =(N*L) / M;

[0072]

[0073]

[0074] Step 6, coordinate conversion: after calculating the three-dimensional coordinates of the target (X n ,Y n ,Z n ​) After that, it may be necessary to convert it to a specific coordinate system. This process involves the position and pose of the camera, as well as the definition of the coordinate system. Depending on the needs, the coordinates of the target can be converted to the world coordinate system or other custom coordinate system and the obtained three-dimensional coordinates are retained in the sensor as state space variables.

[0075] Further, the ABCDEF corresponds to six states, which can be represented as six states in space, where abcdef is the different task type of expelling these birds, referring to the task list, which can be defined as:

[0076] a: expel sparrows b: expel eagles c: expel ravens d: expel black geese e: expel pigeons f: expel swallows

[0077] Among them, the way of expelling can refer to the action space, including: movement (motion control in x, y, z direction), bird expelling action, hovering observation, reset. Among them, the bird expelling action has four ways: sound bird expelling, optical bird expelling, biological agent bird expelling, and fan bird expelling.

[0078] Further, for example Figure 3 The reward machine shown in (a) and (b) requires the UAV to perform tasks based on the current state and obtain rewards. This reward machine has four states and four task execution state transitions. As shown in (a), the initial state of the UAV is A, and the UAV starts to execute the entire strategy of agricultural bird expelling in this state. The UAV can first expel sparrows, and after completing the task, it can choose to expel eagles, reaching state B. Similarly, it continues to execute the tasks of expelling ravens and expelling black geese in sequence after completing the tasks. In this way, a strategy is formed. In (b), after completing the task of expelling eagles, the UAV can choose to execute the task of expelling black geese or expel ravens, forming a new path strategy. At this time, when in state B, there are two choices for task execution, and there are two ways to change states. In the process of learning these strategies, the UAV obtains corresponding rewards through training to evaluate the pros and cons of the strategies.

[0079] Further, Figure 3(c) and (d) are two strategies in the same state space, which add new states and tasks to (a) and (b). In (c), the UAV sequentially executes the tasks a (expel sparrow) b (expel eagle) c (expel raven) d (expel black goose) e (expel pigeon) in the starting state, and then transitions to the E state. At this time, the UAV can execute the task f (expel swallow) to reach the F state, or execute the task c (expel raven) again to return to the C state. (d) provides a more flexible strategy. The UAV can directly execute the task of expelling the raven at the A state, and directly transition to the C state. The UAV can execute the task of expelling the black goose at the C state, or execute the task of expelling the black goose at the B state. Finally, after executing the task e (expel pigeon), the UAV executes the task f (expel swallow) at the E state to return to the initial position. The UAV can select different possible actions to execute decisions. Each state transition represents the formation of a strategy. In order to make this decision, the UAV will consider the current environmental state and learn which strategy it should execute in this state to achieve the highest expected value. After multiple learning and training, the best path strategy is obtained.

[0080] It should be noted that the target speed is obtained: when tracking the target using the Kalman filter, the speed of the target can be estimated by observing the position information of the target. The measured values of the position, speed and acceleration of the target often have noise at any time. Kalman filtering uses the dynamic information of the target to eliminate the influence of noise and obtain a good estimate of the position of the target. This estimate can be an estimate of the current target position or an estimate of the future position.

[0081] S2: Tracking and obtaining the target position state.

[0082] Further, the state of the target at a certain time is represented as a vector. In three-dimensional target tracking, the state vector includes six components of position and velocity Therefore, at time k, the state vector of the target is Prediction step: in the update step of the Kalman filter, the observation data is used to update the estimate of the state vector, which is only related to k-1 at time k.

[0083] Step 1, therefore W(k) = F*W(k-1), where W(k) is the state estimate value at the current time, and F is the state transition matrix. The state transition matrix F can be represented as:

[0084]

[0085] Where △t is the time step.

[0086] Step 2, the covariance matrix update equation in the prediction step is: P(k) = F*P(k-1)*F^T+q, where P(k) is the covariance matrix at the current time, and q is the covariance matrix of the process noise.

[0087]

[0088] where Pxx, Pyy, Pzz represent the variance of the state vector in three dimensions, and Pxy, Pxz, Pyz represent the covariance between variables.

[0089]

[0090] where q1, q2, q3 represent the variance of the process noise in three dimensions, respectively, and the noise is estimated through the physical characteristics of the system, environmental factors.

[0091] Step 3, update step:

[0092] The measurement model equation in the update step is: D(k) = q(k) - H*W(k), where D(k) is the measurement residual, q(k) is the measurement value at the current time, and H is the measurement matrix.

[0093]

[0094] The Kalman gain equation in the update step is: K(k) = P(k)*H^T*(H*P(k)*H^T+R)^-1, where K(k) is the Kalman gain, and R is the covariance matrix of the measurement noise.

[0095]

[0096] where σ1 2 , σ2 2 , and σ3 2 represent the variance of the measurement noise in three dimensions, respectively.

[0097] Further, the state estimate update equation in the update step is: W(k) = W(k) + K(k)*D(k).

[0098] Further, the covariance matrix update equation in the update step is: P(k) = (I-K(k)*H)*P(k), where I is the identity matrix.

[0099] It should be noted that the above prediction and update steps are repeated, and in the update step, the state vector W(k) is corrected according to the measurement model and the Kalman gain model, and the contained velocity component is retained in the sensor as a state space variable.

[0100] S3: Path planning using the method of reinforcement learning.

[0101] Further, the reward automaton is automatically generated according to the recognition result, and a connection mechanism of the reward automaton and the underlying algorithm is constructed to improve the path planning efficiency in continuous state space.

[0102] Further, the dependency between each task needs to be clarified, the task hierarchy is defined, and the state space and action space are constructed, corresponding to different birds and the current state, and the dependency relationship is defined. According to the sorting result of the task, the corresponding reward or weight is allocated to each task. The reward or weight can be defined according to specific needs, such as the characteristics, quantity, group size, driving effect, time and distance of the bird itself, etc. Define state and action space: determine the state and action space of the reward automaton. The state can represent the current state of the system or the observation result of the environment, which can be defined based on the results of the bird classifier and tracking. The action represents the operation or strategy that the reward automaton can perform. Here, the motion of the unmanned aerial vehicle and the use of the bird driving device are directly executed. We assume that each state requires six sensors pointing in different directions, and observes seven variables in each direction: the position (x, y, z) of the target in space, the category to which the target belongs, the motion speed v (the direction is indicated by x, y, z), and such targets have seven categories. In addition, it is necessary to measure all these values of the unmanned aerial vehicle at this time, which are real numbers, and the state is a 52-dimensional continuous state space. At the same time, the state space is a six-degree-of-freedom space, moving in six different directions. Action space (operation executed by the agent): movement (motion control in x, y, z direction), bird driving action, hovering observation, reset. Among them, the bird driving action has four ways of sound driving, optical driving, biological agent driving and fan driving, and the current state and the executed action are mapped to a real reward value using the reward mechanism. Here, the result generated by the automatic reward machine is directly used.

[0103] Further, a top-level reward function is designed: the entire project or task is divided into multiple stages or sub-tasks. There is a dependency relationship between each sub-task, that is, some tasks must be completed before other tasks. These dependencies can be represented as a directed acyclic graph (DAG), where each node represents a sub-task and the edge represents the dependency relationship between tasks. Topological sorting is performed on the DAG of tasks to determine the execution order of the tasks. Topological sorting guarantees that no task has a dependency relationship pointing to it that appears in front of it after sorting. This sorting can be implemented using the classic topological sorting algorithm, depth-first search (DFS). Based on the topological sorting, the reward allocation is performed task by task from the first task. In practical applications, the allocation parameters of the reward are adjusted according to the nature of the project and the performance of the team. The basis for allocation is to balance the priority of the task, the size of the reward, and the relationship between different tasks. There are constraints to be defined in this process, mainly defined according to the execution time and specific conditions.

[0104] Further, a bottom-level strategy is defined: based on reinforcement learning algorithms, the model is trained, where the reward automaton based on topological sorting is used to plan the path of the unmanned aerial vehicle, track different birds and take different ways to drive them away. A reinforcement learning algorithm suitable for the problem is selected as the bottom-level algorithm. This patent uses the DQN-ts method. This is to be compatible with the state and action space of the reward automaton, and to have the ability to adapt to the complexity and performance requirements of the problem.

[0105] Further, the connection mechanism between layers: according to the identified bird information, the corresponding reward automaton is generated. The reward automaton can provide reward or punishment signals for the UAV according to the specific behavior rules to guide its behavior. The reward signal is combined with the path planning algorithm of the UAV to improve the efficiency of path planning in continuous state space. The state and action space of the reward automaton is connected with the bottom algorithm. This can be achieved by establishing a proper interface or interaction method. The result here is the output of the three-dimensional space position result, which unifies the target observation result, UAV movement and bird repelling effect into the three-dimensional space scale, which is similar to the method in two-dimensional space. First, we accept the MDP M=(S,s0,A,P,γ) obtained by observing the environment and the generated reward. S represents a finite state set, s0∈S represents the initial state, A represents a finite action set, P∈S×A→Dist(S) represents the transition probability distribution, and γ represents the discount factor. And use R∈S×A×S→R to represent the reward assigned to the transition between states. The information obtained by observation is complete and continuous state space and action space. For each step, follow the policy derived from the DQN procedure, which is based on the current state S and derived using the function Q(<s,e>,a;θ,α,β), where Q∈S×A→R,<s,e> represents a state, a represents a finite action set, θ represents the network parameters of the convolutional layer, and α and β represent the network parameters of the fully connected layer. This guides the agent to take further action, observes the state transition of the next state after the action is implemented, and uses the reward mechanism in the process to measure the goodness of the action or its contribution to the system goal. That is, the strategy derived from Q(<s,e>,a;θ,α,β) and the action selected from the action space a, the next state (s',e') is observed, and the action is rewarded according to the strategy goal. At the same time, the conversion and reward are saved for sampling, and the dynamic state transition and learning strategy are progressively established. We use the reward machine to extend the MDP to reward each action implementation, and introduce the reward function δ r and the potential function Then the corresponding reward can be calculated at a given state and action (e,a). Suppose M and N share the same labeling function L∈S×A×S→F, F is the given reward machine and the local action set in the learning strategy. If (e,L(s,a,s'))∈dom(δ e ), then the reward machine is transferred from state e to L(s,a,s'), otherwise it remains in state e. The extended transition probability is the same as the transition probability in the original MDP. Given the reward machine

[0106] Further, (E represents a finite state set, E0∈S represents the initial state, δ e ∈E×F→E

[0107] a transition function between states, δ r ∈E×F→

[0108] R is a reward function of state and transition function, is a potential function of state) and MDP M = (S, s0, A, P, γ), where M and N share the label function L, T ∈ S is the set of accepting states. The extended MDP is defined as where the extended MDP is mapped from the original MDP, and there is a mapping relationship between the states, so Because the transition probability is defined based on the state set S and the action set A in the state transition process, the transition probability is:

[0109] the reward obtained in the transition process,

[0110] is defined in the state <s, e> ∈ S × E.

[0111] where the current action is defined in the action space a ∈ A, if the next state of the reward machine is accepting, only the reward function δ r is updated, if the next state is not accepting, the reward function δ r and the potential function need to be updated at the same time. For example, as shown in equation (1):

[0112]

[0113] It should be noted that in order to ensure that the bottom policy and the top policy have appropriate communication and coordination mechanisms, and to achieve smooth execution of the task, the bottom policy and the top policy need to be iteratively trained and optimized to continuously improve the performance of the entire path planning system. For example, after the bottom policy completes the state transition, the top reward function needs to give the corresponding reward. That is, after the bird driving action is implemented, the reward is given according to the bird driving effect, and the next task is switched, forming the entire path planning system. Iteratively train and optimize the bottom policy and the top policy to continuously improve the performance of the entire path planning system.

[0114] S4: Reinforcement learning method based on reward mechanism of topological sorting is used for unmanned aerial vehicle path planning problem.

[0115] Further, we use topological sorting to establish the topological structure of the unmanned aerial vehicle flight path, and use the reward mechanism to guide the unmanned aerial vehicle to learn how to choose the path.

[0116] Further, first, the actual problem needs to be modeled, defining the hierarchy of tasks, decomposing the path planning task into different subtasks. These subtasks can have different objectives and levels of complexity. The classification and attributes of the starting point and target point in the target space are explicitly defined. This problem can be formalized as a graph, where each node is a location (nodes are discrete coordinate points and are in motion), and edges represent paths. Use cameras and sensors to obtain the position, velocity, and other information of the UAV, as well as the current bird's position and species information. Define and build a complete and clear state space and action space.

[0117] Further, after defining the space and hierarchy, it is necessary to establish the topology, design the top-level reward function, and generate the reward signal. The topology sorting method is used to optimize the path, and each subtask can be represented as a path planning from the current location to a certain intermediate point. This topological sorting can help the UAV gradually advance and avoid facing complex paths at the beginning. The reward function should encourage the UAV to choose paths that conform to the topology. In this method, a potential function is also introduced to set the reward mechanism. When the UAV chooses a path according to the topology, it is given a positive reward. When the UAV violates the topology, it is given a negative reward to punish unreasonable path selection.

[0118] Further, in each subtask, define the bottom-level strategy and the corresponding environment to guide the reinforcement learning algorithm to select the best action in each subtask. Use the DQN algorithm to learn the path planning strategy, so that the UAV can choose appropriate actions (such as moving, making sounds, etc.) based on the current state to drive away the birds.

[0119] Further, use the bottom-level strategy to learn to perform each subtask. The goal of the bottom-level strategy is to maximize the reward signal related to the top-level reward function of the corresponding subtask. Learn a top-level strategy that decides when to switch to different subtasks to achieve overall path planning.

[0120] Further, establish a proper interface or interaction method to ensure that there is a proper communication and coordination mechanism between the bottom-level strategy and the top-level strategy to ensure smooth execution of the task.

[0121] Further, iteratively train and optimize the bottom-level strategy and the top-level strategy to continuously improve the performance of the entire path planning system. Debug and evaluate the path planning system to ensure that it can work effectively in different environments and tasks. Using the hierarchical idea, the algorithm can efficiently handle high-dimensional and complex environments, improving the efficiency of path planning.

[0122] It should be noted that in practical applications, path planning usually involves multiple levels of tasks, from high-level tasks (such as target navigation) to low-level tasks (such as resetting, single bird repelling action). Hierarchical reinforcement learning can effectively handle this multi-level task structure and adapt to multi-objective tasks to enable the agent to gradually complete complex tasks. Because the task is decomposed into multiple subtasks, each subtask has a clear goal and reward function, so the behavior of the system can be more easily understood. To achieve automation of path planning.

[0123] An unmanned aerial vehicle agricultural bird repelling system based on a topological sorting reward mechanism, characterized by:

[0124] The bird tracking and monitoring module uses the monitoring unit to identify and classify the bird target and transmits it to the comprehensive computing module to obtain three-dimensional information of the bird.

[0125] The path planning module uses a reinforcement learning method to establish an unmanned aerial vehicle path planning model and optimizes it in combination with a topological sorting reward mechanism.

[0126] The information transceiving module receives information input by each module and sends it according to demand, and coordinates the operation of each module.

[0127] The comprehensive computing module performs comprehensive calculation on the information collected by the monitoring module and the model established by the path planning module, and feeds back through the information transceiving module.

[0128] Example 2

[0129] Reference Figure 6 For an embodiment of the present application, an unmanned aerial vehicle agricultural bird repelling method based on a topological sorting reward mechanism is provided. In order to verify the beneficial effects of the present application, economic benefit calculation and simulation comparison experiments are used for scientific demonstration.

[0130] Case Background: (Include task description) This instance environment is a farmland, surrounded by forest or hills. The farmland is planted with crops such as wheat or rice, which attract a large number of birds during the growing season. Suppose the farmer of this wheat farm is facing the problem of birds such as sparrows and pigeons feeding on crops during the ripening period of wheat. A drone equipped with a camera and sensors can be used. The drone can monitor bird activity by flying over the farmland using its camera and sensors. When the drone detects a bird flock approaching the farmland or near the crops, it can emit sound or flash light signals such as flash bombs to drive away the bird flock. In addition, the drone can also use noise or harmless laser beams to force birds to leave the crop area. Since birds are not a species, nor do they act alone, it is assumed that there are multiple bird species active in the farmland area, and the drone as an intelligent agent acts in this space, using reinforcement learning to derive the optimal bird driving path and strategy. The farmer first uses the drone to detect and evaluate the farm, monitor the types and quantities of birds, understand which birds cause the main feeding problem, and their time and frequency of occurrence. According to the results of target detection and evaluation, generate state space and action space, use algorithms to generate reward machines and plan the path of the drone, and automatically achieve agricultural bird driving

[0131] Birds that cause harm to agriculture include:

[0132] Mainly waterfowl (geese, cranes, cranes, etc.), such as gray cranes, bar-headed geese, black cranes, pigeons, parrots, corvids, thrushes, starlings, mimic starlings, finches, and a small part of the bird family

[0133] Specific species: woodpeckers, ravens, pigeons, black geese, sparrows, swallows, hawks.

[0134] We designed a set of 6 tasks for bird driving in the farm, about the event of expelling a certain bird. For example, a simple task is to expel a sparrow and then expel a hawk. Other more complex tasks include expelling a series of birds in strict order, such as pigeons, swallows, and ravens, so if the drone expels a bird that cannot expel the next bird of a different species, it will fail. The complete task list is shown in Table 1. In Table 1, "(species 1 then species 2)" means expelling a bird of species 1 and then expelling a bird of species 2. Tasks like "(species 1 then species 2, then species 3)" are similar, but require expelling three species of birds. By "(subtask 1) and (subtask 2)", we mean that the agent must complete subtask 1 and subtask 2 to complete the task described, but in any order. "(Species 1 is strictly species 2)" refers to a task like "(Species 1 then Species 2)", but the agent is not allowed to expel birds of other species during execution.

[0135] Table 1 task list

[0136]

[0137] After obtaining the top-level control strategy of the UAV, it is applied to the continuous control of the UAV. In the whole bird driving task, the completion of a task may depend on the completion of other tasks. By completing the tasks in the correct order, the smooth progress of the whole process can be ensured, so this patent adopts the reward mechanism based on topological sorting as shown in Table 2. This mechanism promotes agents to complete tasks in the correct order to ensure the smooth progress of the project. This helps to avoid delays and problems caused by improper task completion order. The reward mechanism of topological sorting decomposes complex projects into a series of ordered tasks. This helps to better understand the structure of the project during the management and execution phases, and it is easier to monitor the completion of tasks. Moreover, by completing tasks in topological order, the project can gradually progress. At the same time, the concept of cumulative reward is usually involved in this process. This means that after completing a series of dependent tasks, a larger cumulative reward can be obtained. The reward mechanism of topological sorting reduces the uncertainty of task completion. Because the tasks have clear dependencies, there is a higher reliability and clarity before starting to execute the tasks, thereby reducing the likelihood of errors.

[0138] Table 2 algorithm for agents to learn optimal strategies Algorithm pseudo code.

[0139] Table 2-1 Algorithm pseudo code for agents to learn optimal strategies

[0140]

[0141] Table 2-2 Reward shaping algorithm pseudo code based on topological sorting

[0142]

[0143]

[0144] Note: v represents a vertex in graph G, eP represents potential energy, represents the reward of the vertex, l represents the index value, u is the adjacent vertex of v, and!u,!v represents that the vertex has not been visited.

[0145] The present application proposes a reward automaton design potential function based on topological sorting for the path planning problem of agricultural bird driving UAV, and applies it to agent learning. In this experiment, the above two algorithms are integrated into the agricultural bird driving implementation case. The UAV needs to complete the bird driving task in the specified environment space. The experimental results show that the UAV can learn the optimal bird driving strategy more quickly, and can also obtain higher cumulative rewards.

[0146] AsFigure 6 As shown, dqn-ts is the method presented in this paper, dqn-rs is the dqn method with reward constraints, dqn is the dqn method without reward constraints, and hrm is the hierarchical reinforcement learning method. The DQN method performs well in solving various reinforcement learning problems. Reward-constrained DQN introduces additional reward constraints to ensure that the agent meets certain specific task requirements or limitations during the learning process. These constraints can be used to address safety, compliance, or performance requirements in some practical problems. Hierarchical reinforcement learning (HRM) solves complex tasks by introducing a hierarchy and multi-layer policy, improving the applicability and performance of reinforcement learning in complex environments. The reinforcement learning algorithm based on topological sorting reward mechanism in this paper enables the agent to explore more targetedly, prioritizing unexplored states, and thus learning the optimal policy more quickly. As shown in the experimental results in the figure, dqn-ts also exhibits optimal performance. At the same time, it introduces a potential function to help the agent better understand the progress and success of the task. It encourages the agent to focus on the relationship between states and the sequential nature, rather than just the independent rewards of individual states or actions, prompting the agent to make longer-term planning.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0148] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A method for agricultural bird repelling of UAV based on topological ranking reward mechanism, characterized in that, The application relates to a method for planning a path of an unmanned aerial vehicle (UAV) for agricultural bird repelling. The method comprises the following steps: Collecting three-dimensional target data of birds, constructing a bird recognizer, and tracking a three-dimensional target; Automatically generating a reward automaton according to a recognition result, constructing a connection mechanism of the reward automaton and a bottom algorithm, and using a reinforcement learning method to plan a path; Using a topological sorting reward mechanism to improve the reinforcement learning method, using iterative training and optimizing the bottom and top strategies to continuously optimize the path planning method of the UAV; Using the UAV for planning a path based on the topological sorting reward mechanism to perform agricultural bird repelling work; The three-dimensional information of a target is obtained by using a double-camera parallax effect to calculate three-dimensional coordinates of birds and adopting a triangulation method. In the same moment, a bird target is detected in images of a left camera and a right camera, and corresponding pixel coordinates are obtained; the parallax M between the corresponding pixels is calculated through an image processing algorithm; the parallax is obtained by calculating the difference between horizontal pixel coordinates: bird target pixel coordinates in the left camera, bird target pixel coordinates in the right camera; According to camera parameters and geometric relations, a relation between the parallax and an actual depth is obtained; The relation can be obtained by a baseline distance N of the camera, a focal length L and the parallax M: J=(N*L) / M Using the obtained baseline length and the distance of the viewpoint and the calculated parallax value, the depth value J of the target is calculated; in the coordinate system with the sitting camera as the coordinate origin, let the three-dimensional coordinates of the target be (X n , Y n , Z n ), and the corresponding pixel coordinates in the two camera images be and For the detection of the target, it is represented as: Z n = (N*L) / M; The three-dimensional coordinates are converted to a specific coordinate system, and the obtained three-dimensional coordinates are reserved in a sensor as state space variables; A Kalman filter is used to track the target and perform noise reduction processing to obtain an optimal estimation about a target position, including an estimation of a current target position and an estimation of a future position; A prediction step: in an updating step of the Kalman filter, observation data are used to update an estimation of a state vector, the state vector at a time point k is only related to k-1, W(k)=F*W(k-1), wherein W(k) is an estimation value of a current state, and F is a state transition matrix; a covariance matrix updating equation in the prediction step is P(k)=F*P(k-1)*F^T+q, wherein P(k) is a covariance matrix at a current time point, and q is a covariance matrix of process noise; An updating step is performed, and an updated measurement model equation is D(k)=q(k)-H*W(k) Wherein D(k) is a measurement residual, q(k) is a measurement value at a current time point, and H is a measurement matrix; An updated Kalman gain equation is K(k)=P(k)*H^T*(H*P(k)*H^T+R)^-1 Wherein K(k) is a Kalman gain, and R is a covariance matrix of measurement noise; An updated state estimation value updating equation is W(k)=W(k)+K(k)*(z(k)-H*W(k)) An updated covariance matrix updating equation is P(k)=P(k)-K(k)*H*P(k) The bird recognizer comprises a bird classifier constructed by using a deep learning method. W ' (k) = W(k) + K(k) * D(k) Image data sets of different kinds of birds are collected and labeled; a convolutional neural network is used to train the image data to construct the bird classifier. P ' (k) = (I - K(k) * H) * P(k) where I is an identity matrix; repeating the above prediction and update steps, in the update step, the state vector W(k) is corrected according to the measurement model and the Kalman gain model, and the contained velocity component is retained in the sensor as a state space variable.

2. The topology ranking reward mechanism based UAV agricultural bird-scaring method according to claim 1, characterized in that: ​ ​ The classifier monitors birds in the farmland in real time through the camera, judges the sensitive data field of the target picture to be detected, and if there is no sensitive data field, judges the next frame of picture; if the picture has a sensitive data field, uses the YOLO algorithm to detect the target in the target area displayed by the picture; collects the boundary box information of the image in all small areas contained in the picture and classifies and judges the detected bird target through the convolutional neural network, then judges the whole picture, and finally outputs the boundary box and the class information of the target contained in the frame of picture.

3. The topology ranking reward mechanism based UAV agricultural bird-scaring method of claim 2, wherein: The target tracking comprises target monitoring using the camera, and acquiring internal and external parameters of the camera; the internal parameters comprise focal length and principal point coordinates of the camera, and the external parameters comprise position and posture of the camera; The picture is subjected to target detection, and the b-class bird target appearing satisfies b∈B, B being a set of all birds; the position of each target is identified, and an image block around the target is extracted as a feature representation of the target; the convolutional neural network method is used to acquire the target feature; in the next time step, the YOLO algorithm is continuously used for target detection, and the similarity measurement method IoU is used to match the target of the current frame with the target of the previous frame; According to the target matching result, the target tracking is realized by updating the position and state of the target; the continuous tracking algorithm Kalman filter is used to predict the activity path of the bird and track the movement of the bird target; The tracking algorithm predicts and estimates the position of the target in the next frame according to the target detection result of the current frame and the tracking result of the previous frame, so as to realize continuous tracking of the target.

4. The topology ranking reward mechanism based UAV agricultural bird-scaring method of claim 3, wherein: The reinforcement learning path planning method comprises, An automatic reward automaton is automatically generated according to the identification result, a connection mechanism of the reward automaton and the bottom algorithm is constructed, the dependency relationship between each task is clarified, the task hierarchy is defined, the state space and the action space are constructed, the dependency relationship of different birds and the current state is defined, the reward or weight of each task is allocated according to the sorting result of the task, and the state and action space are defined according to the specific demand: the state represents the current state of the system or the observation result of the environment, and the action represents the operation or strategy that can be executed by the reward automaton; the current state and the executed action are mapped to a real reward value by using the reward mechanism, and the result generated by the automatic reward machine is directly used here; A top reward function is designed, the whole project is divided into multiple stages, and the task is divided into multiple subtasks; there is a dependency relationship between each subtask, and the dependency relationship can be represented as a directed acyclic graph DAG, wherein each node represents a subtask, and an edge represents the dependency relationship between tasks; the DAG of the task is topologically sorted, a classic topological sorting algorithm depth-first search DFS is used to realize, and the allocation parameters of the reward are adjusted according to the nature of the project and the performance of the team; the allocation is based on balancing the priority of the task, the size of the reward and the relationship between different tasks, and the constraints are defined according to the execution time and specific conditions.

5. The topology ranking reward mechanism based UAV agricultural bird-scaring method of claim 4, wherein: The generation of the reward automaton comprises, According to the identified bird information, a corresponding reward automaton is generated; the reward automaton provides a reward or punishment signal for the UAV according to a specific behavior rule, guiding its behavior; Combine the reward signal with the path planning algorithm of the UAV, and connect the state and action space of the reward automaton with the bottom algorithm through the establishment of an interface; Output the three-dimensional spatial position result, and unify the target observation result, UAV movement and bird driving effect into a three-dimensional spatial scale; Through observation and perception of the environment, and the generated reward, the following is obtained: MDP M=(S,s0,A,P,γ) Where S represents a finite state set, s0∈S represents an initial state, A represents a finite action set, P∈S×A→Dist(S) represents a transition probability distribution, γ represents a discount factor; R∈S×A×S→R represents the reward assigned to the transition between states; for each step, follow the policy derived from the DQN procedure, which is based on the current state S and derived using the function, Q represents as; Q(<s,e>,a;θ,α,β) Wherein, Q∈SxA→R, <s, e> represents a state, a represents a finite action set, θ represents the network parameters of the convolution layer, and α and β represent the network parameters of the fully connected layer; to guide further action, the policy derived from Q(<s, e>, a; θ, α, β) and the action selected from the action space a, the next state (s ' ,e ' ) is observed, it is judged whether the action meets the policy goal, the calculation of the reward is carried out, and the dynamic state transition and learning policy are progressively established; the reward machine is used to expand the MDP to reward the implementation of each action, and the reward function δ r and the potential function are introduced; the corresponding reward is calculated at a given state and action (e, a); it is assumed that M and N share the same label function L∈SxAxS→F, F is a local action set in a given reward machine and learning policy; if (e, L(s, a, s ' ))∈dom(δ e ), then the reward machine is transferred from the state e to L(s, a, s ' ), otherwise it remains in the state e; wherein the expanded transition probability is the same as the transition probability in the original MDP; given the reward machine N: where E denotes the finite state set, E0∈S denotes the initial state, δ e ∈E×F→E denotes the transition function between states, δ r ∈E×F→R is the reward function of state and transition function, is the potential function of state and MDP M=(S,s0,A,P,γ), where M and N share the label function L, T∈S is the set of accepting states; the extended MDP is defined as where the extended MDP is mapped from the original MDP, and there is a mapping relationship between the states, so Because in the state conversion process, the transition probability is defined based on the state set S and the action set A, and the transition probability is represented as: If the next state of the slot machine is accepted, then it is updated with the reward function δ r If the next state is not accepted, then it needs to be updated with both the reward function δ r and the potential function The update equations are given by: Ensure the communication and coordination mechanism between the bottom strategy and the top strategy, iteratively train and optimize the bottom strategy and the top strategy, and continuously optimize the path planning.

6. The topology ranking reward mechanism based UAV agricultural bird-scaring method of claim 5, wherein: The reward mechanism reinforcement learning planning UAV path based on topological sorting includes, Model the problem, define the hierarchical structure of the task, and decompose the path planning task into different subtasks; Clearly define the classification and attributes of the starting point and the target point in the target space; convert the problem form into a graph, where the nodes are discrete coordinate points in motion, and the edges represent the path; Define and build a complete and clear state space and action space; Establish a topological structure, design a top-level reward function, and generate a reward signal; optimize the path by topological sorting method, and each subtask represents the path planning from the current position to a certain intermediate point; The reward function should encourage the UAV to choose a path that conforms to the topological structure; a potential function is introduced to set the reward mechanism, which gives a positive reward when the UAV chooses a path according to the topological structure, and gives a negative reward when the UAV violates the topological structure, punishing unreasonable path selection; In each subtask, define the bottom strategy and the corresponding environment to guide the reinforcement learning algorithm to select the best action in each subtask; use the DQN algorithm to learn the path planning strategy, so that the UAV can select the appropriate action to drive away the birds according to the current state; use the bottom strategy learning to execute each subtask, and learn a top strategy that decides when to switch to different subtasks; Establish a proper interface or interaction method to ensure proper communication and coordination mechanism between the bottom strategy and the top strategy, realize smooth execution of the task, iteratively train and optimize the bottom strategy and the top strategy, and debug and evaluate the path planning system.

7. An agricultural bird driving system for UAVs based on the reward mechanism of topological sorting according to any one of claims 1-6, characterized in that: The bird tracking and monitoring module uses the monitoring unit to identify and classify the bird target and transmits it to the comprehensive calculation module to obtain three-dimensional information of the birds. The path planning module uses a reinforcement learning method to establish a UAV path planning model, and optimizes the model in combination with a reward mechanism of topological sorting. The information transceiving module receives information input by each module, transmits the information according to requirements, and coordinates operation of each module. The comprehensive calculation module comprehensively calculates information collected by the monitoring module and the model established by the path planning module, and feeds back the information through the information transceiving module.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images