Unmanned aerial vehicle path planning method and device for steel bar binding site and storage medium

Abstract

The invention provides an unmanned aerial vehicle path planning method and device for a steel bar binding site and a storage medium. The method comprises the following steps: constructing an unmanned aerial vehicle path planning environment of a reinforcing steel bar binding site with a Markov property, obtaining a historical initial position and a historical termination position, and inputting the historical initial position, the historical termination position and an unmanned aerial vehicle path planning environment model into an action selection strategy model to obtain a historical unmanned aerial vehicle action strategy, a historical unmanned aerial vehicle action strategy is adopted to control the unmanned aerial vehicle to move, historical real-time operation data are obtained in the moving process, the historical real-time operation data are determined as sample data, a dual strategy learning network model is obtained, and the initial position and the end position of the unmanned aerial vehicle are input into the dual strategy learning network model for prediction. And obtaining the flight path of the unmanned aerial vehicle on the steel bar binding site. The problems that the path length planned by an unmanned aerial vehicle path planning method in the prior art is not shortest and optimal, and the planning time often needs to wait for a long time are solved.

Description

Technical Field

[0001] The present application relates to the technical field of drone path planning at a steel bar binding site, and specifically, to a drone path planning method, device, storage medium and electronic device at a steel bar binding site. Background Art

[0002] Drones can quickly and flexibly perform the task of binding quality inspection at the steel bar binding site. Through path planning, it can be ensured that the drone completes the task according to the optimal path, reducing time and resource waste. Good path planning should not only fully cover the inspection points, but also help the drone avoid obstacles at the construction site, reduce the risk of collision with other construction equipment or personnel, and ensure the safety of the construction site. However, the steel bar binding site is usually complex, with a large number of obstacles such as columns, beams, walls, and dynamically changing working conditions, which increases the difficulty of path planning.

[0003] The path planning methods of UAVs are mainly divided into classical algorithms and intelligent algorithms. Classical algorithms include A* algorithm, artificial potential field method, rapidly explored random tree (RRT) and cell decomposition method. However, traditional global path planning algorithms have the characteristics of fast planning speed but lack of optimality. Planning algorithms that rely on previous environmental information can solve general static scene problems, but they are insufficient in the case of moving obstacles, resulting in limitations.

[0004] Therefore, in response to the challenges of drone path planning in complex scenarios, scholars have proposed bio-inspired intelligent algorithms that simulate the foraging behavior of organisms in dynamic environments. These algorithms include genetic algorithms, ant colony algorithms, and particle swarm algorithms. With the increasing complexity and uncertainty of mission environments, drone path planning faces greater challenges. In recent years, research on drone path planning based on artificial intelligence has increased significantly, further expanding the application of Deep Reinforcement Learning (DRL) in drone path planning. The drone path planning methods of the existing technology cannot avoid obstacles while planning the path, and the path length planned by the drone path planning method is not the shortest and optimal, and the planning time often requires a long wait. Summary of the invention

[0005] The main purpose of this application is to provide a drone path planning method, device, storage medium and electronic device for a steel bar binding site, so as to at least solve the problem that the path length planned by the drone path planning method in the prior art is not the shortest and optimal, and the planning time often requires a long waiting time.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for UAV path planning at a steel bar binding site is provided, comprising: constructing a UAV path planning environment for a steel bar binding site with Markov properties, wherein the UAV path planning environment comprises: an obstacle assessment model, a UAV position model, a UAV motion direction model, and a reward function model; obtaining a historical initial position and a historical end position within a historical time period, and inputting the historical initial position, the historical end position, and the UAV path planning environment model into an action selection strategy model to calculate and obtain a historical UAV action strategy, and using the historical UAV action strategy to control the movement of the UAV and obtaining historical real-time operation data during the movement, and determining the historical real-time operation data as sample data, wherein the sample data comprises: the current position information of the UAV, the current direction information of the UAV, the predicted position information of the UAV at the next moment, the reward data of the UAV at the current moment, and the priority of the sample data; obtaining a dual strategy learning network model, wherein the dual strategy learning network model is trained using the sample data; inputting the initial position and the end position of the UAV into the dual strategy learning network model for prediction to obtain the flight path of the UAV at the steel bar binding site.

[0007] Optionally, a UAV path planning environment model for a steel bar binding site with Markov properties is constructed, including: determining the obstacle assessment model based on the distance between the UAV and the obstacle, the maximum radius setting value of the danger zone, and the maximum radius setting value of the obstacle in the danger zone; constructing the UAV position model and the UAV motion direction model, wherein the UAV position model is used to determine the position of the UAV in two-dimensional space, and the UAV motion direction model is used to divide the area of ​​the steel bar binding site and determine the motion direction of the UAV; constructing a reward function model, and using the reward function model to reward the operating status of the UAV; constructing the UAV path planning environment for the steel bar binding site with the Markov property based on the obstacle assessment model, the UAV position model, the UAV motion direction model and the reward function model.

[0008] Optionally, determining an obstacle assessment model according to the distance between the UAV and the obstacle, the maximum radius setting value of the danger zone, and the maximum radius setting value of the obstacle in the danger zone includes: according to a first formula: , determine the probability of the UAV being crashed, where, is the probability of the UAV being crashed, D is the distance between the UAV and the obstacle, Indicates the maximum radius setting value of the danger zone. is the maximum radius setting value of the obstacle in the danger zone, where: ; According to the second formula: Assess the overall risk of damage to the drone, where: is the overall risk of damage to the drone, k represents the number of dangerous areas, is the probability that the UAV is destroyed when passing through the i-th danger zone; the obstacle assessment model is determined according to the first formula and the second formula.

[0009] Optionally, constructing a reward function model, and using the reward function model to reward the operating state of the drone, includes: using the fifth formula: Construct the reward function model, where R is the reward function, The penalty item for the drone flying into the danger zone is value, is the reward item for the drone to fly to the target location, the reward value is A, is the penalty item for the drone flying out of the steel bar binding site, penalty B value, is the penalty item for the UAV's flight time exceeding the flight time threshold, the penalty C value, is the penalty item of the drone in other set states, penalty D value; wherein, according to the sixth formula Sure Value, where is the overall risk of damage to the drone, is the risk threshold.

[0010] Optionally, the historical initial position, the historical ending position and the UAV path planning environment model are input into the action selection strategy model to calculate the historical UAV action strategy, including: according to the historical initial position and the historical ending position, using a heuristic search algorithm to divide the UAV motion direction model to obtain multiple sub-operating directions; using a greedy strategy algorithm to control the UAV action according to the multiple sub-operating directions to generate multiple historical UAV action strategies.

[0011] Optionally, after adopting the historical drone action strategy to control the movement of the drone and obtaining historical real-time operation data during the movement, and determining the historical real-time operation data as sample data, the method further includes: constructing a priority flight experience storage pool, and inputting the sample data into the priority flight experience storage pool; selecting a preset number of the experience sample data from the priority flight experience storage pool in order from high to low priority, and inputting them into the initial dual strategy learning network model, generating an expected Q value of the evaluation network and a target Q value of the target network, wherein the initial dual strategy learning network model includes the evaluation network and the target network; according to the target Q value and the expected Q value, adopting the third formula: , determine the TD target error value, where, represents the TD target error value, represents the parameters of the target network, is the target Q value, Q is the expected Q value, represents the parameters of the evaluation network, represents the discount factor, which is used to indicate the importance of the UAV to future rewards. A is the movement direction of the UAV. Subscript j represents the jth experience sample data. represents the reward obtained by the drone, The feature vector representing the drone position S is used to approximate the state value, Function represents finding an action The target Q value of the evaluation network Q is maximized; the fourth formula is used according to the TD target error value and the expected Q value of the evaluation network: , determine the loss function of the initial dual strategy learning model, where, is the loss function, represents the priority weight of the jth empirical sample data, is the position of the UAV; and optimizing the parameters of the target network and the evaluation network of the dual strategy learning network according to the loss function to obtain the dual strategy learning model.

[0012] Optionally, after optimizing the parameters of the target network and the evaluation network of the dual strategy learning network according to the loss function, the method further includes: using the seventh formula: , recalculate the priority of the sample data in the priority flight experience storage pool, where, Indicates the priority of the sample data, represents the TD target error value, is the expected Q value of the sample data j, represents the exploration rate.

[0013] According to another aspect of the present application, a UAV path planning device for a steel bar binding site is provided, comprising: a construction unit, for constructing a UAV path planning environment for the steel bar binding site with Markov properties, wherein the UAV path planning environment comprises: an obstacle assessment model, a UAV position model, a UAV motion direction model, and a reward function model; a first acquisition unit, for acquiring a historical initial position and a historical termination position within a historical time period, and inputting the historical initial position, the historical termination position, and the UAV path planning environment model into an action selection strategy model to calculate and obtain a historical UAV action strategy, and adopting the historical UAV action strategy to control the UAV The drone moves and obtains historical real-time operation data during the movement, and determines the historical real-time operation data as sample data, wherein the sample data includes: the current position information of the drone, the current direction information of the drone, the predicted position information of the drone at the next moment, the current reward data of the drone, and the priority of the sample data; a second acquisition unit is used to obtain a dual strategy learning network model, wherein the dual strategy learning network model is trained using the sample data; a prediction unit is used to input the initial position and the terminal position of the drone into the dual strategy learning network model for prediction, so as to obtain the flight path of the drone at the steel bar binding site.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the drone path planning methods for the steel bar binding site.

[0015] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the drone path planning methods for the steel bar binding site.

[0016] Applying the technical solution of the present application, a drone path planning environment with Markov properties is constructed for a steel bar binding site, wherein the drone path planning environment includes: an obstacle assessment model, a drone position model, a drone motion direction model, and a reward function model; the historical initial position and the historical end position within a historical time period are obtained, and the historical initial position, the historical end position, and the drone path planning environment model are input into the action selection strategy model to calculate and obtain the historical drone action strategy, and the historical drone action strategy is used to control the movement of the drone and obtain historical real-time operation data during the movement, and the historical real-time operation data is determined as sample data, and the sample data includes: the current position information of the drone, the current direction information of the drone, the predicted position information of the drone at the next moment, the current reward data of the drone, and the priority of the sample data; a dual strategy learning network model is obtained, wherein the dual strategy learning network model is trained using sample data; the initial position and the end position of the drone are input into the dual strategy learning network model for prediction, and the flight path of the drone at the steel bar binding site is obtained. By using drones to perform steel bar binding quality inspection tasks, a drone path planning environment with Markov properties for the steel bar binding site was established, and a dual strategy learning network model was used to allow the drone agent to better select an action to execute. This enables the drone to make the optimal strategy to plan the inspection flight path to the target location in a shorter time, which is suitable for real-time tasks such as steel bar binding quality inspection, and intelligently avoids possible obstacles at the steel bar binding site. It solves the problem that the path length planned by the drone path planning method in the prior art is not the shortest and optimal, and the planning time often requires a long waiting time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for drone path planning at a steel bar binding site provided in an embodiment of the present application is shown;

[0019] Figure 2 A schematic diagram of a process flow of a method for drone path planning at a steel bar binding site provided according to an embodiment of the present application is shown;

[0020] Figure 3 A schematic flow chart of a method for planning a drone path at a specific steel bar binding site provided in accordance with an embodiment of the present application is shown;

[0021] Figure 4The diagram shows the evaluation network and target network structure of the dual strategy learning network model DSL-NetWork provided according to an embodiment of the present application;

[0022] Figure 5 A structural block diagram of a UAV path planning device for a steel bar binding site provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] As introduced in the background technology, the path length planned by the drone path planning method in the prior art is not the shortest and optimal, and the planning time often requires a long waiting time. In order to solve the problem that the path length planned by the drone path planning method in the prior art is not the shortest and optimal, and the planning time often requires a long waiting time, the embodiments of the present application provide a drone path planning method, device, storage medium and electronic device for a steel bar binding site.

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a method for planning a path of a drone at a steel bar binding site according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the drone path planning method at the steel bar binding site in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific network example may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a drone path planning method for a steel bar binding site is provided, which runs on a mobile terminal, a computer terminal or a similar computing device. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] Figure 2 1 is a flow chart of a method for planning a drone path at a steel bar binding site according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0032] Step S201, constructing a drone path planning environment for a steel bar binding site with Markov properties, wherein the drone path planning environment includes: an obstacle assessment model, a drone position model, a drone motion direction model, and a reward function model;

[0033] The path planning environment includes drones, target locations, danger zones, and environmental boundary zones. The obstacle assessment model, drone position model, drone motion direction model, and reward function model are used to determine the drones, target locations, danger zones, and environmental boundary zones in the path planning environment. The drones can only move within the environmental boundary. Danger zones are areas such as beams, columns, and walls in construction sites that need to be avoided in drone path planning. Usually, drones perform steel bar binding identification and detection tasks at a relatively low constant height. Therefore, the path planning environment scene designed in this scheme is a two-dimensional space.

[0034] The Markov property refers to the property of a random process that it has no aftereffect, that is, the future state is only related to the current state, but not to the past state. Specifically, for a random process, if the conditional probability distribution of the future state, given the current state, depends only on the current state, but not on the past state sequence, then the random process has the Markov property. The Markov property is an important assumption in many random processes, which simplifies the modeling and analysis of reinforcement learning.

[0035] Step S202, obtaining the historical initial position and the historical end position within the historical time period, and inputting the historical initial position, the historical end position and the drone path planning environment model into the action selection strategy model to calculate and obtain the historical drone action strategy, and using the historical drone action strategy to control the movement of the drone and obtain historical real-time operation data during the movement, and determining the historical real-time operation data as sample data, the sample data includes: the current moment position information of the drone, the current moment direction information of the drone, the next moment predicted position information of the drone, the current moment reward data of the drone, and the priority of the sample data;

[0036] Specifically, in each time step of the path planning process, the agent outputs an action to make a decision. This scheme combines heuristic rules and ε-greedy strategy to let the drone make action decisions and obtain sample data to optimize the action selection during training and improve learning efficiency.

[0037] Step S203, obtaining a dual strategy learning network model, wherein the dual strategy learning network model is obtained by training with sample data;

[0038] Specifically, a duality strategy learning network DSL-NetWork (Duality Strategy Learning Network) is constructed, which consists of an evaluation network and a target network. The network structure of the duality strategy learning network DSL-NetWork is as follows: Figure 3 As shown in the figure, the neural network structure of the evaluation network and the target network is the same. A fully connected layer L1 with 128 neurons is built, a ReLu activation function layer is built, and a fully connected layer L2 with 64 neurons is built, and a ReLu activation function layer is built. The function of the fully connected layer is to approximate the value function. w is the learnable parameter of the evaluation network, and the target network copies w every C rounds as its own operating parameter w'.

[0039] Step S204, inputting the initial position and the final position of the UAV into the dual strategy learning network model for prediction, and obtaining the flight path of the UAV at the steel bar binding site.

[0040] Specifically, when planning the drone path for the on-site inspection of steel bar binding quality, the dangerous area will be delineated according to the on-site environment, and the starting position and target position will be set. The drone agent will use the trained dual strategy learning network to select a reasonable inspection flight path that covers all routes and intelligently avoids dangerous areas.

[0041] Through this embodiment, a UAV path planning environment for a steel bar binding site with Markov properties is constructed, wherein the UAV path planning environment includes: an obstacle assessment model, a UAV position model, a UAV motion direction model, and a reward function model; the historical initial position and the historical end position within the historical time period are obtained, and the historical initial position, the historical end position and the UAV path planning environment model are input into the action selection strategy model to calculate and obtain the historical UAV action strategy, and the historical UAV action strategy is used to control the movement of the UAV and obtain historical real-time operation data during the movement, and the historical real-time operation data is determined as sample data, and the sample data includes: the current position information of the UAV, the current direction information of the UAV, the predicted position information of the UAV at the next moment, the reward data of the UAV at the current moment, and the priority of the sample data; a dual strategy learning network model is obtained, wherein the dual strategy learning network model is obtained by training with sample data; the initial position and the end position of the UAV are input into the dual strategy learning network model for prediction, and the flight path of the UAV at the steel bar binding site is obtained. By using drones to perform steel bar binding quality inspection tasks, a drone path planning environment with Markov properties for the steel bar binding site was established, and a dual strategy learning network model was used to allow the drone agent to better select an action to execute. This enables the drone to make the optimal strategy to plan the inspection flight path to the target location in a shorter time, which is suitable for real-time tasks such as steel bar binding quality inspection, and intelligently avoids possible obstacles at the steel bar binding site. It solves the problem that the path length planned by the drone path planning method in the prior art is not the shortest and optimal, and the planning time often requires a long waiting time.

[0042] In the specific implementation process, a UAV path planning environment model for the steel bar binding site with Markov properties is constructed, including: determining an obstacle assessment model based on the distance between the UAV and the obstacle, the maximum radius setting value of the danger zone, and the maximum radius setting value of the obstacle in the danger zone; constructing a UAV position model and a UAV motion direction model, wherein the UAV position model is used to determine the position of the UAV in two-dimensional space, and the UAV motion direction model is used to divide the area of ​​the steel bar binding site and determine the motion direction of the UAV; constructing a reward function model, and using the reward function model to reward the operating status of the UAV; constructing a UAV path planning environment for the steel bar binding site with Markov properties based on the obstacle assessment model, the UAV position model, the UAV motion direction model and the reward function model.

[0043] Since the path planning environment scene is a two-dimensional space, the coordinate points are used To describe the position of the UAV in the environment, we obtain the UAV position model and use this position as the position variable S.

[0044] In order to improve the efficiency of path planning, the environment area is divided into nine grids with three rows and three columns. The size of the grid can be designed according to the size of the specific steel bar binding quality inspection area. Assuming that the drone can fly freely to any of the eight surrounding grids, the discrete action space consists of eight actions; the action space A of the drone motion direction model is expressed as: ; The digital codes (0, 1, 2, 3, 4, 5, 6, 7) in the action space A of the drone motion direction model are used to represent the corresponding moving direction, that is, moving from the current grid position to the adjacent grid position, which can be changed by programming. For example: 0 means moving north, 1 means moving south, 2 means moving west, 3 means moving east, 4 means moving northwest, 5 means moving northeast, 6 means moving southwest, and 7 means moving southeast.

[0045] Specifically, the obstacle assessment model is determined according to the distance between the UAV and the obstacle, the maximum radius setting value of the danger zone, and the maximum radius setting value of the obstacle in the danger zone, including: according to the first formula: , determine the probability of the drone being crashed, where is the probability of the UAV being crashed, D is the distance between the UAV and the obstacle, Indicates the maximum radius setting value of the danger zone. is the maximum radius setting value of the obstacle in the danger zone, where: ; According to the second formula: Assess the overall risk of damage to the drone, where: is the overall risk of damage to the drone, k represents the number of dangerous areas, is the probability that the UAV is destroyed when passing through the i-th danger zone; the obstacle assessment model is determined according to the first formula and the second formula.

[0046] in, and The relationship can be set as .

[0047] This method calculates the probability of the drone being destroyed in real time and the probability of the drone passing through the The probability of being destroyed when entering a dangerous area can reduce the probability of the drone crashing.

[0048] More specifically, a reward function model is constructed, and the reward function model is used to reward the operating state of the drone, including: using the fifth formula: Construct a reward function model, where R is the reward function, Penalty for drones flying into danger zones. value, The reward item for the drone to fly to the target location is rewarded with a value of A. This is the penalty item for the drone flying out of the steel bar binding site, the penalty value is B. is the penalty item when the flight time of the drone exceeds the flight time threshold, the penalty C value, is the penalty item of the drone in other set states, penalty D value; according to the sixth formula Sure Value, where is the overall risk of damage to the drone, is the risk threshold.

[0049] Among them, A can be set to 200, B can be set to 50, C can be set to 50, and other states can be the normal flight state or stationary state of the drone. D can be set to 0.5, so as to encourage the drone to reach the target area as soon as possible with a smaller penalty value, and avoid the drone adopting the strategy of staying still.

[0050] The Method Overall risk of damage to the drone Related. Exceeding the threshold When the probability of the drone hitting obstacles such as beams and columns at the steel bar binding site is 100%, the penalty item is set to -50. When the drone is below the threshold, there is a certain probability that the drone will collide. , , , If any of the four required conditions is met, the flight round ends; otherwise, the drone remains in other states.

[0051] Furthermore, the historical initial position, historical ending position and UAV path planning environment model are input into the action selection strategy model to calculate the historical UAV action strategy, including: according to the historical initial position and the historical ending position, a heuristic search algorithm is used to divide the UAV motion direction model to obtain multiple sub-operating directions; a greedy strategy algorithm is used to control the UAV action according to the multiple sub-operating directions to generate multiple historical UAV action strategies.

[0052] This method further divides the action space A of the UAV motion direction model into a smaller action space A' (sub-running direction) according to the UAV's historical initial position and historical end position. The smaller action space consists of three actions selected in the target direction and two actions at the boundary of the target direction. In this way, the UAV selects an action in a smaller action space, which improves training efficiency, shortens training cycle, and prevents falling into the local optimal problem.

[0053] For example: Indicates the current position of the drone. Indicates the location of the target. If the target is located southeast of the drone, and ,but The included movement directions are east, south, southeast, southwest and northeast; if the target position is northeast of the drone, it satisfies and ,but The included movement directions are northwest, north, northeast, east, and southeast; if the target position is northwest of the drone, it satisfies and ,but The included movement directions are northeast, north, northwest, west, and southwest; if the target location is southwest of the drone, it satisfies and ,but The included movement directions are northwest, west, southwest, south, and southeast.

[0054] Furthermore, after adopting the historical UAV action strategy to control the movement of the UAV and obtaining the historical real-time operation data during the movement, and determining the historical real-time operation data as sample data, the method also includes: constructing a priority flight experience storage pool, and inputting the sample data into the priority flight experience storage pool; selecting a preset number of experience sample data from the priority flight experience storage pool in order from high to low priority, and inputting them into the initial dual strategy learning network model, generating the expected Q value of the evaluation network and the target Q value of the target network, wherein the initial dual strategy learning network model includes the evaluation network and the target network; according to the target Q value and the expected Q value, adopting the third formula: , determine the TD target error value, where, represents the TD target error value, represents the parameters of the target network, is the target Q value, Q is the expected Q value, represents the parameters of the evaluation network, represents the discount factor, which is used to indicate the importance that the drone places on future rewards. A is the direction of motion of the drone. Subscript j represents the jth empirical sample data. Indicates the reward obtained by the drone. The feature vector representing the drone position S is used to approximate the state value, Function represents finding an action The target Q value of the evaluation network Q is maximized; the fourth formula is used according to the TD target error value and the expected Q value of the evaluation network: , determine the loss function of the initial dual strategy learning model, where, is the loss function, represents the priority weight of the jth experience sample data, is the position of the UAV; according to the loss function, the parameters of the target network and the evaluation network of the dual strategy learning network are optimized to obtain the dual strategy learning model.

[0055] Among them, the priority flight experience storage pool gives priority to experience samples through the priority function to improve learning efficiency. Unlike the traditional experience playback pool, the priority flight experience storage pool of this scheme does not sample experience samples evenly, but gives priority to samples with higher priority. This allows the learning algorithm to pay more attention to those experiences that are more important for learning, thereby improving learning efficiency.

[0056] Specifically, after optimizing the parameters of the target network and the parameters of the evaluation network of the dual strategy learning network according to the loss function, the method further includes: using the seventh formula: , recalculate the priority of the sample data in the priority flight experience storage pool, where, Indicates the priority of the sample data, represents the TD target error value, is the expected Q value of the sample data j, represents the exploration rate.

[0057] The priority function uses the TD error of the experience sample as the input of the priority function to assign a priority. An experience sample with a large TD error indicates that the UAV agent has not learned the experience sample well. The priority function will assign a relatively large value to this experience sample, making it easier for the dual strategy learning network to sample it.

[0058] ;

[0059] ;

[0060] ;

[0061] represents the probability that the tth experience is sampled, Indicates the priority of sample data, is a hyperparameter that controls the decay rate of the priority. Represents the TD error, which is used to measure the estimated value of the expected reward calculated after the drone agent performs an action With TD target The difference between.

[0062] When the experience samples stored in the experience replay pool exceed After the experience replay pool is built, some old experience samples need to be deleted. This solution adopts a priority-based deletion strategy, that is, those experience samples with lower priorities are deleted first. In order to improve the training speed, after the experience replay pool is built, the priority of each experience sample is no longer dynamically updated during the training process.

[0063] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the drone path planning method for the steel bar binding site of the present application will be described in detail below in combination with specific embodiments.

[0064] This embodiment relates to a specific method for planning a drone path at a steel bar binding site. Figure 4 As shown, the following steps are included:

[0065] Step S1: Building a UAV path planning environment for the steel bar binding site with Markov properties;

[0066] The path planning environment includes the drone, the target location, the danger zone and the environmental boundary zone. The drone can only move within the environmental boundary. The danger zone is the area where the drone path planning needs to avoid, such as beams, columns, walls, etc. in the construction site. Usually, the drone performs the steel bar binding identification and detection task at a lower constant height. Therefore, the path planning environment scene designed in this embodiment is a two-dimensional space.

[0067] Step S 1.1: Construct an obstacle assessment method; construct an obstacle assessment method based on the distance between the UAV and the center of the danger zone. The degree of danger of the center of the danger zone to the UAV Defined by the following formula, Indicates the probability of the drone being crashed.

[0068] ; Where D represents the distance between the drone and the center of the obstacle. Indicates the maximum radius of the danger zone. Indicates the maximum radius of obstacle entities within the danger zone. and The difference is 1 meter.

[0069] Assuming that the probability of a drone crashing in the danger zones formed by obstacles such as columns and beams is independent of each other, the following formula is used to calculate the overall risk of damage to drones flying from each danger zone: .

[0070] ;

[0071] Where k represents the number of dangerous areas. It was a drone passing through The probability of being destroyed when in a danger zone.

[0072] Step S1.2: construct state space;

[0073] The path planning environment scene is a two-dimensional space. We use coordinate points (x, y) to describe the position of the drone in the environment and use this position as the state variable S of the Markov decision process (MDP) model. .

[0074] Step S 1.3: Construct action space;

[0075] In order to improve the efficiency of path planning, the environment area is divided into nine grids with three rows and three columns. The size of the grid can be designed according to the size of the specific steel bar binding quality inspection area. Assuming that the drone can fly freely to any of the eight surrounding grids, the discrete action space consists of eight actions.

[0076] Action Space It is expressed as the following equation: ;

[0077] Action Space The digital codes (0, 1, 2, 3, 4, 5, 6, 7) in are used to represent the corresponding moving direction, that is, moving from the current grid position to the adjacent grid position, which can be changed by programming.

[0078] For example: 0 means moving north, 1 means moving south, 2 means moving west, 3 means moving east, 4 means moving northwest, 5 means moving northeast, 6 means moving southwest, and 7 means moving southeast.

[0079] Step S 1.4: Construct a reward function to guide the UAV agent to learn the optimal strategy;

[0080] Construct the reward function R, which consists of five reward terms and is expressed by the following equation:

[0081] ;

[0082] Among the rewards , , , and Represents the reward and penalty items of the drone in different states. A penalty item representing flying into a danger zone. value. Represents a reward item for flying back to a specified target point, with a reward of 200 points. Represents the penalty item for flying out of the environment boundary, with a penalty value of -50. The penalty item represents the flight time of the drone exceeding the maximum flight duration limit, with a penalty value of -50. Represents the penalty item of the drone in other states, with a penalty value of -0.5. A smaller penalty value is used to encourage the drone to reach the target area as soon as possible and avoid the drone adopting a strategy of staying still.

[0083] Total hazard level Related. Exceeding the threshold When the probability of the drone hitting obstacles such as beams and columns at the steel bar binding site is 100%, the penalty item is set to -50. When the value is lower than the threshold, the drones have a certain probability of collision, which can be expressed as: ;

[0084] When the drone status meets , , , When any of the four required conditions is met, this flight round ends; otherwise, the drone remains in other states, which means that the drone is flying normally or stationary.

[0085] Among them, in reinforcement learning, the intelligent agent (drone) refers to the entity involved in learning and decision-making, which is at the forefront of interacting with the environment. The intelligent agent learns and improves its behavior by observing the state of the environment, selecting actions and receiving rewards to achieve a predetermined goal. Its main task is to select the optimal action based on the feedback (reward signal) provided by the environment to maximize the long-term cumulative reward.

[0086] The Markov property refers to the property of a random process that it has no aftereffect, that is, the future state is only related to the current state, but not to the past state. Specifically, for a random process, if the conditional probability distribution of the future state, given the current state, depends only on the current state, but not on the past state sequence, then the random process has the Markov property. The Markov property is an important assumption in many random processes, which simplifies the modeling and analysis of reinforcement learning.

[0087] Step S2: Formulate action selection strategy;

[0088] In each time step of the path planning process, the agent outputs an action to make a decision. This embodiment combines heuristic rules and ε-greedy strategy to let the agent make action decisions to optimize the action selection during training and improve learning efficiency.

[0089] Step S2.1: Use a heuristic search algorithm to divide the action space into smaller ones;

[0090] The action space constructed from Step 1.2 based on the current position and target position of the drone Further divide the action space into smaller The smaller action space consists of three actions selected in the target direction and two actions at the boundary of the target direction. In this way, the drone selects an action in a smaller action space, which improves training efficiency, shortens training cycle, and prevents falling into the local optimal problem.

[0091] set up Indicates the current position of the drone. Indicates the location of the target. If the target is located southeast of the drone, and ,but The included movement directions are east, south, southeast, southwest and northeast; if the target position is northeast of the drone, it satisfies and ,but The included movement directions are northwest, north, northeast, east, and southeast; if the target position is northwest of the drone, it satisfies and ,but The included movement directions are northeast, north, northwest, west, and southwest; if the target location is southwest of the drone, it satisfies and ,but The included movement directions are northwest, west, southwest, south, and southeast.

[0092] For example:

[0093] According to the action space defined in step S1.3, if the target position is southeast of the drone, and , then the drone agent will try all possible actions from the action space A = (1, 3, 5, 6, 7).

[0094] Similarly, if the target location is northeast of the drone, and , then the drone agent will try all possible actions from the action space A = (0, 3, 4, 5, 7).

[0095] If the target location is northwest of the drone, and , then the drone agent will try all possible actions from the action space A = (0, 2, 4, 5, 6).

[0096] If the target location is southwest of the drone, and , then the drone agent will try all possible actions from the action space A = (0, 2, 4, 6, 7).

[0097] Step S2.2: Use the greedy strategy to select an action to execute in the action space;

[0098] set up Represents the exploration rate, which indicates the probability that the UAV agent randomly selects an action to execute in the action space. It is defined by the following equation, where Represents the initial exploration rate of the UAV agent. represents the final exploration rate. i represents the current training round, Indicates the pre-training iterations that generate the specified amount of data. represents the number of exploration rate decay steps, Indicates the maximum number of exploration rate decay steps.

[0099] ;

[0100] In the entire actual training process, first give Set a relatively high initial value , so that the drone can tend to randomly select different actions to execute in the smaller action space obtained in step S2.1. As the number of training iterations increases, it gradually decreases until it stabilizes so that the agent tends to choose actions with higher Q values ​​in the smaller action space obtained in step S2.1.

[0101] Among them, in reinforcement learning, Q-value is an indicator used to measure the value of taking an action in a given state. Specifically, for state s and action a, the Q-value is expressed as Q(s, a), which represents the expected cumulative reward that can be obtained by taking action a in state s. The Q-value function is a key concept in reinforcement learning. It can help the intelligent agent (drone) choose the optimal action to take in each state, so that the agent can obtain the maximum cumulative reward in the environment.

[0102] Step S3: construct a dual strategy learning network;

[0103] Step S3.1: Construct the dual strategy learning network DSL-NetWork (Duality Strategy Learning Network), which consists of an evaluation network and a target network. The neural network structure of the evaluation network and the target network is the same. Build a fully connected layer L1 with 128 neurons, build a ReLu activation function layer, build a fully connected layer L2 with 64 neurons, and build a ReLu activation function layer. The function of the fully connected layer is to approximate the value function. w is the learnable parameter of the evaluation network, and the target network copies w as its own operating parameter w' every C rounds. The network structure of DSL-NetWork is as follows Figure 3 shown.

[0104] Step S3.2: Both the evaluation network and the target network accept state S as input and output the value function value respectively and advantage function value The value function is completely dependent on the state S and independent of the action A, denoted as , are the parameters of the value function V, The output of is only one value, which represents the expected value of the long-term cumulative reward that the agent can obtain under the input state S. The advantage function is related to both state S and action A, and is recorded as , Represents the advantage function The output of the advantage function has eight values, indicating that when the same state S is input, different actions chosen by the agent will affect the immediate rewards obtained in the short term. The eight output values ​​represent the pros and cons of the eight actions in the action space, and the higher the value, the better the action.

[0105] Step S 3.3: Calculate the expected Q value of the evaluation network Q and the target network The target Q value is calculated as follows:

[0106] ;

[0107] ;

[0108] Where |A| is equal to the size of the action space. The learnable parameters in the neural network are updated during the training process through optimization algorithms such as gradient descent to enable the network to adapt to the training data and improve performance.

[0109] For example: value function value is 100, the advantage function value The eight values ​​are [5, 2, 4, 3, 1, 5, 6, 7, 8]. The calculation is (5+2+4+3+4+1+6+7+8) / 8=4.5. Calculation formula: 100+(5-4.5)=100.5; 100+(2-4.5)=97.5; 100+(4-4.5)=99.5; 100+(3-4.5)=98.5; 100+(1-4.5)=96.5; 100+(5-4.5)=100.5; 100+(6-4.5)=101.5; 100+(7-4.5)=102.5; 100+(8-4.5)=103.5;

[0110] The expected Q value of the evaluation network Q is [100.5, 97.5, 99.5, 98.5, 96.5, 101.5, 102.5, 103.5].

[0111] Step S4: construct a priority flight experience storage pool;

[0112] The priority flight experience storage pool gives priority to experience samples through a priority function to improve learning efficiency. Unlike the traditional experience playback pool, the priority flight experience storage pool of this embodiment does not sample experience samples evenly, but instead prioritizes samples with higher priorities. This allows the learning algorithm to pay more attention to experiences that are more important to learning, thereby improving learning efficiency.

[0113] Step S4.1: Initialize the priority flight experience storage pool;

[0114] Create an empty priority flight experience storage pool that can store N experiences to store experience samples. Each experience sample contains the following information: the state at that moment ,action ,award , the next moment state , whether the task is completed d, priority P. Among them, the storage capacity "N" of the priority flight experience pool of this embodiment is 1,000,000.

[0115] Step S4.2: define a priority function P(t) to measure the importance of each experience sample;

[0116] The priority function uses the TD error of the experience sample as the input of the priority function to assign a priority. An experience sample with a large TD error indicates that the UAV agent has not yet learned the experience sample well. The priority function will assign a relatively large value to this experience sample, making it easier for this experience sample to be sampled by the dual strategy learning network.

[0117] ;

[0118] ;

[0119] ;

[0120] represents the probability that the tth experience is sampled, Indicates the priority of sample data, is a hyperparameter that controls the decay rate of the priority. Represents the TD error, which is used to measure the estimated value of the expected reward calculated after the drone agent performs an action With TD target The difference between.

[0121] When the experience samples stored in the experience replay pool exceed After the experience replay pool is built, some old experience samples need to be deleted. This embodiment adopts a priority-based deletion strategy, that is, those experience samples with lower priorities are deleted first. In order to improve the training speed, after the experience replay pool is built, the priority of each experience sample is no longer dynamically updated with the training process.

[0122] Among them, in reinforcement learning, TD (Temporal Difference) error is an error metric used to evaluate prediction models or value functions. TD error can be calculated by taking the difference between the estimated value of the prediction model and the actual observed reward value or the estimated value of the next state.

[0123] Among them, Experience Replay Pool is an important technology in deep reinforcement learning, which is usually used to train algorithms such as Deep Q Network (DQN). Its main purpose is to store the experience of the agent's interaction with the environment and randomly sample from it during training to smoothly train the neural network and improve the efficiency and stability of training. The working principle of Experience Replay Pool is as follows:

[0124] 1. Storing experience: When the agent interacts with the environment, it saves the experience of each step (including state, action, reward, next state, etc.) into the experience replay pool.

[0125] 2. Random sampling: During the training process, instead of directly using the latest collected experience to update the neural network parameters, a batch of experience is randomly sampled from the experience replay pool.

[0126] 3. Stable training: Random sampling can make the training samples more stable, reduce the data correlation during the training process, and prevent the correlation between training samples from being too high, which will lead to unstable neural network training.

[0127] 4. Reuse of experience: The experience replay pool allows the agent to reuse previous experience multiple times, which can improve the efficiency of data utilization, especially when samples are scarce.

[0128] Among them, fully connected layers play an important role in deep learning. They can learn the complex relationships between input data and map these relationships to the output space. In the forward propagation process of the neural network, the fully connected layer is responsible for linear and nonlinear transformation of the input data to generate the output of the network.

[0129] Step S5: training a dual strategy learning network suitable for UAV path planning at the steel bar binding site;

[0130] Step S5.1: Initialize the UAV path planning environment. Set the initial state of the UAV agent , that is, the starting coordinates of the drone; set the coordinates of the target position; calibrate the positions of dangerous areas such as columns, beams, and walls at the steel bar binding site.

[0131] Step S5.2: Randomly initialize the dual strategy learning network and its network parameters w and w', and set the maximum number of rounds of network training , the number of rounds C of copying the target network, the optimizer used by the network and learning rate ; Set the discount factor for the environment ; Set the greedy strategy parameters in the action selection strategy , the round number parameter required to calculate the exploration rate and ; Set the maximum capacity of the priority flight experience pool and single sampling empirical sample size ;

[0132] Step S5.3: The UAV agent selects an action based on the action selection strategy in S2 Execute and enter the next state The environment calculates the reward value for this action based on the coordinate state of the drone agent. , assign a value to the task completion d according to whether the target position is reached, and the priority P of the experience sample is initialized to 1. Stored in the priority flight experience pool constructed in S5. Repeat S5.3 until the stored experience samples in the priority flight experience pool exceed , execute S5.4.

[0133] Step S5.4: Sampling from the priority flight experience pool according to priority The experience samples are used as the input of the dual strategy learning network, and the expected Q value of the evaluation network Q and the target network Q are obtained through network calculation. The target Q value.

[0134] Step S5.5: Use the target network The target Q value of TD target is calculated, denoted as , the calculation formula is as follows:

[0135] ;

[0136] in Indicates the target network Parameters, Represents the discount factor, which is used to indicate the importance that the drone agent places on future rewards, and its value range is [0,1]. The subscript j represents this round. Indicates the reward obtained in this round according to S2.3. The feature vector representing the state S is used to approximate the state value. The function represents finding an action A that maximizes the target Q value of the evaluation network Q.

[0137] Step S5.6: Use the expected Q value and TD target of the evaluation network Q , the loss function Loss is calculated by the following formula to optimize the dual strategy learning network: ;

[0138] in, represents the priority weight of the jth sample.

[0139] Step S5.7: Update the parameters w and of the dual strategy learning network according to the calculated Loss value , so that the Loss value obtained by subsequent calculations continues to decrease. Every time w is updated C times, execute once .

[0140] Step S5.8: Recalculate the priority of the experience samples in the experience pool according to the priority function of S4.2 .

[0141] Step S5.9: When the state of the drone agent When it is the target state, the current round of UAV path planning task is completed. If the training round reaches episodes times, the training ends and a UAV agent suitable for UAV path planning at the steel bar binding site is obtained. Otherwise, it goes to S5.1 for a new round of training. When S is not the target state, it goes to step S5.3 for execution.

[0142] The following hyperparameters can be set during training: the maximum number of episodes is set to 20000, the number of episodes of copying the target network is set to 8, the optimizer used is set to Adam optimizer, the learning rates of the evaluation network and the target network are set to 5e-4 and 4e-4 respectively, the discount factor γ of the environment is set to 0.99, and the greedy strategy parameter is set to 0. Set to 0.95, the number of rounds required for the exploration rate parameter and Set to 500 and 12000, giving priority to the maximum capacity of the flight experience pool Set to 1,000,000, the empirical sample size for a single sampling Set to 256.

[0143] Step S6: Applying the dual strategy learning network to perform UAV path planning at the steel bar binding quality inspection site;

[0144] When planning the drone path for the rebar binding quality inspection site, the dangerous area is delineated according to the on-site environment and the starting and target positions are set. The drone agent will use the trained dual strategy learning network to select a reasonable inspection flight path that covers all routes and intelligently avoids dangerous areas.

[0145] The embodiment of the present application establishes a Markov decision model of the steel bar binding site environment based on the task of using drones to detect the quality of steel bar binding, constructs the state space, action space and reward function of the drone agent, and uses a heuristic search algorithm and a greedy strategy to allow the drone agent to better select an action to execute. A dual strategy learning network is designed to calculate the real and expected Q values ​​for path search, and combined with the training of the constructed priority flight experience storage pool, it can comprehensively learn the advantages and disadvantages of executing each action, thereby ensuring that the drone agent selects the optimal flight strategy.

[0146] This embodiment enables the drone to make the best strategy to plan the detection flight path to the target location in a shorter time, which is suitable for real-time tasks such as steel bar binding quality inspection, and intelligently avoids obstacles that may exist at the steel bar binding site. Compared with the traditional DQN method and DDAN method, it achieves a more stable cumulative reward and converges faster, can more effectively and accurately approximate the Q function, and can better adapt to the conditions of different steel bar binding sites.

[0147] The embodiment of the present application also provides a UAV path planning device for a steel bar binding site. It should be noted that the UAV path planning device for a steel bar binding site in the embodiment of the present application can be used to execute the UAV path planning method for a steel bar binding site provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and those that have been described will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0148] The following is an introduction to the drone path planning device for the steel bar binding site provided in an embodiment of the present application.

[0149] Figure 5 Schematic diagram of a drone path planning device for a steel bar binding site according to an embodiment of the present application. Figure 5 As shown, the device includes: a construction unit 51, a first acquisition unit 52, a second acquisition unit 53, and a prediction unit 54.

[0150] A construction unit 51 is used to construct a UAV path planning environment for a steel bar binding site with Markov properties, wherein the UAV path planning environment includes: an obstacle assessment model, a UAV position model, a UAV motion direction model, and a reward function model;

[0151] Among them, the Markov property refers to the characteristic of a random process that it has no aftereffect, that is, the future state is only related to the current state, but has nothing to do with the past state.

[0152] The first acquisition unit 52 is used to acquire the historical initial position and the historical end position within the historical time period, and input the historical initial position, the historical end position and the drone path planning environment model into the action selection strategy model to calculate and obtain the historical drone action strategy, and use the historical drone action strategy to control the movement of the drone and acquire the historical real-time operation data during the movement, and determine the historical real-time operation data as sample data, the sample data includes: the current moment position information of the drone, the current moment direction information of the drone, the next moment predicted position information of the drone, the current moment reward data of the drone, and the priority of the sample data;

[0153] Specifically, in each time step of the path planning process, the agent outputs an action to make a decision. This scheme combines heuristic rules and ε-greedy strategy to let the drone make action decisions and obtain sample data to optimize the action selection during training and improve learning efficiency.

[0154] A second acquisition unit 53 is used to acquire a dual strategy learning network model, wherein the dual strategy learning network model is obtained by training with sample data;

[0155] Among them, the duality strategy learning network DSL-NetWork (Duality Strategy Learning Network) consists of an evaluation network and a target network.

[0156] The prediction unit 54 is used to input the initial position and the terminal position of the UAV into the dual strategy learning network model for prediction, so as to obtain the flight path of the UAV at the steel bar binding site.

[0157] Specifically, when planning the drone path for the on-site inspection of steel bar binding quality, the dangerous area will be delineated according to the on-site environment, and the starting position and target position will be set. The drone agent will use the trained dual strategy learning network to select a reasonable inspection flight path that covers all routes and intelligently avoids dangerous areas.

[0158] In this embodiment, a construction unit is used to construct a drone path planning environment for a steel bar binding site with Markov properties, wherein the drone path planning environment includes: an obstacle assessment model, a drone position model, a drone motion direction model, and a reward function model; a first acquisition unit is used to obtain a historical initial position and a historical end position within a historical time period, and the historical initial position, the historical end position, and the drone path planning environment model are input into an action selection strategy model to calculate and obtain a historical drone action strategy, and the historical drone action strategy is used to control the movement of the drone and obtain historical real-time operation data during the movement, and the historical real-time operation data is determined as sample data, and the sample data includes: the current moment position information of the drone, the current moment direction information of the drone, the next moment predicted position information of the drone, the current moment reward data of the drone, and the priority of the sample data; a second acquisition unit is used to obtain a dual strategy learning network model, wherein the dual strategy learning network model is obtained by training with sample data; a prediction unit is used to input the initial position and the end position of the drone into the dual strategy learning network model for prediction, and obtain the flight path of the drone at the steel bar binding site. By using drones to perform steel bar binding quality inspection tasks, a drone path planning environment with Markov properties for the steel bar binding site was established, and a dual strategy learning network model was used to allow the drone agent to better select an action to execute. This enables the drone to make the optimal strategy to plan the inspection flight path to the target location in a shorter time, which is suitable for real-time tasks such as steel bar binding quality inspection, and intelligently avoids possible obstacles at the steel bar binding site. It solves the problem that the path length planned by the drone path planning method in the prior art is not the shortest and optimal, and the planning time often requires a long waiting time.

[0159] As an optional solution, the construction unit includes a first determination module, a first construction module, a second construction module, and a third construction module; the first determination module is used to determine the obstacle assessment model based on the distance between the drone and the obstacle, the maximum radius setting value of the danger zone, and the maximum radius setting value of the obstacle in the danger zone; the first construction module is used to construct a drone position model and a drone movement direction model, wherein the drone position model is used to determine the position of the drone in two-dimensional space, and the drone movement direction model is used to divide the area of ​​the steel bar binding site and determine the movement direction of the drone; the second construction module is used to construct a reward function model, and the reward function model is used to reward the operating status of the drone; the third construction module is used to construct a drone path planning environment for the steel bar binding site with Markov properties based on the obstacle assessment model, the drone position model, the drone movement direction model, and the reward function model. Since the path planning environment scene is a two-dimensional space, the coordinate point To describe the position of the UAV in the environment, obtain the UAV position model, and use this position as the position variable .

[0160] In an optional solution, the first determination module includes a first determination submodule, an evaluation submodule, and a second determination submodule; the first determination submodule is used to determine the value of the first formula: , determine the probability of the drone being crashed, where, is the probability of the drone being crashed, D is the distance between the drone and the obstacle, Indicates the maximum radius setting value of the danger zone. is the maximum radius setting value of obstacles in the danger zone, where: ; According to the second formula: Assess the overall risk of damage to the drone, where: is the overall risk of damage to the drone, k represents the number of dangerous areas, is the probability that the UAV is destroyed when passing through the i-th danger zone; the obstacle assessment model is determined according to the first formula and the second formula.

[0161] Specifically, the probability of the drone being destroyed is calculated in real time and the probability of the drone passing through the The probability of being destroyed when entering a dangerous area can reduce the probability of the drone crashing.

[0162] In an optional solution, the second building block includes a building sub-module for adopting the fifth formula: Construct a reward function model, where R is the reward function, Penalty for drones flying into danger zones. value, The reward item for the drone to fly to the target location is rewarded with a value of A. This is the penalty item for the drone flying out of the steel bar binding site, the penalty value is B. is the penalty item when the flight time of the drone exceeds the flight time threshold, the penalty C value, is the penalty item of the drone in other set states, penalty D value; according to the sixth formula Sure Value, where is the overall risk of damage to the drone, is the risk threshold.

[0163] Among them, A can be set to 200, B can be set to 50, C can be set to 50, and other states can be the normal flight state or stationary state of the drone. D can be set to 0.5, so as to use a smaller penalty value to encourage the drone to reach the target area as soon as possible and avoid the drone adopting the strategy of staying in place.

[0164] An optional solution, the first acquisition unit includes a division module and a control module; the division module is used to divide the UAV motion direction model according to the historical initial position and the historical end position using a heuristic search algorithm to obtain multiple sub-operating directions; the control module is used to control the UAV action according to the multiple sub-operating directions using a greedy strategy algorithm to generate multiple historical UAV action strategies.

[0165] According to the historical initial position and historical end position of the drone, the action space A of the drone motion direction model is further divided into a smaller action space A' (sub-running direction). The smaller action space consists of three actions selected in the target direction and two actions at the boundary of the target direction. In this way, the drone selects an action in a smaller action space, which improves training efficiency, shortens training cycle, and prevents falling into the local optimal problem.

[0166] In an optional solution, the device also includes an input unit, a selection unit, a first determination unit, a second determination unit and an optimization unit; the input unit is used to control the movement of the drone by adopting a historical drone action strategy and obtain historical real-time operation data during the movement, and after determining the historical real-time operation data as sample data, build a priority flight experience storage pool, and input the sample data into the priority flight experience storage pool; the selection unit is used to select a preset number of experience sample data from the sample data of the priority flight experience storage pool in order from high to low priority, and input them into the initial dual strategy learning network model to generate an expected Q value of the evaluation network and a target Q value of the target network, wherein the initial dual strategy learning network model includes the evaluation network and the target network; the first determination unit is used to adopt the third formula according to the target Q value and the expected Q value: , determine the TD target error value, where, represents the TD target error value, represents the parameters of the target network, is the target Q value, Q is the expected Q value, represents the parameters of the evaluation network, represents the discount factor, which is used to indicate the importance that the drone places on future rewards. A is the direction of motion of the drone. Subscript j represents the jth empirical sample data. Indicates the reward obtained by the drone. The feature vector representing the drone position S is used to approximate the state value, The function represents finding an action A that maximizes the target Q value of the evaluation network Q; the fourth formula is used based on the TD target error value and the expected Q value of the evaluation network: , determine the loss function of the initial dual strategy learning model, where, is the loss function, represents the priority weight of the jth experience sample data, is the position of the UAV; according to the loss function, the parameters of the target network and the evaluation network of the dual strategy learning network are optimized to obtain the dual strategy learning model.

[0167] Among them, the priority flight experience storage pool gives priority to experience samples through the priority function to improve learning efficiency. Unlike the traditional experience playback pool, the priority flight experience storage pool of this scheme does not sample experience samples evenly, but gives priority to samples with higher priority. This allows the learning algorithm to pay more attention to those experiences that are more important for learning, thereby improving learning efficiency.

[0168] In an optional solution, the device further includes a computing unit, which is used to use the seventh formula after optimizing the parameters of the target network and the evaluation network of the dual strategy learning network according to the loss function: , recalculate the priority of the sample data of the priority flight experience storage pool, where, Indicates the priority of sample data, represents the TD target error value, is the expected Q value of sample data j, represents the exploration rate.

[0169] The priority function uses the TD error of the experience sample as the input of the priority function to assign a priority. An experience sample with a large TD error indicates that the UAV agent has not learned the experience sample well. The priority function will assign a relatively large value to this experience sample, making it easier for the dual strategy learning network to sample it.

[0170] The unmanned aerial vehicle path planning device for the steel bar binding site includes a processor and a memory. The construction unit, the first acquisition unit, the second acquisition unit, the prediction unit, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions. The above modules are all located in the same processor; or, the above modules are located in different processors in any combination.

[0171] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the path length planned by the existing drone path planning method is not the shortest and optimal, and the planning time often requires a long wait.

[0172] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0173] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the drone path planning method for the steel bar binding site.

[0174] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the drone path planning method for the steel bar binding site when it is running.

[0175] The embodiment of the present invention provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the method for planning the path of a drone at a steel bar binding site when executing the program. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0176] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a drone path planning method for initializing the steel bar binding site.

[0177] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0178] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0179] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0182] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0183] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0184] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0185] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0186] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for UAV path planning at a steel bar binding site, characterized in that: include: Constructing a drone path planning environment for a steel bar binding site with Markov properties, wherein the drone path planning environment includes: an obstacle assessment model, a drone position model, a drone motion direction model, and a reward function model; Obtain the historical initial position and the historical end position within the historical time period, and input the historical initial position, the historical end position and the drone path planning environment model into the action selection strategy model to calculate and obtain the historical drone action strategy, and use the historical drone action strategy to control the movement of the drone and obtain historical real-time operation data during the movement, and determine the historical real-time operation data as sample data, the sample data includes: the current moment position information of the drone, the current moment direction information of the drone, the next moment predicted position information of the drone, the current moment reward data of the drone, and the priority of the sample data; Acquire a dual strategy learning network model, wherein the dual strategy learning network model is trained using the sample data; The initial position and the final position of the UAV are input into the dual strategy learning network model for prediction, so as to obtain the flight path of the UAV at the steel bar binding site.

2. The method according to claim 1, characterized in that Construct a UAV path planning environment model for the steel bar binding site with Markov properties, including: Determining the obstacle assessment model according to the distance between the UAV and the obstacle, the maximum radius setting value of the danger zone, and the maximum radius setting value of the obstacle in the danger zone; Constructing the drone position model and the drone movement direction model, wherein the drone position model is used to determine the position of the drone in a two-dimensional space, and the drone movement direction model is used to divide the area of ​​the steel bar binding site and determine the movement direction of the drone; Constructing a reward function model, and using the reward function model to reward the operating state of the drone; The drone path planning environment for the steel bar binding site with the Markov property is constructed according to the obstacle assessment model, the drone position model, the drone motion direction model and the reward function model.

3. The method according to claim 2, characterized in that The obstacle assessment model is determined according to the distance between the UAV and the obstacle, the maximum radius setting value of the danger zone, and the maximum radius setting value of the obstacle in the danger zone, including: According to the first formula: , determine the probability of the UAV being crashed, where, is the probability of the UAV being crashed, D is the distance between the UAV and the obstacle, Indicates the maximum radius setting value of the danger zone. is the maximum radius setting value of the obstacle in the danger zone, where: ; According to the second formula: Assess the overall risk of damage to the drone, where: is the overall risk of damage to the drone, k represents the number of dangerous areas, is the probability that the UAV is destroyed when it passes through the i-th danger zone; The obstacle assessment model is determined according to the first formula and the second formula.

4. The method according to claim 2, characterized in that: Constructing a reward function model, and using the reward function model to reward the operating state of the drone, including: Using the fifth formula: Construct the reward function model, where R is the reward function, The penalty item for the drone flying into the danger zone is value, is the reward item for the drone to fly to the target location, the reward value is A, is the penalty item for the drone flying out of the steel bar binding site, penalty B value, is the penalty item for the UAV's flight time exceeding the flight time threshold, the penalty C value, is the penalty item of the drone in other set states, penalty D value; According to the sixth formula Sure Value, where is the overall risk of damage to the drone, is the risk threshold.

5. The method according to claim 2, characterized in that: The historical initial position, the historical end position and the UAV path planning environment model are input into the action selection strategy model to calculate and obtain the historical UAV action strategy, including: According to the historical initial position and the historical end position, a heuristic search algorithm is used to divide the UAV motion direction model to obtain a plurality of sub-movement directions; A greedy strategy algorithm is adopted to control the UAV action according to the multiple sub-operation directions, and generate multiple historical UAV action strategies.

6. The method according to claim 1, characterized in that After the historical UAV action strategy is adopted to control the movement of the UAV and the historical real-time operation data is acquired during the movement, and the historical real-time operation data is determined as sample data, the method further includes: Constructing a priority flight experience storage pool, and inputting the sample data into the priority flight experience storage pool; Selecting a preset number of experience sample data from the sample data of the priority flight experience storage pool in order of priority from high to low, and inputting them into the initial dual strategy learning network model, generating an expected Q value of the evaluation network and a target Q value of the target network, wherein the initial dual strategy learning network model includes the evaluation network and the target network; According to the target Q value and the expected Q value, the third formula is adopted: , determine the TD target error value, where, represents the TD target error value, represents the parameters of the target network, is the target Q value, Q is the expected Q value, represents the parameters of the evaluation network, represents the discount factor, which is used to indicate the importance of the UAV to future rewards. A is the movement direction of the UAV. Subscript j represents the jth experience sample data. represents the reward obtained by the drone, The feature vector representing the drone position S is used to approximate the state value, Function represents finding an action Maximize the target Q value of the evaluation network Q; The fourth formula is adopted according to the TD target error value and the expected Q value of the evaluation network: , determine the loss function of the initial dual strategy learning network model, where, is the loss function, represents the priority weight of the jth empirical sample data, is the position of the UAV; and optimizing the parameters of the target network and the evaluation network of the dual strategy learning network according to the loss function to obtain the dual strategy learning network model.

7. The method according to claim 6, characterized in that After optimizing the parameters of the target network and the evaluation network of the dual strategy learning network according to the loss function, the method further includes: Using the seventh formula: , recalculate the priority of the sample data in the priority flight experience storage pool, where, Indicates the priority of the sample data, represents the TD target error value, is the expected Q value of the sample data j, represents the exploration rate.

8. A drone path planning device for a steel bar binding site, characterized in that: include: A construction unit is used to construct a UAV path planning environment for a steel bar binding site with Markov properties, wherein the UAV path planning environment includes: an obstacle assessment model, a UAV position model, a UAV motion direction model, and a reward function model; A first acquisition unit is used to acquire a historical initial position and a historical end position within a historical time period, and input the historical initial position, the historical end position and the drone path planning environment model into an action selection strategy model to calculate and obtain a historical drone action strategy, and use the historical drone action strategy to control the movement of the drone and acquire historical real-time operation data during the movement, and determine the historical real-time operation data as sample data, wherein the sample data includes: the current moment position information of the drone, the current moment direction information of the drone, the next moment predicted position information of the drone, the current moment reward data of the drone, and the priority of the sample data; A second acquisition unit is used to acquire a dual strategy learning network model, wherein the dual strategy learning network model is trained by using the sample data; The prediction unit is used to input the initial position and the terminal position of the UAV into the dual strategy learning network model for prediction, so as to obtain the flight path of the UAV at the steel bar binding site.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the drone path planning method for the steel bar binding site as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing a drone path planning method for a steel bar binding site as described in any one of claims 1 to 7.

Images