UAV inspection scheduling method and related equipment based on improved two-layer reinforcement learning

Abstract

The present application belongs to the technical field of drone scheduling, and discloses a drone inspection scheduling method and related equipment based on improved double-layer reinforcement learning. On the basis of gridding the inspected city, a layer of reinforcement learning algorithm is used to plan candidate paths, and then another layer of reinforcement learning algorithm is used to select a preferred path from multiple candidate paths and assign it to the drone; each drone does not need to be paired with each machine nest one by one, and each drone does not need to patrol in a fixed area, which is beneficial to reduce the number of machine nests and drones, improve the utilization rate of machine nests and drones, and reduce the waste of machine nests and drone resources; in addition, the drone does not need to return to a fixed machine nest for charging during the patrol process, which saves time, improves the efficiency of urban inspection, and reduces the waste of drone resources caused by the inability to effectively utilize the drone due to returning for charging. Even if some drones cannot perform inspection tasks due to emergencies, other drones can be mobilized to complete the inspection tasks, thereby improving resource utilization.

Description

Technical Field

[0001] The present application relates to the technical field of drone scheduling, and more specifically, to a drone inspection scheduling method and related equipment based on improved double-layer reinforcement learning. Background Art

[0002] The integrated air-space-ground social governance work requires the use of drone swarms to patrol the entire city and promptly discover various urban illegal, irregular, and unauthorized construction activities.

[0003] In traditional drone inspection systems, drones and nests are paired one by one. Drones only patrol in fixed areas near corresponding nests, and can only return to corresponding nests for charging when they need to be charged. In order to complete the patrol and aerial photography tasks within the specified period, more resources need to be invested in building nests and purchasing drones, resulting in serious waste. In addition, due to the limited battery life of drones, they need to be charged multiple times. Patrolling in fixed areas causes drones to fly back and forth, wasting a lot of time and battery life (generally, the battery life for returning to charge cannot be effectively used for inspection work, thus wasting battery life). In addition, the drone resources that can be mobilized in the event of an emergency will also be limited, resulting in low resource utilization.

[0004] Therefore, it is necessary to find a UAV inspection scheduling method to reduce the waste of machine nests and UAV resources and improve urban inspection efficiency and resource utilization. Summary of the invention

[0005] The purpose of this application is to provide a drone inspection scheduling method and related equipment based on improved two-layer reinforcement learning, which can reduce the waste of machine nests and drone resources and improve urban inspection efficiency and resource utilization.

[0006] In the first aspect, the present application provides a drone inspection scheduling method based on improved double-layer reinforcement learning, which is applied to a machine nest of a drone inspection system, wherein the drone inspection system includes multiple drones and multiple machine nests, and the machine nests are communicatively connected; the drone inspection scheduling method based on improved double-layer reinforcement learning includes the steps of:

[0007] A1. Divide the inspected cities into grids;

[0008] A2. When a drone reaches the local nest, the drone that has reached the local nest is taken as the target drone, and the current inspection task completion information and the current time occupancy information of each nest are updated. According to the grid division result and the cruising range of the target drone, the reachable nest that the target drone can reach next is determined to obtain the reachable nest set;

[0009] A3. According to the current inspection task completion information and the cruising distance, the candidate path generation network model based on the deep reinforcement learning algorithm is used to generate the optimal path from the local nest to each of the reachable nests, recorded as the candidate path, and the candidate path set is obtained;

[0010] A4. According to the current inspection task completion information, the current time occupancy information of each machine nest and the reachable machine nest set, a candidate path selection network model based on a deep reinforcement learning algorithm is used to select a candidate path from the candidate path set as the target path;

[0011] A5. Perform conflict judgment based on the current time occupancy information of each machine nest and the target path, and assign the target path to the target UAV or re-plan the target path and assign it to the target UAV based on the conflict judgment result.

[0012] By adopting the above method to conduct drone inspection and dispatch, each drone does not need to be paired with each nest one by one, and each drone does not need to patrol in a fixed area, which is conducive to reducing the number of nests and drones, improving the utilization rate of nests and drones, and reducing the waste of nests and drone resources. In addition, the drone does not need to return to the fixed nest for charging during the patrol process, which saves time, improves the efficiency of urban inspection, and reduces the waste of drone resources caused by the inability to effectively utilize drones due to returning for charging. Even if some drones cannot perform inspection tasks due to emergencies, other drones can be mobilized to complete the inspection tasks, thereby improving resource utilization.

[0013] Preferably, step A1 comprises:

[0014] Obtain the minimum circumscribed rectangle of the boundary line of the inspected city;

[0015] Performing grid division on the minimum circumscribed rectangle as a grid division result of the inspected city;

[0016] The inspection task completion information includes grid status information of each grid area; the grid status information includes first status information indicating that the grid area is a no-fly zone or an out-of-city area, second status information indicating that the grid area is temporarily unsuitable for flying, third status information indicating that the grid area has completed the inspection, and fourth status information indicating that the grid area has not completed the inspection.

[0017] The above-mentioned method of dividing the grid and setting the grid status information of each grid area can help guide the UAV to avoid the no-fly zone and the area temporarily unsuitable for flying, making the scheduling result more reasonable and safe.

[0018] Preferably, in step A2, determining the reachable nest that the target UAV can reach next according to the grid division result and the cruising range of the target UAV includes:

[0019] Other machine nests that meet the following conditions are used as candidate machine nests: in a grid matrix with the grid area where the other machine nests are located and the grid area where the local machine nest is located as diagonal grids, there is no grid area with grid state information of the first state information and no grid area with grid state information of the second state information;

[0020] According to the Chebyshev distance calculation formula, the minimum grid distance from the host to each of the candidate host is calculated respectively, and converted into a corresponding flight distance, which is recorded as the minimum flight distance;

[0021] The candidate aircraft nest whose minimum flight distance is not greater than the endurance distance is used as the reachable aircraft nest.

[0022] Preferably, step A3 includes taking each of the reachable machine nests as the target machine nest in turn and executing:

[0023] A301. Obtaining the action space of the current time step; the action space of the current time step includes actions that are allowed to be executed, and the actions that are allowed to be executed include moving to each adjacent grid area whose grid state information is neither the first state information nor the second state information;

[0024] A302. Obtain the first reference state information of the current time step; the first reference state information of the current time step includes the inspection task completion information of the current time step, the endurance distance and the minimum grid distance from the position of the current time step to the target machine nest; in the first time step, the inspection task completion information of the current time step is the current inspection task completion information, and the position of the current time step is the position of the machine nest;

[0025] A303. Calculate the first value function of each of the actions using the candidate path generation network model according to the first reference state information and the action space;

[0026] A304. According to the first value function, the Softmax strategy and the ε-greedy strategy are combined to select an action, and the selected action is executed;

[0027] A305. If the grid area where the target machine nest is located is reached after executing the selected action, the path planning is terminated, and the path composed of the grid areas pointed to by the actions selected at each time step is used as the candidate path from the local machine nest to the target machine nest. Otherwise, enter the next time step and return to execute step A301.

[0028] Generating candidate paths based on the candidate path generation network model can improve the efficiency of path generation and the rationality of candidate paths. In the process of generating candidate paths, combining the Softmax strategy and the ε-greedy strategy for action selection is beneficial to avoid falling into the local optimal solution and improve planning efficiency.

[0029] Preferably, step A303 includes taking each of the actions as the target action in sequence and performing:

[0030] Inputting the first reference state information and the target action into the candidate path generation network model to obtain a first value function prediction value output by the candidate path generation network model;

[0031] If the remaining cruising distance of the target UAV after executing the target action is not less than the flight distance corresponding to the minimum grid distance from the position after executing the target action to the target nest, the predicted value of the first value function is used as the first value function of the target action;

[0032] If the remaining cruising distance of the target UAV after executing the target action is less than the flight distance corresponding to the minimum grid distance from the position after executing the target action to the target machine nest, the first value function of the target action is set to 0.

[0033] Preferably, step A304 includes:

[0034] Generate a random number b in the range [0,1];

[0035] If b>ε, then randomly select an action from the action space and execute the selected action; ε is the reference probability of the ε-greedy strategy, and ε gradually increases with the increase of time steps;

[0036] If b≤ε, then according to the first value function, an action is selected from the action space using the Softmax strategy, and the selected action is executed.

[0037] Preferably, step A4 comprises:

[0038] A401. The current inspection task completion information, the current time occupancy information of each machine nest and the reachable machine nest set are used as the second reference state information, and the second value function of each candidate path is calculated using the candidate path selection network model according to the second reference state information and the candidate path set;

[0039] A402. According to the second value function, a candidate path is selected from the candidate path set as the target path based on the Softmax strategy.

[0040] Preferably, step A5 comprises:

[0041] A501. Extract the current time occupancy information of the terminal nest of the target path from the current time occupancy information of each nest; the terminal nest is the reachable nest at the end of the target path;

[0042] A502. Calculate the target drone's use period of the terminal machine nest according to the target path, recorded as the first period;

[0043] A503. Compare the first period of time with the terminal machine nest current time occupancy information to determine whether the first period of time is available;

[0044] A504. If the first time period is not available, after removing the target path from the candidate path set, return to step A4;

[0045] A505. If the first time period is available and does not conflict with other machine nests requesting to use the terminal machine nest, the target path is assigned to the target drone;

[0046] A506. If the first time period is available and conflicts with the time period required by other machine nests to use the destination machine nest, the bid value of the target UAV is calculated based on the target path for bidding, and the target path is allocated to the target UAV after the bidding is successful. If the bidding fails, return to execute step A2.

[0047] In a second aspect, the present application provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the drone inspection scheduling method based on improved two-layer reinforcement learning as described in the preceding item.

[0048] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, runs the steps in the drone inspection scheduling method based on improved two-layer reinforcement learning as described above.

[0049] Beneficial effects: The drone inspection scheduling method and related equipment based on improved double-layer reinforcement learning provided by the present application do not need to be paired with each nest one by one, and each drone does not need to patrol in a fixed area, which is beneficial to reducing the number of nests and drones, improving the utilization rate of nests and drones, and reducing the waste of nests and drone resources; in addition, the drone does not need to return to a fixed nest for charging during the patrol process, which saves time, improves the efficiency of urban inspections, and reduces the waste of drone resources caused by the inability to effectively utilize drones due to returning for charging. Even if some drones are unable to perform inspection tasks due to emergencies, other drones can be mobilized to complete the inspection tasks, thereby improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a drone inspection scheduling method based on improved double-layer reinforcement learning provided in an embodiment of the present application.

[0051] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0052] Figure 3 This is the schematic diagram for screening candidate machine nests.

[0053] Figure 4 A schematic grid area distribution diagram is shown in FIG.

[0054] Description of reference numerals: 301, processor; 302, memory; 303, communication bus. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0056] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0057] Please refer to Figure 1 , Figure 1 It is a drone inspection scheduling method based on improved double-layer reinforcement learning in some embodiments of the present application, which is applied to the machine nest of the drone inspection system. The drone inspection system includes multiple drones and multiple machine nests (these machine nests are distributed in different locations of the inspected city), and the machine nests are connected by communication;

[0058] The UAV inspection scheduling method based on improved double-layer reinforcement learning includes the following steps:

[0059] A1. Divide the inspected cities into grids;

[0060] A2. When a drone reaches the local nest, the drone that reaches the local nest is taken as the target drone, and the current inspection task completion information and the current time occupancy information of each nest are updated. According to the grid division result and the endurance distance of the target drone, the reachable nest that the target drone can reach next is determined to obtain the reachable nest set;

[0061] A3. Based on the current inspection task completion information and cruising distance, the candidate path generation network model based on the deep reinforcement learning algorithm is used to generate the optimal paths from the local nest to each reachable nest, recorded as candidate paths, and obtain a candidate path set;

[0062] A4. According to the current inspection task completion information, the current time occupancy information of each machine nest and the reachable machine nest set, a candidate path is selected from the candidate path set as the target path using the candidate path selection network model based on the deep reinforcement learning algorithm;

[0063] A5. Perform conflict judgment based on the current time occupancy information of each nest and the target path, and assign the target path to the target UAV or re-plan the target path and assign it to the target UAV based on the conflict judgment result.

[0064] By adopting the above method to conduct drone inspection and dispatch, each drone does not need to be paired with each nest one by one, and each drone does not need to patrol in a fixed area, which is conducive to reducing the number of nests and drones, improving the utilization rate of nests and drones, and reducing the waste of nests and drone resources. In addition, the drone does not need to return to the fixed nest for charging during the patrol process, which saves time, improves the efficiency of urban inspection, and reduces the waste of drone resources caused by the inability to effectively utilize drones due to returning for charging. Even if some drones cannot perform inspection tasks due to emergencies, other drones can be mobilized to complete the inspection tasks, thereby improving resource utilization.

[0065] In this embodiment, step A1 includes:

[0066] Get the minimum enclosing rectangle of the border line of the inspected city;

[0067] The minimum circumscribed rectangle is gridded to obtain the gridded result of the inspected city.

[0068] Specifically, when dividing, the minimum circumscribed rectangle is grid-divided using horizontal grid lines parallel to the horizontal sides of the minimum circumscribed rectangle and vertical grid lines parallel to the vertical sides of the minimum circumscribed rectangle, and the spacing between the horizontal grid lines is equal to the spacing between the vertical grid lines (the side length of the minimum circumscribed rectangle can be appropriately increased to ensure that all grid areas obtained by division are square areas). Among them, the spacing between the horizontal grid lines and the spacing between the vertical grid lines can be set according to actual needs. By dividing, the minimum circumscribed rectangle is divided into multiple grid areas, and the grid matrix composed of all grid areas can be called a full grid matrix, wherein each grid area contains at most one machine nest.

[0069] The center point of each grid area is the shooting point (or inspection point) of the aircraft inspection.

[0070] Among the grid areas divided in the above manner, some grid areas are generally located outside the inspected city and thus do not need to be inspected.

[0071] Among them, the mesh division results of all machine nests must be consistent.

[0072] The inspection task completion status information includes grid status information of each grid area; the grid status information includes first status information indicating that the grid area is a no-fly zone (such as an airport area, a military facility area, etc.) or an out-of-city area, second status information indicating that the grid area is temporarily unsuitable for flying (for example, temporarily unsuitable for flying due to bad weather), third status information indicating that the grid area has completed the inspection, and fourth status information indicating that the grid area has not completed the inspection.

[0073] The grid status information may be represented by symbols, letters or numerical values, for example, the first status information is represented by -2, the second status information is represented by -1, the third status information is represented by 0, and the fourth status information is represented by 1, but not limited thereto; thus, the inspection task completion information may be represented by the following task status matrix:

[0074] ;

[0075] Among them, X is the task state matrix, It is the grid state information of the grid area in the i-th row and j-th column in the minimum bounding rectangle.

[0076] The above-mentioned method of dividing the grid and setting the grid status information of each grid area can help guide the UAV to avoid the no-fly zone and the area temporarily unsuitable for flying, making the scheduling result more reasonable and safe.

[0077] Among them, the current time occupancy information of the machine nest can be recorded using a usage schedule, and a tuple data structure is used in the usage schedule to record the previous time occupancy information. For example, the usage schedule is recorded as [(10:20,10:50),(11:00,11:25)], (10:20,10:50) and (11:00,11:25) are tuple data of two occupancy time periods, among which (10:20,10:50) indicates that the first occupancy time period is from 10:20 to 10:50, and (11:00,11:25) indicates that the second occupancy time period is from 11:00 to 11:25.

[0078] As the inspection task progresses, the inspection task completion information and the time occupancy information of each nest will change. Whenever a drone arrives at a nest, the docked nest will plan and allocate the next inspection path for the docked drone, so that the docked drone will perform the inspection according to the allocated inspection path in the next step. When the docked nest starts to plan the inspection path, it needs to update the inspection task completion information and the time occupancy information of each nest to ensure that the planned path is consistent with the current task execution status and system status. In the above method, the description is made from the perspective of the docked nest, so the docked nest is called the local nest.

[0079] In fact, while this machine nest is performing path planning, other machine nests may complete path planning and allocation, resulting in changes in the current inspection task completion information and the current time occupancy information of each machine nest. After each machine nest completes path planning and allocation, it will update the current inspection task completion information and the current time occupancy information of each machine nest locally, and then broadcast the updated current inspection task completion information and the updated current time occupancy information of each machine nest. After receiving the updated broadcast information (i.e., the broadcast information containing the updated current inspection task completion information and the updated current time occupancy information of each machine nest), other machine nests will update the current inspection task completion information and the current time occupancy information of each machine nest locally in real time.

[0080] Among them, the cruising range of the target UAV is a known quantity, and the cruising range of the target UAV can adopt the nominal cruising range of the target UAV or the maximum cruising range measured in advance.

[0081] Furthermore, in step A2, according to the grid division result and the cruising range of the target UAV, the reachable nest that the target UAV can reach next is determined, including:

[0082] Other machine nests that meet the following conditions are used as candidate machine nests: in the grid matrix with the grid area where the other machine nests are located and the grid area where the local machine nest is located as diagonal grids, there is no grid area with grid state information of the first state information and no grid area with grid state information of the second state information (for example, Figure 3 In the figure, it is assumed that the local nest J1 is in grid area a, and the other nest J2 is in grid area b. The grid matrix with grid area a and grid area b as diagonal grids is the area in the thick solid frame in the figure. It is assumed that there is no grid area with grid state information of the first state information and no grid area with grid state information of the second state information in the area in the thick solid frame, then the nest J2 is taken as one of the candidate nests; thereby ensuring that the target aircraft can fly freely in the thick solid frame. When performing candidate path planning later, there is no need to consider the influence of the grid area that cannot fly, thereby simplifying the planning process and improving planning efficiency);

[0083] According to the Chebyshev distance calculation formula, the minimum grid distance from the aircraft nest to each candidate aircraft nest is calculated respectively, and converted into the corresponding flight distance, which is recorded as the minimum flight distance;

[0084] The candidate nest with the minimum flight distance not greater than the endurance distance is regarded as the reachable nest.

[0085] Among them, according to the Chebyshev distance calculation formula, the minimum grid distance from the host nest to each candidate host nest is calculated, specifically including:

[0086] The minimum grid distance from the host nest to the candidate host nest is calculated by the following formula:

[0087] ;

[0088] in, is the minimum grid distance from the local nest to the candidate nest, ii is the row number of the grid area where the local nest is located (i.e., the row number of the grid area where the local nest is located in the full grid matrix), jj is the column number of the grid area where the local nest is located (i.e., the column number of the grid area where the local nest is located in the full grid matrix), kk is the row number of the grid area where the candidate nest is located (i.e., the row number of the grid area where the candidate nest is located in the full grid matrix), and ll is the column number of the grid area where the candidate nest is located (i.e., the column number of the grid area where the candidate nest is located in the full grid matrix).

[0089] Since the drone needs to take pictures at the center point of the grid area when performing inspection flights, it mainly flies between the center points of each grid area. In this article, the path formed by connecting the center points of each grid area is called a grid path, and the total number of grid areas in the grid path except the grid area where the starting point is located is taken as the grid distance corresponding to the grid path; among them, the grid distance between two position points refers to the grid distance of the grid path between the grid areas where the two position points are located.

[0090] For example Figure 4 In the example, assume that the local nest J1 is in the grid area , candidate nest J3 is in the grid area , from the grid area To the grid area The minimum grid path is: -> -> -> -> Therefore, the grid distance of the minimum grid path is 4, and the minimum grid distance from the local nest J1 to the candidate nest J3 is 4.

[0091] Among them, when the drone flies from one grid area to the next grid area, it may fly in the horizontal, vertical or diagonal direction. When flying in the horizontal or vertical direction, the unit grid distance is actually equal to the side length of a single grid area. When flying in the diagonal direction, the unit grid distance is actually equal to the diagonal length of a single grid area. In order to simplify the calculation and ensure that the drone can reliably reach the reachable machine nest, here, the actual flight distance corresponding to the unit grid distance is unified as the diagonal length of a single grid area. Therefore, when converting the minimum grid distance from the machine nest to each candidate machine nest into the corresponding flight distance, the diagonal length of a single grid area can be multiplied by the minimum grid distance from the machine nest to each candidate machine nest.

[0092] In some possible implementations, step A3 includes taking each reachable machine nest as a target machine nest in turn and executing:

[0093] A301. Obtaining the action space of the current time step (each action of the candidate path is planned, that is, a time step is completed, so the time step represents the process of single action planning); the action space of the current time step includes actions that are allowed to be executed, and the actions that are allowed to be executed include moving to each adjacent grid area whose grid state information is neither the first state information nor the second state information;

[0094] A302. Get the first reference state information of the current time step; the first reference state information of the current time step includes the inspection task completion information of the current time step, the endurance distance and the minimum grid distance from the position of the current time step to the target machine nest; in the first time step, the inspection task completion information of the current time step is the current inspection task completion information, and the position of the current time step is the position of the machine nest;

[0095] A303. Calculate the first value function of each action using the candidate path generation network model according to the first reference state information and action space;

[0096] A304. According to the first value function, the Softmax strategy and the ε-greedy strategy are combined to select an action, and the selected action is executed;

[0097] A305. If the grid area where the target machine nest is located is reached after executing the selected action, the path planning is terminated, and the path (grid path) composed of the grid areas pointed to by the actions selected at each time step is used as the candidate path from the local machine nest to the target machine nest. Otherwise, enter the next time step and return to execute step A301.

[0098] Generating candidate paths based on the candidate path generation network model can improve the efficiency of path generation and the rationality of candidate paths. In the process of generating candidate paths, combining the Softmax strategy and the ε-greedy strategy for action selection is beneficial to avoid falling into the local optimal solution and improve planning efficiency.

[0099] Among them, in step A301, since the drone mainly moves between the center points of each grid area, after the drone reaches the center point of a grid area, the next step can only fly to the center point of an adjacent grid area. Among these adjacent grid areas, there may be grid areas where flight is not allowed (that is, grid areas whose grid state information is the first state information and grid areas whose grid state information is the second state information). Therefore, the action in the action space only includes moving to each adjacent grid area whose grid state information is neither the first state information nor the second state information.

[0100] For example, when there are 8 adjacent grid areas, and all of them are grid areas where flight is allowed (i.e., grid areas with grid state information of the third state information or grid areas with grid state information of the fourth state information), the action space can be expressed as {Nth, Sth, Wst, Est, NE, NW, SE, SW}, where action Nth means that the UAV moves from the center point of the current grid area to the north to the center point of an adjacent grid area; action Sth means that the UAV moves from the center point of the current grid area to the south to the center point of an adjacent grid area; action Wst means that the UAV moves from the center point of the current grid area to the west to the center point of the adjacent grid area. Action Est means that the drone moves from the center point of the current grid area to the east to the center point of an adjacent grid area; Action NE means that the drone moves from the center point of the current grid area to the northeast to the center point of an adjacent grid area; Action NW means that the drone moves from the center point of the current grid area to the northwest to the center point of an adjacent grid area; Action SE means that the drone moves from the center point of the current grid area to the southeast to the center point of an adjacent grid area; Action SW means that the drone moves from the center point of the current grid area to the southwest to the center point of an adjacent grid area. If the adjacent grid area in the north direction is not allowed to fly, action Nth is eliminated from the action space.

[0101] Among them, in step A302, at the first time step, the position of the current time step is the position of the local nest, and at other time steps, the position of the current time step is the position reached after the action selected at the previous time step is executed. At the first time step, the current inspection task completion information is assigned to the inspection task completion information of the current time step, and at other time steps, the grid state information of the grid area corresponding to the position of the current time step in the inspection task completion information of the current time step is set as the third state information.

[0102] Among them, the minimum grid distance from the position of the current time step to the target machine nest can be calculated according to the Chebyshev distance calculation formula. The specific calculation process is referred to the previous text and will not be repeated here.

[0103] Further, step A303 includes taking each action as the target action in sequence and executing:

[0104] Inputting the first reference state information and the target action into the candidate path generation network model to obtain a first value function prediction value output by the candidate path generation network model;

[0105] If the remaining cruising distance of the target drone after executing the target action is not less than the flight distance corresponding to the minimum grid distance from the position after executing the target action to the target drone nest, the predicted value of the first value function is used as the first value function of the target action;

[0106] If the remaining endurance distance of the target UAV after executing the target action is less than the flight distance corresponding to the minimum grid distance from the position after executing the target action to the target nest, the first value function of the target action is set to 0.

[0107] When the remaining cruising distance of the target UAV after executing the target action is less than the flight distance corresponding to the minimum grid distance from the position after executing the target action to the target nest, it means that executing the target action will make the target UAV unable to reach the target nest. Therefore, the target action should not be selected. Therefore, setting the first value function of the target action to 0 will result in the probability of selecting the target action in the future to be 0. Therefore, the target UAV can be effectively guided to gradually fly to the target nest, ensuring that the end point of the candidate path is the target nest.

[0108] Preferably, step A304 includes:

[0109] Generate a random number b in the range [0,1];

[0110] If b>ε, then randomly select an action from the action space and execute the selected action; ε is the reference probability of the ε-greedy strategy, and ε gradually increases with the increase of time steps (for example, in step A305, after entering the next time step and before returning to step A301, if ε+d_c<1, then let ε=ε+d_c, d_c is a preset increment, such as 0.0001, but not limited to this);

[0111] If b≤ε, then according to the first value function, an action is selected from the action space using the Softmax strategy and the selected action is executed.

[0112] Among them, according to the first value function, an action is selected from the action space using the Softmax strategy, specifically including:

[0113] The selection probability of each action in the action space is calculated using the following formula:

[0114] ;

[0115] in, is the probability of selecting the i-th action, is the first value function of the ith action, is the first value function of the jth action, n is the number of actions in the action space;

[0116] According to the selection probability of each action in the action space, an action is selected using the roulette method.

[0117] The roulette method is prior art and will not be described in detail here.

[0118] In fact, if only the Softmax strategy is used for action selection, it is easy to fall into the local optimal solution. If only the ε-greedy strategy is used for action selection, the efficiency of path planning is low. Here, at the beginning of path planning, an initial value of ε is given (set according to actual needs). With the continuous exploration of the environment, ε gradually increases (the maximum is close to 1, but not 1). In the early stage of path planning, there is a high probability of random selection through the ε-greedy strategy, so as to avoid falling into the local optimal solution. In the later stage of path planning, there is a high probability of action selection through the Softmax strategy, so as to improve the convergence speed of the algorithm. Therefore, it can avoid falling into the local optimal solution and improve the efficiency of path planning.

[0119] Among them, the candidate path generation network model can be pre-trained in the following ways:

[0120] B1. Initialize the network parameters of the first target network and the first prediction network (preferably make the network parameters obey the Gaussian distribution with an expectation of 0 and a standard deviation of 0.2, wherein the network structures of the first target network and the first prediction network are the same), initialize the reference probability ε, initialize the number of updates Up to 0, initialize the number of iterations Dn to 0, and retrieve the first experience database;

[0121] B2. Randomly extract B (B is a preset positive integer, which can be set according to actual needs) first sample data from the first experience database to form a first sample data set, where the first sample data is recorded as , is the first reference state information before executing the action, For the selected action, For Execute actions under conditions The reward function (where if the action is performed After reaching the uninspected grid area or the target machine nest, is 1, otherwise, is 0, this reward function can guide the drone to fly to the grid area and target nest where the inspection is not completed). To perform actions The first reference state information updated later;

[0122] B3. and Input the first target network, obtain the first value function prediction value output by the first target network, record it as the first initial target value, and calculate the first target value according to the following method:

[0123] like The position of the current time step in is in the grid area where the target machine nest is located, so , otherwise, let ,in, is the first target value, is the discount coefficient (can be set according to actual needs), is the first initial target value;

[0124] B4. and Input the first prediction network to obtain the first value function prediction value output by the first prediction network, which is recorded as the first prediction value;

[0125] B5. Calculate the first loss function according to the following formula:

[0126] ;

[0127] in, is the first loss function, is the first predicted value, To find the average function;

[0128] B6. According to the first loss function, the network parameters of the first prediction network are updated using the gradient descent method, and Up=Up+1, Dn=Dn+1;

[0129] B7. If Up reaches the preset update number threshold, the updated network parameters of the first prediction network are copied to the first target network to update the network parameters of the first target network (at this time, the network parameters of the first prediction network and the first target network are the same), and Up=0, ε=ε+d_c (it should be noted that ε gradually increases during the training process, and ε will be reinitialized when generating candidate paths in step A3. Therefore, the ε value during training will not affect the ε value of the candidate path generation process in step A3); if Up does not reach the preset update number threshold, ε=ε+d_c;

[0130] B8. If Dn reaches the preset maximum number of iterations, stop training and use the updated first target network as the trained candidate path generation network model; otherwise, return to execute step B2.

[0131] In fact, in step A3, during the process of generating candidate paths, the first sample data can be collected in real time. Add the first experience database (specifically, in step A3, record the first reference state information of the current time step in a time step to obtain , record the selected action to get , record the first reference state information after executing the selected action to obtain , and execute actions according to After that, whether it reaches the grid area that has not been inspected or reaches the target machine nest, obtain ) to expand the first experience database, and regularly perform enhanced training on the candidate path generation network model (the training process refers to the above steps B1 to B8) to gradually optimize the candidate path generation network model.

[0132] Specifically, step A4 includes:

[0133] A401. The current inspection task completion information, the current time occupancy information of each machine nest and the reachable machine nest set are used as the second reference state information, and the second value function of each candidate path is calculated using the candidate path selection network model according to the second reference state information and the candidate path set;

[0134] A402. According to the second value function, a candidate path is selected from the candidate path set as the target path based on the Softmax strategy.

[0135] Step A401 includes the following steps, which are performed sequentially for each candidate path in the candidate path set:

[0136] The second reference state information and the candidate path are input into the candidate path selection network model to obtain a second value function output by the candidate path selection network model.

[0137] Wherein, step A402 includes:

[0138] The selection probability of each candidate path in the candidate path set is calculated using the following formula:

[0139] ;

[0140] in, is the selection probability of the i-th candidate path, is the second value function of the i-th candidate path, is the second value function of the jth candidate path, and m is the number of candidate paths in the path selection set;

[0141] According to the selection probability of each candidate path in the candidate path set, a candidate path is selected as the target path using the roulette wheel method.

[0142] By selecting the target path in the above manner, the optimal path can be quickly selected.

[0143] Among them, the candidate path selection network model can be pre-trained in the following ways:

[0144] C1. Initialize the network parameters of the second target network and the second prediction network (preferably make the network parameters obey the Gaussian distribution with an expectation of 0 and a standard deviation of 0.4, wherein the network structures of the second target network and the second prediction network are the same), initialize the number of updates Up to 0, initialize the number of iterations Dn to 0, and retrieve the second experience database;

[0145] C2. Randomly extract C (C is a preset positive integer, which can be set according to actual needs) second sample data from the second experience database to form a second sample data set, where the second sample data is recorded as , The second reference state information before executing the selected path, is the selected path, For Select path under conditions The reward function of , The average inspection time for a single grid area obtained by the most recent drone dispatch in each machine nest before the second sample data collection time can be calculated based on the actual time after the drone completes the corresponding task; For path The average inspection time of a single grid area can be calculated based on the path. The reward function can be used to guide the UAV to fly along a path with a smaller average inspection time of a single grid area). For path The second reference status information updated after the inspection task;

[0146] C3. The second sample data and Input the second target network, obtain the second value function output by the second target network, record it as the second initial target value, and calculate the second target value according to the following method:

[0147] like The current inspection task completion status information in the table indicates that all inspection tasks have been completed. , otherwise, let ,in, is the second target value, is the second initial target value;

[0148] C4. The second sample data and Input the second prediction network to obtain a second value function output by the second prediction network, recorded as a second prediction value;

[0149] C5. Calculate the second loss function according to the following formula:

[0150] ;

[0151] in, is the second loss function, is the second predicted value;

[0152] C6. According to the second loss function, the network parameters of the second prediction network are updated using the gradient descent method, and Up=Up+1, Dn=Dn+1;

[0153] C7. If Up reaches the preset update number threshold, the updated network parameters of the second prediction network are copied to the second target network to update the network parameters of the second target network (at this time, the network parameters of the second prediction network and the second target network are the same), and Up=0;

[0154] C8. If Dn reaches the preset maximum number of iterations, stop training and use the updated second target network as the trained candidate path selection network model; otherwise, return to step C2.

[0155] In fact, in step A4, during the process of selecting the target path, the second sample data can be collected in real time. Add the second experience database (specifically, in step A4, record the second reference state information obtained in step A401 to obtain , record the target path selected in step A402 to obtain , the execution path The second reference status information updated after the corresponding inspection task is used as , and based on the most recent drone dispatch in each nest, the average inspection time and path for completing a single grid area are obtained. The average inspection time of a single grid area is calculated as ) to expand the second experience database, and regularly strengthen the candidate path selection network model (the training process refers to the above steps C1 to C8) to gradually optimize the candidate path selection network model.

[0156] In actual applications, the target path selected by this machine nest may conflict with the paths currently planned by other machine nests, or may conflict with the paths currently performed by other UAVs. The conflict mainly manifests itself in that the destination machine nest to be reached is the same and the use period of the destination machine nest at least partially overlaps. At this time, it is necessary to bid with other machine nests or re-plan the target path.

[0157] Thus, step A5 includes:

[0158] A501. Extract the current time occupancy information of the target path from the current time occupancy information of each machine nest; the terminal machine nest is the reachable machine nest at the end of the target path;

[0159] A502. Calculate the target UAV's use period of the terminal nest according to the target path, and record it as the first period;

[0160] A503. Compare the first period with the terminal machine nest current time occupancy information to determine whether the first period is available;

[0161] A504. If the first time period is not available, remove the target path from the candidate path set and return to step A4;

[0162] A505. If the first time period is available and does not conflict with other machine nests requesting to use the terminal machine nest, the target path is assigned to the target drone;

[0163] A506. If the first time period is available and conflicts with the time period required by other machine nests to use the destination machine nest, the bid value of the target UAV is calculated based on the target path for bidding, and the target path is allocated to the target UAV after the bidding is successful. If the bidding fails, return to execute step A2.

[0164] Among them, in step A502, the first time period is determined by the arrival time of the target drone at the destination nest, the transmission time required for data transmission, and the battery replacement time required for battery replacement, wherein the arrival time can be calculated based on the flight distance of the target path, the flight speed of the target drone, and the acceleration and deceleration data of the turn, and the transmission time required for data transmission can be estimated based on the number of tasks corresponding to the target path (that is, the number of grid areas that have not completed the inspection in each grid area passed by the target path) (for example, multiplying the number of tasks by the average transmission time of single-task data, which can be obtained through data statistics), and the battery replacement time required for battery replacement is a known fixed value. Therefore, the starting time of the first time period is the arrival time, and the time interval between the end time and the starting time of the first time period is the sum of the transmission time and the battery replacement time.

[0165] Among them, in step A503, if the first time period at least partially overlaps with at least one occupied time period in the current time occupancy information of the terminal machine nest, the first time period is determined to be unavailable, otherwise, the first time period is determined to be available. Since the occupied time period is the estimated time period of use of the assigned path (including the path that is currently executing the task and the path that has not yet started to execute the task after assignment) for the terminal machine nest, in order to avoid interference with the UAV that is executing the task, the occupied time period is no longer adjusted. Therefore, for the target UAV, the first time period that at least partially overlaps with the occupied time period is unavailable.

[0166] In step A504, when the first time period is not available, other candidate paths are selected as the target path.

[0167] Wherein, step A505 includes:

[0168] According to the selected target path, broadcast the use request information to other machine nests to request to use the first time period of the terminal machine nest;

[0169] Receive the request usage information broadcast by other machine nests and extract the required time period (i.e., the first time period corresponding to the target path selected by other machine nests) of the other machine nests requesting to use the destination machine nest;

[0170] If the first time period does not overlap with any required time periods at all, the target path is assigned to the target UAV.

[0171] Among them, in step A506, if the first time period at least partially overlaps with at least one required time period, the competitive value of the target UAV is calculated according to the target path to bid with the conflicting machine nest; the conflicting machine nest is other machine nests corresponding to the required time period that at least partially overlaps with the first time period.

[0172] Specifically, the competitive value of the target drone can be calculated by the following formula:

[0173] ;

[0174] ;

[0175] in, The competitive value of the target drone, is the reward function corresponding to the target path, is the second value function corresponding to the target path, is the average time required to complete the inspection of a single grid area obtained by the most recent drone dispatch in each nest before the current moment, is the average inspection time of a single grid area corresponding to the target path, , is the weight coefficient, and Preferably, The initial value is greater than The initial value of Gradually decreases over time to less than , Gradually increases over time to greater than ; Since the reward function actually represents the quality of the current average inspection time (i.e., the average inspection time of a single grid area), the second value function actually represents the quality of the expected average inspection time of the system as a whole. In the early stage of the algorithm, in order to enhance the exploratory nature of the algorithm, the current average inspection time is mainly used (by setting Greater than To achieve this), in the later stage of the algorithm, in order to speed up the convergence of the algorithm, the competition is mainly based on the overall expected average single time (by setting Less than to achieve this).

[0176] When bidding with conflicting machine nests, the target UAV's bidding value is compared with the bidding values ​​of each conflicting machine nest. If the target UAV's bidding value is the largest, the target UAV is judged to have successfully bid. Otherwise, the target UAV is judged to have failed in bidding.

[0177] Furthermore, after step A5, the method further comprises the following steps:

[0178] A6. After the target path is assigned to the target UAV, the grid state information of the grid area through which the target path passes is updated to the third state information (to avoid the corresponding grid area being regarded as an uninspected area when each nest performs subsequent path planning, thereby causing repeated inspections of the same grid area), and the current inspection task completion information is updated; and the first time period information is added to the time occupancy information of the terminal nest (for example, the tuple data corresponding to the first time period is added to the usage time table of the terminal nest), and the current time occupancy information of each nest is updated;

[0179] A7. According to the updated current inspection task completion information and the updated current time occupancy information of each machine nest, broadcast the updated broadcast information to other machine nests so that other machine nests can update the current inspection task completion information and the current time occupancy information of each machine nest.

[0180] Therefore, further, in the present application, after step A2, there is also a step of: when receiving the update broadcast information broadcasted by other machine nests, updating the current inspection task completion information and the current time occupancy information of each machine nest in real time according to the update broadcast information. In some preferred embodiments, if the execution returns to step A2 due to bidding failure, when receiving the update broadcast information broadcasted by the conflicting machine nest, the execution returns to step A2, thereby ensuring that the latest inspection task completion information and time occupancy information are obtained in step A2.

[0181] From the above, it can be seen that the UAV inspection scheduling method based on improved double-layer reinforcement learning divides the inspected city into grids; when a UAV reaches the local nest, the UAV that reaches the local nest is taken as the target UAV, and the current inspection task completion information and the current time occupancy information of each nest are updated. According to the grid division result and the endurance distance of the target UAV, the reachable nest that the target UAV can reach next is determined to obtain the reachable nest set; according to the current inspection task completion information and endurance distance, the candidate path generation network model based on the deep reinforcement learning algorithm is used to generate the network from the local nest to each reachable nest. The optimal path of the machine nest is recorded as the candidate path, and a candidate path set is obtained; according to the current inspection task completion information, the current time occupancy information of each machine nest and the reachable machine nest set, a candidate path is selected from the candidate path set as the target path using a candidate path selection network model based on a deep reinforcement learning algorithm; according to the current time occupancy information of each machine nest and the target path, a conflict judgment is made, and according to the conflict judgment result, the target path is assigned to the target drone or the target path is replanned and then assigned to the target drone; thereby, the waste of machine nest and drone resources can be reduced, and the efficiency of urban inspection and resource utilization can be improved. Specifically, it has the following advantages:

[0182] 1. A two-layer deep reinforcement learning algorithm is proposed. The second layer is used to plan the path of each flight so that the maximum number of inspection tasks can be completed each time the flight is taken off. The first layer is used to select an optimal path from the candidate path set generated by the second layer to minimize the average time required to complete the inspection of a single grid area. This makes the path planned and allocated by the machine nest for the drone have the characteristics of a large number of inspection tasks and a short average inspection time for a single grid area.

[0183] 2. In the second-layer deep reinforcement learning, based on the grid technology, a shortest distance penalty mechanism is proposed to improve the value function. When the next action of the drone causes the remaining endurance to be less than the shortest distance, the accumulated reward is cleared, that is, the value function value is 0, and the action selection strategy in the present invention is judged according to the value function. When the value function value is 0 after executing a certain step, it means that the probability of this action being selected is 0; at the same time, an adaptive hybrid strategy action selection mechanism is proposed, which combines the ε-greedy strategy and the Softmax strategy to form an adaptive hybrid strategy, which solves the problem that the traditional strategy is prone to fall into the local optimal solution and the algorithm converges slowly;

[0184] 3. An adaptive competition mechanism is proposed. When bidding, the competition is mainly based on the current average time per unit (i.e., the reward function value) and the overall expected average time per unit (value function value). If a fixed weight coefficient is set for competitive value calculation, the robustness of the algorithm will deteriorate. Therefore, the present invention sets an adaptive weight coefficient change. In the early stage of the algorithm, in order to enhance the exploratory nature of the algorithm, the current average time per unit is mainly used. In the later stage of the algorithm, in order to accelerate the convergence of the algorithm, the overall expected average time per unit is mainly used for competition.

[0185] Please refer to Figure 2 , Figure 2A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the present application provides an electronic device, including: a processor 301 and a memory 302, the processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not shown), the memory 302 stores a computer program executable by the processor 301, when the electronic device is running, the processor 301 executes the computer program to execute the unmanned aerial vehicle inspection scheduling method based on improved double-layer reinforcement learning in any optional implementation of the above-mentioned embodiment, so as to achieve the following functions: gridding the inspected city; when a drone reaches the local nest, the drone that has arrived at the local nest is taken as the target drone, the current inspection task completion information and the current time occupancy information of each nest are updated, and the inspection task completion information and the current time occupancy information of each nest are updated according to the network. The grid division result and the cruising range of the target UAV are used to determine the reachable nests that the target UAV can reach next, and a reachable nest set is obtained; according to the current inspection task completion information and the cruising range, the candidate path generation network model based on the deep reinforcement learning algorithm is used to generate the optimal paths from the local nest to each reachable nest, which are recorded as candidate paths, and a candidate path set is obtained; according to the current inspection task completion information, the current time occupancy information of each nest and the reachable nest set, a candidate path selection network model based on the deep reinforcement learning algorithm is used to select a candidate path from the candidate path set as the target path; according to the current time occupancy information of each nest and the target path, a conflict judgment is made, and the target path is assigned to the target UAV according to the conflict judgment result, or the target path is replanned and then assigned to the target UAV.

[0186] The embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the improved double-layer reinforcement learning-based drone inspection scheduling method in any optional implementation of the above embodiment is executed to achieve the following functions: gridding the inspected city; when a drone reaches the local nest, taking the drone that has reached the local nest as the target drone, updating the current inspection task completion status information and the current time occupancy status information of each nest, and determining the reachable nest that the target drone can reach next based on the grid division result and the cruising range of the target drone, and obtaining a reachable nest set; according to the current inspection task completion status information and the current time occupancy status information of each nest, the inspection task completion status information and the current time occupancy status information of each nest are updated, and according to the grid division result and the cruising range of the target drone, the reachable nest that the target drone can reach next is determined, and a reachable nest set is obtained; according to the current inspection task completion status information and the current time occupancy status information of each nest, the inspection task completion status information and the target drone are updated, and the inspection task completion status information and the target drone are updated; according to the current inspection task completion status information and the current time occupancy status information of each nest are updated, the inspection task completion status information and the target drone are updated, and the inspection task completion status information and the target drone are updated; according to the current inspection task completion status information and the current time occupancy status information of each nest are updated, the inspection task completion status information and the target drone are updated, and the inspection task completion status information and the reachable nest set are obtained; according to the current inspection task completion status information and the current time occupancy status information of each nest are updated, the inspection task completion status information and the reachable nest set are obtained ...; according to the current inspection task completion status information and the current time occupancy Based on the task completion information and the cruising distance, the candidate path generation network model based on the deep reinforcement learning algorithm is used to generate the optimal paths from the local nest to each reachable nest, which are recorded as candidate paths, and a candidate path set is obtained; according to the current inspection task completion information, the current time occupancy information of each nest and the reachable nest set, a candidate path is selected from the candidate path set as the target path using the candidate path selection network model based on the deep reinforcement learning algorithm; according to the current time occupancy information of each nest and the target path, a conflict judgment is made, and according to the conflict judgment result, the target path is assigned to the target UAV or the target path is replanned and then assigned to the target UAV. Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0187] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0188] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network model units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0189] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0190] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0191] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for scheduling UAV inspection based on improved double-layer reinforcement learning, applied to a nest of a UAV inspection system, wherein the UAV inspection system comprises a plurality of UAVs and a plurality of nests, wherein the nests are connected in communication; characterized in that: The UAV inspection scheduling method based on improved double-layer reinforcement learning includes the following steps: A1. Divide the inspected cities into grids; A2. When a drone reaches the local nest, the drone that has reached the local nest is taken as the target drone, and the current inspection task completion information and the current time occupancy information of each nest are updated. According to the grid division result and the cruising range of the target drone, the reachable nest that the target drone can reach next is determined to obtain the reachable nest set; A3. According to the current inspection task completion information and the cruising distance, the candidate path generation network model based on the deep reinforcement learning algorithm is used to generate the optimal path from the local nest to each of the reachable nests, recorded as the candidate path, and the candidate path set is obtained; A4. According to the current inspection task completion information, the current time occupancy information of each machine nest and the reachable machine nest set, a candidate path selection network model based on a deep reinforcement learning algorithm is used to select a candidate path from the candidate path set as the target path; A5. Conflict judgment is performed based on the current time occupancy information of each machine nest and the target path, and the target path is assigned to the target drone or the target path is replanned and assigned to the target drone based on the conflict judgment result; The inspection task completion information includes grid status information of each grid area; the grid status information includes first status information indicating that the grid area is a no-fly zone or an out-of-city area, second status information indicating that the grid area is temporarily unsuitable for flying, third status information indicating that the grid area has completed inspection, and fourth status information indicating that the grid area has not completed inspection; Step A3 includes taking each of the reachable machine nests as a target machine nest in turn and executing: A301. Obtaining the action space of the current time step; the action space of the current time step includes actions that are allowed to be executed, and the actions that are allowed to be executed include moving to each adjacent grid area whose grid state information is neither the first state information nor the second state information; A302. Obtain the first reference state information of the current time step; the first reference state information of the current time step includes the inspection task completion information of the current time step, the endurance distance and the minimum grid distance from the position of the current time step to the target machine nest; in the first time step, the inspection task completion information of the current time step is the current inspection task completion information, and the position of the current time step is the position of the machine nest; A303. Based on the first reference state information and the action space, the candidate path generation network model is used to calculate the first value function of each action; A304. According to the first value function, the Softmax strategy and the ε-greedy strategy are combined to select an action, and the selected action is executed; A305. If the selected action reaches the grid area where the target machine nest is located after execution, the path planning is terminated, and the path consisting of the grid areas pointed to by the actions selected at each time step is used as the candidate path from the local nest to the target machine nest. Otherwise, the next time step is entered and the execution returns to step A301; Step A4 includes: A401. The current inspection task completion information, the current time occupancy information of each machine nest and the reachable machine nest set are used as the second reference state information, and the second value function of each candidate path is calculated using the candidate path selection network model according to the second reference state information and the candidate path set; A402. According to the second value function, a candidate path is selected from the candidate path set as the target path based on the Softmax strategy.

2. The unmanned aerial vehicle inspection scheduling method based on improved double-layer reinforcement learning according to claim 1 is characterized in that: Step A1 includes: Obtain the minimum circumscribed rectangle of the boundary line of the inspected city; The minimum circumscribed rectangle is grid-divided to obtain a grid-divided result of the inspected city.

3. The unmanned aerial vehicle inspection scheduling method based on improved double-layer reinforcement learning according to claim 2 is characterized in that: In step A2, according to the grid division result and the cruising range of the target UAV, the reachable nest that the target UAV can reach next is determined, including: Other machine nests that meet the following conditions are used as candidate machine nests: in a grid matrix with the grid area where the other machine nests are located and the grid area where the local machine nest is located as diagonal grids, there is no grid area with grid state information of the first state information and no grid area with grid state information of the second state information; According to the Chebyshev distance calculation formula, the minimum grid distance from the host to each of the candidate host is calculated respectively, and converted into a corresponding flight distance, which is recorded as the minimum flight distance; The candidate aircraft nest whose minimum flight distance is not greater than the endurance distance is used as the reachable aircraft nest.

4. The unmanned aerial vehicle inspection scheduling method based on improved double-layer reinforcement learning according to claim 1 is characterized in that: Step A303 includes taking each of the actions as the target action in sequence and executing: Inputting the first reference state information and the target action into the candidate path generation network model to obtain a first value function prediction value output by the candidate path generation network model; If the remaining cruising distance of the target UAV after executing the target action is not less than the flight distance corresponding to the minimum grid distance from the position after executing the target action to the target nest, the predicted value of the first value function is used as the first value function of the target action; If the remaining cruising distance of the target UAV after executing the target action is less than the flight distance corresponding to the minimum grid distance from the position after executing the target action to the target machine nest, the first value function of the target action is set to 0.

5. The unmanned aerial vehicle inspection scheduling method based on improved double-layer reinforcement learning according to claim 1 is characterized in that: Step A304 includes: Generate a random number b in the range [0,1]; If b>ε, then randomly select an action from the action space and execute the selected action; ε is the reference probability of the ε-greedy strategy, and ε gradually increases with the increase of time steps; If b≤ε, then according to the first value function, an action is selected from the action space using the Softmax strategy, and the selected action is executed.

6. The unmanned aerial vehicle inspection scheduling method based on improved double-layer reinforcement learning according to claim 1 is characterized in that: Step A5 includes: A501. Extract the current time occupancy information of the terminal nest of the target path from the current time occupancy information of each nest; the terminal nest is the reachable nest at the end of the target path; A502. Calculate the target drone's use period of the terminal machine nest according to the target path, recorded as the first period; A503. Compare the first period of time with the terminal machine nest current time occupancy information to determine whether the first period of time is available; A504. If the first time period is not available, after removing the target path from the candidate path set, return to step A4; A505. If the first time period is available and does not conflict with other machine nests requesting to use the terminal machine nest, the target path is assigned to the target drone; A506. If the first time period is available and conflicts with the time period required by other machine nests to use the destination machine nest, the bid value of the target UAV is calculated based on the target path for bidding, and the target path is allocated to the target UAV after the bidding is successful. If the bidding fails, return to execute step A2.

7. An electronic device, characterized in that: It includes a processor and a memory, the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the drone inspection scheduling method based on improved two-layer reinforcement learning as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps in the drone inspection scheduling method based on improved double-layer reinforcement learning as described in any one of claims 1 to 6 are executed.

Images