Method and device for determining flight route of unmanned aerial vehicle and readable storage medium

Abstract

The invention discloses a method and a device for determining a flight route of an unmanned aerial vehicle, and a readable storage medium. The method comprises the following steps: acquiring a flight area into which the unmanned aerial vehicle is to enter and a flight state, the flight state being at least used for indicating at least one of the following information of the unmanned aerial vehicle: a flight position, a flight height and an execution action; determining an initial flight route of the unmanned aerial vehicle based on the flight area and the flight state; based on the initial flight route, a target obstacle in the flight area is detected, and the target obstacle is used for preventing the unmanned aerial vehicle from flying in the flight area; and adjusting the initial flight route based on the target obstacle to obtain a target flight route. The technical problem that the flight path planning of the unmanned aerial vehicle is not flexible is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle navigation and control, and in particular to a method, device and readable storage medium for determining a flight route of an unmanned aerial vehicle. Background Art

[0002] At present, with the rapid development of drone technology, its application in many fields such as urban renewal and power inspection is becoming more and more extensive. However, with the continuous expansion of application scenarios, the requirements for the intelligence of drones are also increasing. Especially in complex and changing environments, it is necessary to have a high level of path planning and autonomous obstacle avoidance capabilities to ensure the safety of flight and the efficiency of the mission. This not only requires drones to be able to complete mission objectives, but also requires the ability to avoid obstacles and optimize flight paths. Therefore, the development of efficient and reliable autonomous flight and obstacle avoidance technology has become a key research direction in the field of drones.

[0003] In the related technologies, existing UAV path planning is mostly based on traditional graph search algorithms and optimization algorithms, such as A* algorithm, Dijkstra algorithm and genetic algorithm. These methods can effectively calculate the optimal path in a static environment, but they perform poorly in a complex and dynamically changing environment. They often lack the ability to perceive obstacles in real time and dynamically adjust the path, and are difficult to deal with moving obstacles or sudden changes in the environment. In addition, the traditional methods have low computational efficiency and are prone to falling into local optimality, resulting in the planned path being inflexible. Therefore, there is a technical problem that the flight route planning of UAVs is inflexible.

[0004] Currently, no effective solution has been proposed to the above-mentioned technical problem of inflexible flight route planning of UAVs. Summary of the invention

[0005] The embodiments of the present invention provide a method, a device and a readable storage medium for determining a flight route of a drone, so as to at least solve the technical problem that the flight route planning of the drone is inflexible.

[0006] According to one aspect of an embodiment of the present invention, a method for determining a flight route of a drone is provided. The method may include: obtaining a flight area and a flight status that the drone is to enter, wherein the flight status is used to indicate at least one of the following information of the drone: flight position, flight altitude, and execution action; determining an initial flight route of the drone based on the flight area and the flight status; detecting target obstacles in the flight area based on the initial flight route, wherein the target obstacles are used to prevent the drone from flying in the flight area; adjusting the initial flight route based on the target obstacles to obtain a target flight route.

[0007] Optionally, obtaining the flight area and flight status to be entered by the UAV includes: obtaining the flight starting point and flight end point of the UAV; and determining the flight area based on the flight starting point and flight end point.

[0008] Optionally, based on the flight area and the flight status, determining the initial flight route of the UAV includes: dividing the flight area to obtain multiple flight sub-areas; and determining the initial flight route of the UAV based on the flight sub-areas and the flight status.

[0009] Optionally, based on the flight sub-area and the flight status, an initial flight route of the UAV is determined, including: obtaining the position of an initial obstacle in the flight area; predicting the flight result of the UAV based on the position of the initial obstacle, the flight sub-area and the flight status, wherein the flight result is used to indicate the degree to which the UAV is close to a preset position; inputting the flight result into an initial decision model for analysis to obtain an initial flight route, wherein the initial decision model is a pre-set decision model.

[0010] Optionally, based on the initial flight route, detecting target obstacles within the flight area includes: acquiring an environmental image of the UAV flying along the initial flight route, wherein the environmental image is used to indicate images collected by the UAV within the flight area; and identifying target obstacles in the environmental image.

[0011] Optionally, based on the target obstacle, the initial flight route is adjusted to determine the target flight route, including: determining the position of the target obstacle; comparing the position of the target obstacle with the position of the initial obstacle to obtain a comparison result, wherein the initial obstacle is used to indicate an obstacle that has been marked within the flight area; in response to the comparison result that the position of the target obstacle is inconsistent with the position of the initial obstacle, mapping the position of the target obstacle to the flight area to obtain a mapping result; based on the mapping result, adjusting the initial flight route to obtain the target flight route.

[0012] According to another aspect of an embodiment of the present invention, a device for determining a flight route of a drone is also provided. The device may include: an acquisition unit, used to acquire a flight area and a flight status that the drone is to enter, wherein the flight status is used to indicate at least one of the following information of the drone: flight position, flight altitude, and execution action; a first determination unit, used to determine an initial flight route of the drone based on the flight area and the flight status; a detection unit, used to detect the position of a target obstacle in the flight area based on the initial flight route, wherein the target obstacle is used to hinder the drone from flying in the flight area; and an adjustment unit, used to adjust the initial flight route based on the target obstacle to obtain a target flight route.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein when the program is executed by a processor, the device where the storage medium is located is controlled to execute the method for determining the flight route of the drone in an embodiment of the present invention.

[0014] According to another aspect of an embodiment of the present invention, a processor is further provided. The processor is used to run a program, wherein when the program is run, the method for determining the flight route of the drone in the embodiment of the present invention is executed.

[0015] According to another aspect of an embodiment of the present invention, a computer program product is further provided, which includes computer instructions, and when the computer instructions are executed by a processor, the method for determining the flight route of the UAV in the embodiment of the present invention is implemented.

[0016] In an embodiment of the present invention, the flight area and flight status that the drone is about to enter are obtained, wherein the flight status is used to indicate at least one of the following information of the drone: flight position, flight altitude, and execution action; based on the flight area and flight status, the initial flight route of the drone is determined; based on the initial flight route, target obstacles in the flight area are detected, wherein the target obstacles are used to hinder the drone from flying in the flight area; based on the target obstacles, the initial flight route is adjusted to obtain the target flight route. In other words, the embodiment of the present invention analyzes the flight area and flight status of the drone to determine the initial flight route, thereby determining the target obstacle, and then adjusting the flight route in real time according to the target obstacle to determine the target flight route, thereby solving the technical problem of inflexible flight route planning of the drone and achieving the technical effect of improving the flexibility of the flight route planning of the drone. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of a method for determining a flight route of a drone according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a method for dynamic path planning and obstacle avoidance of a drone based on a time difference reinforcement learning algorithm according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a UAV dynamic path planning and obstacle avoidance system based on a time difference reinforcement learning algorithm according to an embodiment of the present invention;

[0021] Figure 4is a schematic diagram of a device for determining a flight route of a drone according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, functional component or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, functional components or devices.

[0024] According to an embodiment of the present invention, an embodiment of a method for determining a flight route of an unmanned aerial vehicle is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0025] Figure 1 is a flow chart of a method for determining a flight route of a drone according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0026] Step S101, obtaining the flight area and flight status of the drone to be entered.

[0027] In the technical solution provided in the above step S101 of the present invention, the flight status is used to indicate at least one of the following information of the drone: flight position, flight altitude and execution action.

[0028] In this embodiment, the flight area and flight status of the drone to be entered are obtained. The flight status can be the state of the drone (such as position, altitude) and the actions it can perform (such as up, down, left turn, right turn, etc.).

[0029] For example, the flight area is determined according to the flight starting point and end point of the drone, and its boundary range is delineated. This is only an illustrative example and does not limit the specific method for obtaining the flight area to be entered by the drone.

[0030] Step S102, determining the initial flight route of the UAV based on the flight area and flight status.

[0031] In the technical solution provided in the above step S102 of the present invention, the initial flight route at least includes the UAV flight path and the execution action.

[0032] In this embodiment, after the flight area and flight status that the drone is to enter are acquired in step S101, the initial flight route of the drone is determined according to the flight area and the flight status.

[0033] Optionally, known obstacles are marked so as to determine the initial flight path of the UAV according to a decision reference table (Q table).

[0034] For example, a decision reference table (Q-table) is created to record the expected effects of different actions in each state. Its structure is a two-dimensional array, where rows represent different states and columns represent different actions. A state-action mapping relationship is established based on the defined states and actions. Each array element represents the expected reward value (Q value) of a state-action pair. The Q values ​​of these state-action pairs are initialized to 0, indicating that the drone has the same expected reward for all actions in each state at the beginning. Reward and penalty values ​​are set for each state-action pair to reflect the effect of the drone performing an action in a specific state. State-action pairs that are close to the target position will receive higher rewards, while those that are near obstacles or collide with obstacles will be punished. These defined reward and penalty values ​​are mapped to state-action pairs in the Q-table to guide the subsequent learning and decision-making process of the drone. Determine the initial flight route based on the expected effect of the same action.

[0035] Step S103: Detect target obstacles within the flight area based on the initial flight route.

[0036] In the technical solution provided in the above step S103 of the present invention, the target obstacle is used to prevent the UAV from flying within the flight area.

[0037] In this embodiment, after the initial flight route of the UAV is determined in step S102, the UAV is controlled to fly according to the initial flight route and detects target obstacles within the flight area.

[0038] For example, the target obstacles in the flight area are detected by using images collected in real time by the drone sensor. This is only an illustrative example and does not limit the specific method of detecting the target obstacles in the flight area.

[0039] Step S104: adjusting the initial flight path based on the target obstacle to obtain the target flight path.

[0040] In the technical solution provided in the above step S104 of the present invention, after the target obstacle in the flight area is detected in step S103, the initial flight route is adjusted according to the target obstacle to obtain the target flight route.

[0041] In this embodiment, the target obstacle is mapped into the flight area, so that the initial flight route is adjusted and the target flight route is determined.

[0042] For example, the detected obstacle information is mapped to the corresponding grids, thereby marking which grids have obstacles, as shown in the following formula (1):

[0043] utput=Conv 1×1 (σ(UpSampling(σ(Conv(F low-level ))))+F skip ) (1)

[0044] Among them, F low-level Represents the low-level features extracted by the encoder part; UpSampling is an upsampling operation used to restore the feature map to a higher resolution; F skip is the feature in the skip connection, which is used to fuse low-level and high-level features; σ is the ReLU activation function, which is used to introduce nonlinearity.

[0045] It should be noted that the above embodiments can be implemented by a UAV dynamic path planning and obstacle avoidance system based on a time difference reinforcement learning algorithm.

[0046] In the above steps S101 to S104 of the present invention, the flight area and flight status to be entered by the UAV are obtained, wherein the flight status is used to indicate at least one of the following information of the UAV: ​​flight position, flight altitude and execution action; based on the flight area and flight status, the initial flight route of the UAV is determined; based on the initial flight route, target obstacles in the flight area are detected, wherein the target obstacles are used to hinder the flight of the UAV in the flight area; based on the target obstacles, the initial flight route is adjusted to determine the target flight route. In other words, the embodiment of the present invention analyzes the flight area and flight status of the UAV to determine the initial flight route, thereby determining the target obstacle, and then adjusts the flight route in real time according to the target obstacle to determine the target flight route, thereby solving the technical problem of inflexible flight route planning of the UAV and achieving the technical effect of improving the flexibility of flight route planning of the UAV.

[0047] The above method of this embodiment is further introduced below.

[0048] As an optional implementation method, obtaining the flight area and flight status of the drone to be entered includes: obtaining the flight starting point and flight end point of the drone; and determining the flight area based on the flight starting point and flight end point.

[0049] In this embodiment, the flight start point and the flight end point of the drone are obtained, and the flight start point and the flight end point are used to define a boundary range, thereby determining the flight area.

[0050] Alternatively, determining the start and end points of a flight can help drone pilots avoid entering unknown or no-fly areas, thereby avoiding interference with other aircraft or ground facilities.

[0051] As an optional implementation method, the initial flight route of the UAV is determined based on the flight area and the flight status, including: dividing the flight area to obtain multiple flight sub-areas; and determining the initial flight route of the UAV based on the flight sub-areas and the flight status.

[0052] In this embodiment, the flight area is segmented to obtain flight sub-areas. For example, the flight area is gridded.

[0053] For example, the regular grid division method is used to divide the entire flight area into several square grid units of fixed size at equal intervals. Each grid unit represents a specific location in the flight area, which facilitates subsequent UAV path planning and obstacle avoidance.

[0054] Optionally, after obtaining the flight sub-area, the initial flight route of the UAV is determined according to the flight sub-area and the flight state. The specific determination method is as follows.

[0055] As an optional implementation method, the initial flight route of the UAV is determined based on the flight sub-area and the flight status, including: obtaining the position of the initial obstacle in the flight area; predicting the flight result of the UAV based on the position of the initial obstacle, the flight sub-area and the flight status, wherein the flight result is used to indicate the degree to which the UAV is close to a preset position; inputting the flight result into an initial decision model for analysis to obtain an initial flight route, wherein the initial decision model is a pre-set decision model.

[0056] In this embodiment, the position of the initial obstacle in the flight area is obtained, and the position of the initial obstacle is marked in the flight sub-area, so as to predict the flight result of the drone according to the position of the initial obstacle, the flight sub-area and the flight state. For example, the flight result of the drone is predicted by creating a decision reference table (Q table).

[0057] For example, a decision reference table (Q table) is created to record the expected effects of different actions in each state. Its structure is a two-dimensional array, where rows represent different states and columns represent different actions. A state-action mapping relationship is established based on the defined states and actions. Each array element represents the expected reward value (Q value) of a state-action pair. The Q values ​​of these state-action pairs are initialized to 0, indicating that the drone has the same expected reward for all actions in each state at the beginning. Reward and penalty values ​​are set for each state-action pair to reflect the effect of the drone performing an action in a specific state.

[0058] Optionally, the flight result is input into an initial decision model for analysis to obtain an initial flight route, wherein the initial decision model may be an ∈-greedy strategy.

[0059] For example, the drone queries and evaluates the Q values ​​of all actions in the grid it is in, and selects and executes the next action based on the ∈-greedy strategy. Specifically: the action with the highest current Q value is selected with a probability of 1-∈, thereby making the best decision based on existing experience; an action is randomly selected with a probability of ∈ to avoid always choosing the same path, thereby having the opportunity to discover a better solution, as shown in the following formula (2):

[0060]

[0061] Among them, λ t represents the action selected by the drone at time step t; γ t represents the state of the drone at time step t; Q(γ t ,λ) is in state γ t The Q value of executing action λ is the expected reward that can be obtained; ∈ represents the parameter of exploration probability.

[0062] As an optional implementation method, based on the initial flight route, detecting target obstacles in the flight area includes: obtaining an environmental image of the UAV flying along the initial flight route, wherein the environmental image is used to indicate the image collected by the UAV in the flight area; and identifying the target obstacles in the environmental image.

[0063] In this embodiment, an image of the environment in which the drone flies along the initial flight path is obtained, for example, by collecting images in real time through the sensors of the drone.

[0064] Optionally, target obstacles in the environment image are identified from the collected images, for example, using a deep learning network to identify the target obstacles.

[0065] For example, deep learning networks such as U-Net are used to process the collected images to identify and mark the locations of obstacles in the images.

[0066] As an optional implementation method, based on the target obstacle, the initial flight route is adjusted to determine the target flight route, including: determining the position of the target obstacle; comparing the position of the target obstacle with the position of the initial obstacle to obtain a comparison result, wherein the initial obstacle is used to indicate an obstacle that has been marked in the flight area; in response to the comparison result that the position of the target obstacle is inconsistent with the position of the initial obstacle, mapping the position of the target obstacle to the flight area to obtain a mapping result; based on the mapping result, adjusting the initial flight route to obtain the target flight route.

[0067] In this embodiment, the position of the target obstacle is determined; the position of the target obstacle is compared with the position of the initial obstacle to obtain a comparison result. When the comparison result shows that the target position is inconsistent with the position of the initial obstacle, it means that the obstacle is not marked in the flight area. Based on this, the position of the target obstacle is mapped to the flight area.

[0068] For example, the sensors equipped on the drone are used to collect environmental data in real time, and the U-Net network is used to identify and segment different areas or objects in the image to accurately detect obstacles in the image. The detected obstacle information will be mapped to the corresponding grids, thereby marking which grids have obstacles, as expressed in the above formula (1), which will not be repeated here.

[0069] Optionally, after determining the target obstacle, the initial flight route is adjusted to determine the target flight route. For example, the Q table is continuously updated using a time difference algorithm, experiences under different states are gradually accumulated, and the Q value is adjusted according to reward feedback to optimize the path selection.

[0070] It should be noted that the above embodiments can be implemented by a UAV dynamic path planning and obstacle avoidance system based on a time difference reinforcement learning algorithm.

[0071] In this embodiment, the flight area and flight status that the drone is about to enter are obtained, wherein the flight status is used to indicate at least one of the following information of the drone: flight position, flight altitude, and execution action; based on the flight area and flight status, the initial flight route of the drone is determined; based on the initial flight route, the target obstacle in the flight area is detected, wherein the target obstacle is used to hinder the drone from flying in the flight area; based on the target obstacle, the initial flight route is adjusted to obtain the target flight route. In other words, the embodiment of the present invention analyzes the flight area and flight status of the drone to determine the initial flight route, thereby determining the target obstacle, and then adjusting the flight route in real time according to the target obstacle to determine the target flight route, thereby solving the technical problem of inflexible flight route planning of the drone and achieving the technical effect of improving the flexibility of the flight route planning of the drone.

[0072] The technical solution of the embodiment of the present invention is illustrated below in conjunction with preferred implementation modes.

[0073] At present, with the rapid development of drone technology, its application in many fields such as urban renewal and power inspection is becoming more and more extensive. However, with the continuous expansion of application scenarios, the requirements for the intelligence of drones are also increasing. Especially in complex and changing environments, it is necessary to have a high level of path planning and autonomous obstacle avoidance capabilities to ensure the safety of flight and the efficiency of the mission. This not only requires drones to be able to complete mission objectives, but also requires the ability to avoid obstacles and optimize flight paths. Therefore, the development of efficient and reliable autonomous flight and obstacle avoidance technology has become a key research direction in the field of drones.

[0074] In the related technology, the existing UAV path planning is mostly based on traditional graph search algorithms and optimization algorithms, such as A* algorithm, Dijkstra algorithm and genetic algorithm. These methods can effectively calculate the optimal path in a static environment, but they perform poorly in a complex and dynamically changing environment. They often lack the ability to perceive obstacles in real time and dynamically adjust the path, and are difficult to deal with moving obstacles or sudden changes in the environment. In addition, the traditional method has low computational efficiency and is prone to falling into local optimality, resulting in the planned path being inflexible. Therefore, there is a technical problem that the flight route planning of the UAV is inflexible. At present, no effective solution has been proposed for the above-mentioned technical problem of inflexible flight route planning of UAVs.

[0075] However, the embodiment of the present invention proposes a method for dynamic path planning and obstacle avoidance of a drone based on a time difference reinforcement learning algorithm, which determines the flight area according to the flight starting point and end point of the drone, delimits its boundary range, and performs grid processing on the flight area. At the same time, the obstacle position in the grid is identified and marked by the image collected in real time by the drone sensor. The state of the drone (such as position, height) and its executable actions (such as up, down, left turn, right turn, etc.) are defined for each grid, and a decision reference table is initialized. The drone selects an action based on the current state and the ∈-greedy strategy, which balances the needs of exploring new paths and using the existing best path. After the drone executes the action, the position of the drone is adjusted in real time according to the sensor feedback, and the corresponding reward or penalty is obtained according to the environmental feedback, and the Q value in the decision reference table is dynamically updated based on the time difference algorithm; the drone generates the optimal route according to the optimized decision reference table and avoids all obstacles, and flies along the optimal route from the starting point until it reaches the end point, which solves the technical problem of inflexible flight route planning of the drone and achieves the technical effect of improving the flexibility of flight route planning of the drone.

[0076] The embodiments of the present invention are further described below.

[0077] Figure 2 : is a flowchart of a method for dynamic path planning and obstacle avoidance of a drone based on a time difference reinforcement learning algorithm according to an embodiment of the present invention. The analysis method includes the following steps:

[0078] Step S201, determine the flight area and perform grid processing on the flight area.

[0079] In this embodiment, the flight area is determined according to the flight start and end point of the drone, its boundary range is demarcated, and the flight area is gridded. At the same time, the obstacle positions in the grid are identified and marked through the images collected in real time by the drone sensor.

[0080] Optionally, the starting point and end point of the UAV flight are first determined, and the actual flight area boundary of the UAV is further refined and delineated by drawing a minimum circumscribed rectangle containing the starting point and the end point, which not only covers the path between the starting point and the end point, but also takes into account possible flight deviations and path planning requirements, providing a reasonable operating space for the path planning of the UAV.

[0081] Optionally, after determining the flight area, the area needs to be gridded. Using the regular grid division method, the entire flight area is divided into a number of square grid units of fixed size at equal intervals. Each grid unit represents a specific position in the flight area, which is convenient for subsequent drone path planning and obstacle avoidance.

[0082] Optionally, the sensors equipped by the drone are used to collect environmental data in real time, and the different areas or objects in the image are identified and segmented through the U-Net network to achieve accurate detection of obstacles in the image. The detected obstacle information will be mapped to the corresponding grids, thereby marking which grids have obstacles, as expressed in the above formula (1), which will not be repeated here.

[0083] Step S202, initializing a decision reference table.

[0084] In this embodiment, the state of the drone (such as position, altitude) and its executable actions (such as up, down, left turn, right turn, etc.) are defined for each grid, and a decision reference table is initialized to record the expected effects of selecting different actions in each state.

[0085] Optionally, the state information of the drone is defined for each grid cell, including position, altitude, etc., to describe the specific position of the drone in the flight area and the current flight conditions. At the same time, unique coordinate information is assigned to each grid according to the gridding result, and the state definition is expanded into three-dimensional information in combination with the flight altitude.

[0086] Optionally, the executable actions of the drone are defined in each state, including up, down, forward and backward, etc., which represent the adjustments or obstacle avoidance options that the drone can make in the current state. Based on the grid division, the adjacent grids around each grid unit provide movable directions, ensuring that the drone can flexibly respond to environmental changes.

[0087] Optionally, a decision reference table (Q table) is created to record the expected effects of different actions in each state. Its structure is a two-dimensional array, where rows represent different states and columns represent different actions. A state-action mapping relationship is established based on the defined states and actions. Each array element represents the expected reward value (Q value) of a state-action pair. The Q values ​​of these state-action pairs are initialized to 0, indicating that the drone has the same expected reward for all actions in each state at the beginning.

[0088] Optionally, set reward and penalty values ​​for each state-action pair to reflect the effect of the drone performing an action in a specific state. State-action pairs that are close to the target location will receive higher rewards, while those that are near obstacles or collide with obstacles will be penalized. Map these defined reward and penalty values ​​to state-action pairs in the Q-table to guide the subsequent drone's learning and decision-making process.

[0089] Step S203: The drone selects an action and executes it, while adjusting the position of the drone in real time.

[0090] In this embodiment, the drone selects an action based on the current state and the ∈-greedy strategy, which balances the needs of exploring new paths and utilizing the existing best path. After the drone performs the action, the drone's position is adjusted in real time according to sensor feedback. At the same time, it obtains corresponding rewards or penalties based on environmental feedback and dynamically updates the Q value in the decision reference table based on the time difference algorithm.

[0091] Optionally, the drone queries and evaluates the Q values ​​of all actions in the grid it is in, and selects and executes the next action based on the ∈-greedy strategy. Specifically: the action with the highest current Q value is selected with a probability of 1-∈, so as to make the best decision using existing experience; an action is randomly selected with a probability of ∈, avoiding always choosing the same path, so as to have the opportunity to find a better solution, as expressed in the above formula (2), which will not be repeated here.

[0092] Optionally, when the drone moves to a new grid after performing an action, it obtains the reward or penalty of the current state through sensor feedback and calculates the time difference target value The specific formula is as shown in the following formula (3):

[0093]

[0094] in, Represents the sum of the current reward and the best possible reward in the future; R t represents the current reward or penalty; γ is the discount factor, which is used to measure the importance of future rewards; maxQ(γ t+1 ,λ) reflects the maximum reward that the drone can expect to obtain in the new state.

[0095] Optionally, the time difference error δ is calculated based on the result of the time difference target value t , which represents the difference between the actual reward of the current state-action pair and the existing Q value, and reflects the deviation between the drone's decision and the actual environment feedback. It is specifically shown in the following formula (4):

[0096]

[0097] in, is the time difference target value; Q(γ t ,λ t ) is the current state γ t Next select action λ t The predicted return value (Q value).

[0098] Optionally, the Q value is updated according to the calculated time difference error, and the Q value is gradually adjusted to more accurately reflect the long-term effect of each action. The specific example is shown in the following formula (5):

[0099] Q(s,a)←Q(s,a)+α[r+γ·maxQ(s ′ ,a ′ )-Q(s,a)](5)

[0100] Where Q(s,a) is the Q value of the current state-action pair (s,a); α is the learning rate, which controls the degree of Q value update; r is the actual reward or penalty obtained; γ is the discount factor, which is used to measure the importance of future rewards; maxQ(s ′ ,a ′ ) indicates that in the new state s ′ All possible actions a ′ The maximum expected return value.

[0101] Step S204, generating an optimal route according to the optimized decision reference table.

[0102] In this embodiment, after the training is completed, the drone generates an optimal route based on the optimized decision reference table and avoids all obstacles, and flies along the optimal route from the starting point until it reaches the end point.

[0103] Optionally, through multiple iterative training, the Q table is continuously updated using the time difference algorithm, experience in different states is gradually accumulated, and the Q value is adjusted according to reward feedback to optimize path selection.

[0104] Optionally, after training is completed, the drone generates the optimal route using the optimized Q-table. The Q-table stores the best action in each state, and generates an optimal path from the starting point to the end point by selecting the highest Q-value action for each state-action pair.

[0105] Optionally, under the guidance of the generated optimal route and obstacle avoidance strategy, the drone flies along the planned path from the starting point. At the same time, in actual flight, the drone makes dynamic adjustments based on real-time sensor data to avoid static and dynamic obstacles and maintain safe flight.

[0106] Figure 3 It is a schematic diagram of a UAV dynamic path planning and obstacle avoidance system based on a time difference reinforcement learning algorithm according to an embodiment of the present invention. The UAV dynamic path planning and obstacle avoidance system 300 includes: a data acquisition and preprocessing module 301, a flight state perception module 302 and a path planning and obstacle avoidance module 303.

[0107] Data collection and preprocessing module 301 is mainly responsible for collecting environmental data of the flight area in real time through sensors equipped by the drone, including terrain information, specific locations of obstacles and dynamic changes, etc. The collected images are processed using deep learning networks such as U-Net to identify and mark the locations of obstacles in the images, and finally these obstacle information are mapped to the corresponding grids to facilitate subsequent path planning and obstacle avoidance decisions. This module is also responsible for preprocessing the data to ensure the timeliness and accuracy of the data.

[0108] Flight state perception module 302, in this module, the system will perceive and evaluate the state of the drone's environment. By gridding the flight area, each grid represents a specific geographical location, and each grid is assigned a state and action. At the same time, the real-time feedback data from the sensor will be used to update the state information of obstacles and dynamically adjust the drone's environmental perception. This module uses a time difference algorithm to dynamically evaluate the rewards or penalties of each state and action, and further correct the drone's decision reference table (Q table).

[0109] Path planning and obstacle avoidance module 303, based on the time difference reinforcement learning algorithm, the drone selects the optimal action according to the real-time environmental information. First, based on the current state, the drone selects the optimal action through the ∈-greedy strategy. Then, the system dynamically adjusts the path through sensor data to avoid obstacles, and updates the Q value in the Q table according to the reward or penalty mechanism. With the continuous iteration of training, the data in the Q table is gradually optimized, and the drone can eventually autonomously generate an optimal path from the starting point to the end point, and avoid dynamic and static obstacles in real time during flight.

[0110] In this embodiment, through the reinforcement learning algorithm based on time difference, the drone can plan the optimal path in real time in a dynamic environment and avoid obstacles, and continuously optimize flight efficiency by using adaptive learning. The deep learning model is combined to detect and update obstacles in real time to ensure the accuracy of path planning. At the same time, the decision-making efficiency is improved through the ∈-greedy strategy, so that the drone has higher safety and flexibility in complex environments, and is widely applicable to a variety of drone application scenarios.

[0111] In this embodiment, the flight area is determined according to the flight starting point and end point of the drone, its boundary range is delineated, and the flight area is gridded. At the same time, the obstacle positions in the grid are identified and marked through the images collected in real time by the drone sensor. The state of the drone (such as position, height) and its executable actions (such as up, down, left turn, right turn, etc.) are defined for each grid, and a decision reference table is initialized. The drone selects an action based on the current state and the ∈-greedy strategy. The strategy balances the needs of exploring new paths and using the existing best path. After the drone executes the action, the position of the drone is adjusted in real time according to the sensor feedback, and the corresponding reward or penalty is obtained according to the environmental feedback, and the Q value in the decision reference table is dynamically updated based on the time difference algorithm; the drone generates the optimal route according to the optimized decision reference table and avoids all obstacles, and flies along the optimal route from the starting point until it reaches the end point, which solves the technical problem of the inflexible flight route planning of the drone and achieves the technical effect of improving the flexibility of the flight route planning of the drone.

[0112] According to an embodiment of the present invention, a device for determining a flight path of a drone is also provided. It should be noted that the device for determining a flight path of a drone can be used to execute the method for determining a flight path of a drone in the method embodiment.

[0113] Figure 4 Schematic diagram of a device for determining a flight path of a drone according to an embodiment of the present invention. Figure 4 As shown, the device 400 for determining the flight path of the UAV may include: an acquisition unit 401 , a first determination unit 402 , a detection unit 403 and an adjustment unit 404 .

[0114] The acquisition unit 401 is used to acquire the flight area and flight status of the drone to be entered, wherein the flight status is used to indicate at least one of the following information of the drone: flight position, flight altitude and execution action.

[0115] The first determining unit 402 is used to determine an initial flight route of the UAV based on the flight area and the flight status.

[0116] The detection unit 403 is used to detect the position of the target obstacle in the flight area based on the initial flight route, wherein the target obstacle is used to prevent the UAV from flying in the flight area.

[0117] The adjustment unit 404 is used to adjust the initial flight path based on the target obstacle to obtain a target flight path.

[0118] Optionally, the acquisition unit 401 may include: a first acquisition module, used to acquire the flight start point and the flight end point of the drone; and a first determination module, used to determine the flight area based on the flight start point and the flight end point.

[0119] Optionally, the first determination unit 402 may include: a segmentation module, used to segment the flight area to obtain multiple flight sub-areas; and a second determination module, used to determine the initial flight route of the UAV based on the flight sub-areas and the flight status.

[0120] Optionally, the second determination module may include: an acquisition submodule, used to acquire the position of the initial obstacle in the flight area; a prediction submodule, used to predict the flight result of the UAV based on the position of the initial obstacle, the flight subarea and the flight status, wherein the flight result is used to indicate the degree to which the UAV is close to the preset position; an analysis submodule, used to input the flight result into an initial decision model for analysis to obtain an initial flight route, wherein the initial decision model is a pre-set decision model.

[0121] Optionally, the detection unit 403 may include: a second acquisition module, used to acquire an environmental image of the UAV flying along the initial flight route, wherein the environmental image is used to indicate the image collected by the UAV within the flight area; and an identification module, used to identify target obstacles in the environmental image.

[0122] Optionally, the adjustment unit 404 may include: a third determination module, used to determine the position of the target obstacle; a comparison module, used to compare the position of the target obstacle with the position of the initial obstacle to obtain a comparison result, wherein the initial obstacle is used to indicate an obstacle that has been marked in the flight area; a mapping module, used to map the position of the target obstacle to the flight area in response to the comparison result being that the position of the target obstacle is inconsistent with the position of the initial obstacle to obtain a mapping result; and an adjustment module, used to adjust the initial flight route based on the mapping result to obtain the target flight route.

[0123] In this embodiment, the flight area and flight status that the drone is about to enter are obtained, wherein the flight status is used to indicate at least one of the following information of the drone: flight position, flight altitude, and execution action; based on the flight area and flight status, the initial flight route of the drone is determined; based on the initial flight route, the target obstacle in the flight area is detected, wherein the target obstacle is used to hinder the drone from flying in the flight area; based on the target obstacle, the initial flight route is adjusted to obtain the target flight route. In other words, the embodiment of the present invention analyzes the flight area and flight status of the drone to determine the initial flight route, thereby determining the target obstacle, and then adjusting the flight route in real time according to the target obstacle to determine the target flight route, thereby solving the technical problem of inflexible flight route planning of the drone and achieving the technical effect of improving the flexibility of the flight route planning of the drone.

[0124] According to an embodiment of the present invention, a computer-readable storage medium is also provided, which storage medium includes a stored program, wherein the program executes the method for determining the flight route of the drone in the method embodiment.

[0125] According to an embodiment of the present invention, a processor is further provided, the processor being used to run a program, wherein when the program is run, the method for determining the flight route of the drone in the method embodiment is executed.

[0126] According to an embodiment of the present invention, a computer program product is also provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the method for determining the flight route of the drone in the method embodiment is implemented.

[0127] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0128] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0131] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent functional component, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software functional component, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program codes.

[0133] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for determining a flight route of an unmanned aerial vehicle, characterized in that: include: Acquire a flight area and a flight status of the drone to be entered, wherein the flight status is used to indicate at least one of the following information of the drone: flight position, flight altitude, and execution action; Determining an initial flight route of the UAV based on the flight area and the flight status; Based on the initial flight route, detecting a target obstacle within the flight area, wherein the target obstacle is used to hinder the UAV from flying within the flight area; Based on the target obstacle, the initial flight route is adjusted to obtain a target flight route.

2. The method according to claim 1, characterized in that Get the flight area and flight status that the drone is about to enter, including: Obtaining the flight start point and flight end point of the UAV; The flight area is determined based on the flight start point and the flight end point.

3. The method according to claim 1, characterized in that Determining an initial flight route of the UAV based on the flight area and the flight state includes: Segmenting the flight area to obtain a plurality of flight sub-areas; Based on the flight sub-area and the flight status, the initial flight route of the UAV is determined.

4. The method according to claim 3, characterized in that Determining the initial flight route of the UAV based on the flight sub-area and the flight state includes: Obtaining the position of the initial obstacle in the flight area; Based on the position of the initial obstacle, the flight sub-area and the flight state, predicting a flight result of the UAV, wherein the flight result is used to indicate the degree to which the UAV is close to a preset position; The flight result is input into an initial decision model for analysis to obtain the initial flight route, wherein the initial decision model is a pre-set decision model.

5. The method according to claim 1, characterized in that Detecting target obstacles within the flight area based on the initial flight route includes: Acquire an environmental image of the UAV flying along the initial flight route, wherein the environmental image is used to indicate an image collected by the UAV within the flight area; The target obstacle in the environment image is identified.

6. The method according to claim 1, characterized in that Based on the target obstacle, the initial flight route is adjusted to determine a target flight route, including: determining a position of the target obstacle; Comparing the position of the target obstacle with the position of the initial obstacle to obtain a comparison result, wherein the initial obstacle is used to indicate an obstacle that has been marked within the flight area; In response to the comparison result being that the position of the target obstacle is inconsistent with the position of the initial obstacle, mapping the position of the target obstacle to the flight area to obtain a mapping result; Based on the mapping result, the initial flight route is adjusted to obtain the target flight route.

7. A device for determining a flight path of an unmanned aerial vehicle, characterized in that: include: An acquisition unit, used for acquiring a flight area to be entered by the UAV and a flight status, wherein the flight status is used to indicate at least one of the following information of the UAV: ​​flight position, flight altitude and execution action; A first determining unit, configured to determine an initial flight route of the UAV based on the flight area and the flight state; a detection unit, configured to detect a position of a target obstacle within the flight area based on the initial flight route, wherein the target obstacle is used to prevent the UAV from flying within the flight area; An adjustment unit is used to adjust the initial flight path based on the target obstacle to obtain a target flight path.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 6 when running.

10. A computer program product, characterized in that The computer program product comprises computer instructions, which implement the method according to any one of claims 1 to 6 when executed by a processor.

Images