End-to-end unmanned aerial vehicle forced landing point decision-making method and system based on deep reinforcement learning under multi-constraint condition

Abstract

The invention discloses an end-to-end unmanned aerial vehicle forced landing point decision-making method and system based on deep reinforcement learning under a multi-constraint condition. The system comprises a data acquisition module, a data preprocessing module, a feature extraction module and a deep reinforcement learning evaluation module. A multi-source sensor is used for collecting information, multiple constraint conditions are considered, and a deep reinforcement learning end-to-end learning method is used for evaluating the safe forced landing point of the unmanned aerial vehicle. According to the system, a model capable of comprehensively, accurately and efficiently carrying out forced landing point safety assessment can be realized, and the application of the unmanned aerial vehicle in the aspect of forced landing point safety assessment is perfected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle safety, in particular to an end-to-end unmanned aerial vehicle forced landing point decision-making method and system based on deep reinforcement learning under multiple constraint conditions. BACKGROUND

[0002] With the wide application of unmanned aerial vehicles in various fields, their flight safety has attracted much attention. During flight, unmanned aerial vehicles may encounter various unexpected situations, such as mechanical failure, battery depletion, signal interference, etc., at which time a safe forced landing point needs to be found in time. However, the traditional forced landing point evaluation method often relies on simple rules and limited information, and is difficult to cope with complex and variable environments and multiple constraint conditions. For example, considering only terrain factors may lead to neglect of the impact of constraints such as insufficient power of the aircraft itself, flight performance, and limited flight range; experience-based judgment may fail when facing new environments or unknown factors. At the same time, the existing evaluation method lacks adaptive learning ability and cannot adjust the strategy in time according to the changing situation.

[0003] Deep reinforcement learning, as an advanced machine learning technology, has great potential in dealing with complex decision-making problems. End-to-end learning is a common approach to solving problems in machine learning. Traditional data processing systems or learning systems require multiple stages of preprocessing and intermediate processing, i.e., breaking down a problem into multiple steps to solve it separately. End-to-end learning ignores the intermediate process and replaces it with a single neural network or a combination of multiple neural networks, directly obtaining the output from the input data. The advantage of end-to-end is to reduce manual processing, including preprocessing and subsequent processing, while increasing the space for the model to adjust itself based on learning data. Extending the end-to-end approach to deep reinforcement learning algorithms means that the input end inputs multi-dimensional data representing the state of the unmanned aerial vehicle, and the output end directly outputs the speed control quantity, eliminating all intermediate processes. It enables the agent to continuously learn the optimal strategy in interaction with the environment, but its application in the field of unmanned aerial vehicle safety forced landing point evaluation is not mature enough, and there is a lack of a comprehensive, efficient and adaptable evaluation system and method under multiple constraint conditions. SUMMARY

[0004] The purpose of the present application is to provide an end-to-end unmanned aerial vehicle forced landing point decision-making method and system based on deep reinforcement learning under multiple constraint conditions to overcome the deficiencies in the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions: The present application discloses an end-to-end unmanned aerial vehicle forced landing point decision-making method based on deep reinforcement learning under multiple constraint conditions, which specifically comprises the following steps: S1, initialization setting before takeoff of the unmanned aerial vehicle; S2, during the flight of the unmanned aerial vehicle, various data are continuously collected, including two-dimensional image data, radar point cloud data, satellite positioning information, battery state information, flight state information, and weather information; and the data are preprocessed; S3, terrain distribution features, obstacle features, battery power features, power consumption speed features, maximum flight distance features, ground flatness features, landing point area features, and weather features are extracted from the preprocessed data, and the above features are combined into a state vector required by a deep reinforcement learning model; S4, a deep deterministic policy gradient method model is constructed, and model training is performed, including an actor network and a critic network; the actor network is responsible for generating a forced landing point selection and landing strategy action according to an input state, and the critic network is used to evaluate the value of the action-state pair; the policy is continuously optimized through interaction with the environment and training to realize safe evaluation of the forced landing point.

[0006] Preferably, the preprocessing in step S2 includes cleaning the collected various types of data to remove noise and outliers; performing format conversion to unify the data format; and time synchronization processing.

[0007] Preferably, step S3 specifically includes the following operations: S31, terrain distribution features: image recognition analysis technology is used to process the ground pictures taken by the camera, classify the pixels in the image into different terrain categories, count the area proportion and distribution of each type of terrain in the image, and extract the terrain distribution feature vector; S32, obstacle features: the types, positions, sizes, and distributions of obstacles are identified using visual images and laser radar data, and the minimum distance between the aircraft and the obstacles and the obstacle density features are calculated; S33, battery power features: real-time battery power information is obtained, and the remaining power percentage and power change trend features are calculated; S34, battery power consumption speed features: the change of battery power with time is monitored, and the battery power consumption per unit time is calculated; S35, maximum flight distance features: the flight distance is obtained according to the battery state information obtained by the data acquisition module; S36, ground flatness features: the ground height change is measured by laser radar, and the ground flatness index is calculated; S37, landing point area features: the boundaries of potential landing points are determined according to visual images and laser radar data, and the area size of the landing point is calculated; S38, weather features: from the temperature and humidity data obtained from the weather sensor, the features related to the flight performance of the unmanned aerial vehicle are extracted; according to the influence degree of different temperature and humidity intervals on the flight of the unmanned aerial vehicle, the temperature and humidity, wind speed and direction data are discretized and divided into different levels, and the corresponding levels are coded as feature values, so that the model can consider the influence of temperature and humidity environment factors on the forced landing safety.

[0008] As preferred, the actor network has the following characteristics: The input layer receives the state vector generated by the feature extraction module, which contains obstacle feature, battery power feature, battery power consumption speed feature, ground flatness feature and landing point area feature information; The hidden layer adopts a multi-layer fully connected neural network structure, and the number of neurons in each layer is set according to the actual situation, and an activation function is used to increase the non-linear expression ability of the network; The output layer outputs the forced landing point selection action, including the coordinate position of the forced landing point, and the landing strategy action, such as the output value of the landing speed and landing angle, which is normalized to make it within the action space range.

[0009] As preferred, the critic network has the following characteristics: The input layer receives the splicing of the state vector and the action vector generated by the actor network, and the dimension is the sum of the dimension of the state vector and the dimension of the action vector; The hidden layer also adopts a multi-layer fully connected neural network structure, and the number of neurons in each layer is set according to the actual situation, and an activation function is used to increase the non-linear expression ability of the network; The output layer outputs a scalar value representing the value estimate of the current action-state pair, which is used to evaluate the quality of the strategy generated by the actor network.

[0010] As preferred, the training mechanism of the deep deterministic policy gradient method model includes: Experience replay: an experience replay buffer is established to store experience tuples of the interaction between the agent and the environment, and a batch of experiences is randomly sampled from the buffer for training; Target network update: a target network is established for the actor network and the critic network respectively, and the parameter update of the target network adopts a soft update method to stabilize the training process; Reward function design: a reward function is designed according to the obstacle feature, battery power feature, battery power consumption speed feature, ground flatness feature and landing point area feature, which comprehensively considers the influence of various factors on the safety of the forced landing point; Training algorithm: based on the deep deterministic policy gradient method, the actor network updates the policy parameters by maximizing the value estimate given by the critic network, and the critic network updates the parameters by minimizing the mean square error between the predicted value and the target value, and the training process continues until the preset convergence condition is met.

[0011] The application also discloses an end-to-end unmanned aerial vehicle forced landing point decision device based on deep reinforcement learning under multiple constraint conditions. A data acquisition module is used to acquire multi-source data related to the forced landing point. Two-dimensional image information is acquired through a camera; three-dimensional terrain information is acquired through a laser radar; accurate position data is acquired through a satellite positioning signal; detailed weather information of temperature, humidity and wind force is acquired through an airborne temperature and humidity sensor and a meteorological sensor; battery power and power consumption speed data are acquired through a battery management system; a data preprocessing module is used to clean, format convert and time synchronize the acquired data; a feature extraction module is used to extract features related to safety evaluation of the forced landing point from the preprocessed environmental data, and the features include obstacle features, battery power features, battery power consumption speed features, ground flatness features and landing point area features; and a deep reinforcement learning evaluation module is constructed based on a deep deterministic policy gradient method, and includes an actor network and a critic network, the actor network generates forced landing point selection and landing strategy actions according to a current state, and the critic network evaluates a value function of an action-state pair, and continuously optimizes the strategy through interaction with the environment and training to realize safety evaluation of the forced landing point.

[0012] The application has the advantages that: the application acquires data by means of multi-sensor fusion, comprehensively considers key information such as the state of the aircraft itself, the working condition of the battery and the geographical features of the landing area, and the model can accurately and meticulously evaluate the safety of the forced landing point of the unmanned aerial vehicle. Compared with the traditional single-index evaluation method, the application breaks away from the limitations of one-sided judgment, the deep reinforcement learning algorithm excavates the complex correlation between data, accurately locks the safe forced landing point that meets the actual flight condition, greatly reduces the misjudgment rate, and improves the evaluation reliability. The application enables the unmanned aerial vehicle to autonomously make decisions in an emergency, automatically drives to a safe forced landing point according to the evaluation result of the model, and enables ordinary users to easily control the unmanned aerial vehicle, reduces the operation training cost, and improves the automation level of emergency disposal of the unmanned aerial vehicle. Through the above specific embodiments, the modules closely cooperate, fully exert the advantages of the end-to-end forced landing point decision method of the unmanned aerial vehicle based on deep reinforcement learning under multiple constraint conditions, accurately and efficiently find a safe forced landing point for the unmanned aerial vehicle, and effectively ensure the flight safety of the unmanned aerial vehicle.

[0013] The features and advantages of the application will be described in detail with reference to the embodiments combined with the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of an end-to-end forced landing point decision method of an unmanned aerial vehicle based on deep reinforcement learning under multiple constraint conditions provided by the application.

[0015] Figure 2 is a technical route schematic diagram of an end-to-end forced landing point decision method of an unmanned aerial vehicle based on deep reinforcement learning under multiple constraint conditions provided by the application.

[0016] Figure 3 is the architecture diagram of a deep deterministic policy gradient (DDPG).

[0017] Figure 4 is the reward function design diagram of an end-to-end unmanned aerial vehicle forced landing point decision method based on deep reinforcement learning under multiple constraint conditions provided by the application. DETAILED DESCRIPTION

[0018] In order to make the object, technical scheme and advantages of the present application more clear and obvious, the present application is further described in detail below by means of drawings and examples. However, it should be understood that the specific examples described herein are only used to explain the present application and do not limit the scope of the present application. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.

[0019] Unmanned aerial vehicles inevitably encounter various sudden conditions when flying in complex and changeable natural environments and human activity areas. Once trapped in these emergency situations, the unmanned aerial vehicle urgently needs to quickly lock a safe and reliable forced landing point and land smoothly to avoid equipment damage, data loss, and even serious consequences such as ground personnel injury and property loss. Traditional unmanned aerial vehicle forced landing schemes mostly rely on the experience judgment of operators or preset fixed rules, and such methods have inherent limitations. In the face of complex topography, it is difficult to accurately control the ground flatness and obstacle distribution only by naked eye and experience; when encountering an emergency power crisis, it is impossible to quickly weigh the flight distance and forced landing point adaptability; in the scene where the weather conditions change rapidly, it is more difficult to make the optimal decision by comprehensively considering multiple factors.

[0020] With the rapid development of artificial intelligence technology, deep reinforcement learning, as a promising branch, has made a breakthrough in many fields and shown its excellent ability to solve complex decision-making problems. Its core is that the agent learns and optimizes the strategy autonomously according to the reward feedback by constantly interacting with the environment, which is theoretically very suitable for the evaluation needs of unmanned aerial vehicle forced landing points.

[0021] As shown in Figure 1 , the present application provides an end-to-end unmanned aerial vehicle forced landing point decision method based on deep reinforcement learning under multiple constraint conditions, which utilizes the end-to-end characteristics of deep reinforcement learning to realize the safety evaluation of unmanned aerial vehicle forced landing points under multiple constraint conditions.

[0022] As shown in Figure 2 , the method comprises the following steps: Step S100, system initialization: Before the UAV takes off, the sensor module, data preprocessing module, feature extraction module, deep reinforcement learning evaluation module and evaluation result output module are initialized and set. The pre-trained deep reinforcement learning model parameters are loaded, and it is ensured that each sensor is working normally and is calibrated.

[0023] Step S200, data acquisition and processing during flight: During the flight of the UAV, the sensor module continuously acquires various data, including two-dimensional image data, radar point cloud data, satellite positioning information, battery state information, flight state information, and weather information.

[0024] After data acquisition, the data is immediately transmitted to the data preprocessing module for format unification, cleaning and time synchronization processing. The processed data is transmitted to the feature extraction module.

[0025] Step S300, feature extraction: The feature extraction module extracts terrain distribution features, obstacle features, battery power features, power consumption speed features, maximum flight distance features, ground flatness features, landing point area features, and weather features from the preprocessed data, and combines these features into a state vector required by the deep reinforcement learning model.

[0026] Step S400, as Figure 3 shown, in an implementation mode, the DDPG design includes: The input layer is designed to adapt to multi-source feature input. According to the output dimension of the feature extraction module, the number of neurons is reasonably determined to accurately receive multi-element information such as terrain distribution features, obstacle features (type, location, size and distribution density, etc.), battery features (power, power consumption rate, health status), ground features (flatness index, area size), flight distance features (straight line and actual reachable distance), etc. The input data is normalized to the interval to ensure that the weights of each feature are balanced in network training, avoid learning bias caused by dimensional differences, and lay a precise data foundation for subsequent network decision-making.

[0027] The output layer single neuron accurately outputs the action-state pair value estimation value Q, which is initially randomly set and iteratively approximated to the true value after training. This value is used as a core index for policy evaluation, closely related to reward feedback, and drives the actor network policy improvement. Through Q dynamic feedback, the actor network is guided to explore higher value policy space, balancing the safety and efficiency of forced landing, and realizing intelligent and efficient forced landing decision optimization of the UAV under multiple constraint conditions.

[0028] Step S500, reward function design: As Figure 4 shown, in an implementation mode, a comprehensive reward function is constructed according to the key elements of forced landing: Each sub-reward function is designed to closely around the corresponding element calculation rules as follows: Step S510, obtaining terrain type sub-reward : Wherein, T is the terrain type (such as grass, sand, rock, water, etc.) of the candidate forced landing point, is the reward value of the corresponding safe terrain type ({ } is the set of safe terrain types, { } is the set of dangerous terrain types. For example, for grass terrain, higher, because the grass is relatively soft, has a certain buffer effect on the landing impact; and for water terrain, 0, because the water landing risk is greater), different rewards are given according to the terrain type to guide the unmanned aerial vehicle to preferentially select safe terrain for forced landing.

[0029] Step S520, obtaining obstacle distance : First, calculate the distance between the unmanned aerial vehicle and the nearest obstacle When ( is a pre-set safety distance threshold), =1, indicating that outside the safety distance, no obstacle risk reward.

[0030] When ( is the minimum acceptable distance, close to 0), a linearly decreasing reward function is adopted: When , , indicating that it is in the state of close to the obstacle in extreme danger, the lowest reward is given.

[0031] Step S530, obtaining battery power : Let the current battery power be B, and the full power be When (critical power), When , (k is an adjustment coefficient).

[0032] Step S540, obtain the battery power consumption rate : Calculate the power consumption rate of the battery in a period of time Set the normal power consumption rate threshold to When , =1 When , = Step S550, obtain the landing point area : Set the available area of the emergency landing point to A, the minimum safety area threshold to , and the maximum acceptable emergency landing area to When A≥ , When A< , Step S560, obtain the ground flatness : Use sensors such as laser radar to obtain the height of each point in the landing point area (such as a circular area with the landing point as the center and a radius of R) relative to a certain reference plane (such as the average sea level or a set local reference plane) ( is the number of sampling points in the area).

[0033] Calculate the arithmetic mean of these height values Then calculate the integer value of each sampling point from the mean , and then calculate the mean of these difference values When is smaller, it indicates that the ground height changes relatively small, the ground is more flat, and a higher reward is given. For example, set the reward function as where is the maximum acceptable height difference mean set in advance (determined according to the requirements of the unmanned aerial vehicle landing and the accuracy of the sensor), when =0, =1, indicating complete flatness; when = ,​ =0. Step S570, obtaining the maximum flight distance : The power consumption rate r (unit: mAh / min) reflects the rate of battery power consumption over time. During flight, the battery power will gradually decrease over time, and the flight distance will also be limited accordingly. When the battery power Q and the power consumption rate r are known, the maximum flight distance can be estimated by calculating the time t required for the battery power to be depleted t = Q / r, and then combining the average flight distance v of the UAV in unit time (unit: km / min). Let the maximum flight distance estimated by the UAV according to the current battery power Q and the power consumption rate r be , the flight distance from the current position to the candidate landing point be d, and the relative flight distance ratio be This ratio reflects the relative ease of reaching the landing point with the remaining power, and the larger the value of p, the easier it is to reach the landing point, and the more sufficient the remaining power reserve is.

[0034] Based on the above ratio, the sub-reward function is constructed , for example, in the form of a linear function: . Where k is a proportionality coefficient, used to adjust the contribution of this sub-reward in the overall reward. When p = 1 (i.e. d = 0, the UAV has already landed at the landing point), , indicating the maximum reward when reaching the landing point with sufficient power; when p = 0 (i.e. d = , reaching the landing point just by depleting the power), r = 0, no reward; when p < 0 (i.e. d > , unable to reach the landing point with the existing power), r < 0, giving a penalty to prompt the UAV to avoid selecting a landing point that cannot be reached.

[0035] Step S580, obtaining the meteorological condition sub-reward : Where is the wind speed, is the maximum tolerable wind speed, is the temperature deviation (difference from the optimal working temperature of the UAV), is the maximum tolerable temperature deviation, is the humidity, is the maximum tolerable humidity, , and are weight coefficients, and + + ​​= 1. This sub-reward function takes into account the influence of meteorological factors such as wind speed, temperature and humidity on forced landing safety. The smaller the wind speed, the closer the temperature and humidity to the appropriate range, the higher the reward, because such meteorological conditions are more conducive to the stable flight and safe landing of the UAV.

[0036] With the above carefully designed reward function, the deep reinforcement learning model can obtain corresponding reward feedback according to different states and actions in the interaction process with the environment, thereby gradually learning the optimal forced landing strategy, and realizing safe and efficient forced landing point selection and landing operation under the premise of meeting multiple constraint conditions.

[0037] Step S600, in the present application, the adaptive adjustment module comprises: During flight, the changes of flight conditions are monitored in real time, and the farthest flight distance estimate value is adaptively adjusted according to the changes. The influence of wind resistance on flight energy consumption is calculated by using meteorological data and combining the pre-established wind resistance model, and then the farthest flight distance estimate value is corrected. At the same time, the air density related energy consumption parameters are adjusted according to the flight height data, so as to improve the estimation accuracy. Through these adaptive adjustment measures, the influence of flight condition changes on estimation error can be reduced to a certain extent, so that the farthest flight distance estimate is more in line with the actual flight situation.

[0038] If it is detected that the battery power consumption speed abnormally accelerates (exceeds a certain percentage of the normal power consumption speed threshold), the power consumption speed reward weight is timely increased, and the weight coefficients related to flight distance and obstacle distance are reasonably adjusted according to the current flight scene and other sensor information, so as to prompt the UAV to change the flight strategy as soon as possible and approach the forced landing point that is more energy-saving and has no collision risk, in order to cope with the sudden high energy consumption of the battery.

[0039] By using the scanning of the ground features by the laser radar and the presentation of the terrain and topography by the camera image, the complexity of the current terrain is judged (for example, by comprehensively measuring the ground flatness index and terrain undulation). When in a complex terrain environment (such as mountainous area, hilly area, etc.), the weight coefficients of the ground flatness sub-reward and the landing point area sub-reward are increased, so as to guide the UAV to find a relatively flat and large enough forced landing area, and to ensure the safety of landing; if flying in a flat and simple terrain area, the two weights can be correspondingly adjusted to be lower, so that the UAV can comprehensively consider more other factors when selecting the forced landing point, and realize more flexible decision-making.

[0040] To sum up, the end-to-end unmanned aerial vehicle forced landing point decision method based on deep reinforcement learning under multiple constraint conditions provided by the application, the method comprises: a data acquisition module, a camera captures high-definition images to present environmental details, a laser radar accurately maps three-dimensional environmental information, a satellite positioning signal provides accurate coordinates, a meteorological sensor obtains temperature, humidity, wind speed and wind direction information, a battery management system monitors the power and power consumption speed in real time, and multiple source data cooperates to lay a solid foundation for the subsequent process; the data preprocessing module uses advanced algorithms to efficiently eliminate noise, calibrate drift, unify format and synchronize time, so that the data is neat and orderly, meets the model input requirements and avoids misjudgment risks; the feature extraction module deeply digs the data value, accurately extracts the features of obstacles, batteries, ground and flight distance, generates a highly representative feature vector, and helps the model quickly understand the environmental situation.

[0041] The core deep reinforcement learning evaluation module (based on DDPG) reasonably defines the state and action space, and is suitable for flight reality; the reward function skillfully balances multiple key factors to guide the agent to learn the optimal strategy; the actor and critic networks are iteratively optimized under the cooperation of experience replay and target network, which promotes the strategy to converge to the best. The application breaks away from the limitations of traditional experience judgment and fixed rules, can flexibly cope with complex scenes such as mountains, cities, bad weather and power crisis, quickly lock the safe forced landing point, and greatly improve the forced landing success rate and safety.

[0042] The above only describes the preferred embodiments of the application and is not intended to limit the application. Any modification, equivalent replacement or improvement made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. An end-to-end unmanned aerial vehicle (UAV) emergency landing point decision-making method based on deep reinforcement learning under multiple constraints, characterized in that, Specifically comprising the following steps: S1, initialization setting before the UAV takes off; S2, during the flight of the UAV, continuously collecting various data, including two-dimensional image data, radar point cloud data, satellite positioning information, battery state information, flight state information, weather information; and preprocessing the data; S3, extracting terrain distribution features, obstacle features, battery power features, power consumption speed features, maximum flight distance features, ground flatness features, landing point area features, and weather features from the preprocessed data, and combining the above features into a state vector required by a deep reinforcement learning model; S4, constructing a deep deterministic policy gradient method model and training the model, including an actor network and a critic network; the actor network is responsible for generating forced landing point selection and landing strategy actions according to the input state, and the critic network is used to evaluate the value of the action-state pair; the strategy is continuously optimized through interaction with the environment and training to realize safe evaluation of the forced landing point.

2. The method of claim 1, wherein the method is based on deep reinforcement learning under multiple constraints. The preprocessing in step S2 includes cleaning the collected various data to remove noise and outliers; format conversion to unify the data format; time synchronization processing. 3.The method of claim 1, wherein, Step S3 specifically includes the following operations: S31, terrain distribution features: using image recognition analysis technology, processing the ground pictures taken by the camera, classifying the pixels in the image into different terrain categories, and calculating the area proportion and distribution of each terrain in the image to extract the terrain distribution feature vector; S32, obstacle features: using visual image and laser radar data to identify the type, position, size and distribution of obstacles, and calculating the minimum distance between the aircraft and the obstacles and the obstacle density feature; S33, battery power feature: real-time acquisition of battery power information, calculation of remaining power percentage and power change trend feature; S34, battery power consumption speed feature: monitoring the change of battery power with time, calculating the battery power consumption per unit time feature; S35, maximum flight distance feature: obtaining the flight distance according to the battery state information obtained by the data acquisition module; S36, ground flatness feature: measuring the ground height change by laser radar and calculating the ground flatness index; S37, landing point area feature: determining the boundary of the potential landing point according to the visual image and laser radar data, and calculating the area size of the landing point; S38, weather feature: extracting features related to the flight performance of the UAV from the temperature and humidity data obtained by the weather sensor; according to the influence degree of different temperature and humidity intervals on the flight of the UAV, the temperature and humidity, wind speed and direction data are discretized, divided into different levels, and the corresponding levels are encoded as feature values, so that the model can consider the influence of temperature and humidity environmental factors on the forced landing safety.

4. The method of claim 1, wherein, The actor network has the following characteristics: The input layer receives the state vector generated by the feature extraction module, which contains obstacle features, battery power features, battery power consumption speed features, ground flatness features and landing point area feature information; The hidden layer adopts a multi-layer fully connected neural network structure, and the number of neurons in each layer is set according to actual conditions, and an activation function is used to increase the nonlinear expression capability of the network; The output layer outputs the forced landing point selection action, including the coordinate position of the forced landing point and the landing strategy action, such as the landing speed and angle output value, which is normalized to be within the action space range.

5. The method of claim 1, wherein, The critic network has the following characteristics: The input layer receives the splicing of the state vector and the action vector generated by the actor network, and the dimension is the sum of the dimensions of the state vector and the action vector; The hidden layer also adopts a multi-layer fully connected neural network structure, and the number of neurons in each layer is set according to actual conditions, and an activation function is used to increase the nonlinear expression capability of the network.

6. The output layer outputs a scalar value representing the value estimate of the current action-state pair, which is used to evaluate the quality of the policy generated by the actor network.

7. The method of claim 1, wherein, The training mechanism of the deep deterministic policy gradient method model includes: Experience replay: an experience replay buffer is established to store experience tuples of the interaction between the agent and the environment, and a batch of experiences is randomly sampled from the buffer for training; Target network update: a target network is established for the actor network and the critic network respectively, and the parameter update of the target network adopts a soft update method to stabilize the training process; Reward function design: a reward function is designed according to the characteristics of obstacles, battery capacity, battery power consumption speed, ground flatness and landing point area, which comprehensively considers the influence of various factors on the safety of the forced landing point; Training algorithm: based on the deep deterministic policy gradient method, the actor network updates the policy parameters by maximizing the value estimate given by the critic network, and the critic network updates the parameters by minimizing the mean square error between the predicted value and the target value, and the training process continues until the preset convergence condition is met.

8. An end-to-end unmanned aerial vehicle forced landing point decision device based on deep reinforcement learning under multiple constraints, characterized in that, The data acquisition module is used to collect multi-source data related to the forced landing point; two-dimensional image information is obtained through the camera; three-dimensional terrain information is obtained through the laser radar; accurate position data is obtained through the satellite positioning signal; detailed weather information such as temperature, humidity and wind force is obtained through the on-board temperature and humidity sensor and the weather sensor; battery capacity and power consumption speed data are obtained through the battery management system; The data preprocessing module cleans, formats and synchronizes the collected data; The feature extraction module is used to extract features related to the safety evaluation of the forced landing point from the preprocessed environmental data, including obstacle features, battery capacity features, battery power consumption speed features, ground flatness features and landing point area features; The deep reinforcement learning evaluation module is constructed based on the deep deterministic policy gradient method, including an actor network and a critic network, the actor network generates forced landing point selection and landing strategy actions according to the current state, and the critic network evaluates the value function of the action-state pair, and continuously optimizes the strategy through interaction with the environment and training to realize the safety evaluation of the forced landing point.

Images