Fire rescue unmanned aerial vehicle route automatic planning system based on AI identification

By integrating multi-sensor fusion, deep learning and reinforcement learning technologies in drones, real-time perception and dynamic route planning for complex fire rescue environments are achieved, and the problem of insufficient perception and obstacle avoidance capabilities of drones in complex environments is solved. , improves the security and execution efficiency of tasks.

CN119045508AInactive Publication Date: 2024-11-29NANJING BANGFEI TECHNOLOGY DEVELOPMENT CO LTD

Patent Information

Application Number
CN202411179284.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing drones have low perception capabilities in complex fire rescue environments, difficult to cope with real-time changes, lack of dynamicity and autonomy in route planning, and poor flexibility in obstacle avoidance strategies, resulting in low mission safety and efficiency.

Method used

Adopting AI-based multi-sensor fusion and environment perception module, deep learning environment identification module, adaptive route planning module, intelligent task allocation and scheduling module, self-organized communication and data transmission module, intelligent obstacle avoidance and dynamic adjustment module, AI control and Task execution module, fault prediction and emergency treatment module, combined with deep learning, reinforcement learning and SLAM technology, realize real-time environment perception, route planning and obstacle avoidance optimization.

Benefits of technology

It improves the perception and decision-making capabilities of drones in complex environments, realizes real-time adaptive planning and dynamic obstacle avoidance of routes, enhances the safety and execution efficiency of tasks, and reduces manual intervention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045508A_ABST
    Figure CN119045508A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of unmanned aerial vehicle autonomous navigation, and discloses a fire rescue unmanned aerial vehicle route automatic planning system based on AI identification. Comprising a multi-sensor fusion and environment perception module, a deep learning environment identification module, a self-adaptive route planning module, an intelligent task distribution and scheduling module, a self-organizing communication and data transmission module, an intelligent obstacle avoidance and dynamic adjustment module, an AI control and task execution module and a fault prediction and emergency processing module. And the multi-sensor fusion and environment perception module integrates multi-mode environment perception of a plurality of sensors on a fire scene. Through a deep learning technology of a convolutional neural network and principal component analysis, high-precision identification of key features in a complex scene of a fire scene is realized, and in a complex illumination condition and a smoke shielding environment, a deep learning model can automatically extract effective features, and the identification accuracy is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous navigation of unmanned aerial vehicles, and in particular to an automatic route planning system for firefighting and rescue unmanned aerial vehicles based on AI recognition. Background Art

[0002] With the development of drone technology, drones are widely used in various fields, especially in firefighting and rescue. Drones can quickly reach the disaster site, provide real-time monitoring from a high altitude perspective, and assist firefighters in decision-making and rescue work. However, the existing drone technology still has many shortcomings in complex firefighting and rescue environments as follows:

[0003] Traditional drones rely on simple sensors and rule-based algorithms to perceive the environment. They have poor recognition effects when faced with the complex and changing environment of a fire scene. In addition, they lack sufficient complexity and are unable to cope with real-time changing environments, resulting in low perception capabilities, which affects the decision-making and execution efficiency of drones.

[0004] Existing UAV route planning is mostly based on preset paths or simple global optimization algorithms, which lack adaptability to dynamic environments. If the situation at the fire scene changes, traditional route planning cannot adjust the path in real time, causing the UAV to enter a dangerous area, reducing the safety and effectiveness of the mission. In addition, traditional planning methods cannot autonomously learn and optimize path selection strategies, resulting in poor performance when dealing with complex environments.

[0005] Most current obstacle avoidance technologies rely on simple sensor perception and fixed-rule obstacle avoidance strategies. When faced with complex environments with dynamic obstacles or multiple obstacles, they react slowly and their obstacle avoidance strategies are inflexible, which can easily lead to collisions or frequent route adjustments, affecting flight stability and mission execution efficiency.

[0006] Therefore, those skilled in the art provide an automatic route planning system for firefighting and rescue drones based on AI recognition to solve the problems raised in the above background technology. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides an automatic route planning system for firefighting and rescue drones based on AI recognition to solve the problems raised in the above background technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: an automatic route planning system for fire rescue drones based on AI recognition, including a multi-sensor fusion and environment perception module, a deep learning environment recognition module, an adaptive route planning module, an intelligent task allocation and scheduling module, a self-organizing communication and data transmission module, an intelligent obstacle avoidance and dynamic adjustment module, an AI control and task execution module, and a fault prediction and emergency processing module;

[0009] The multi-sensor fusion and environment perception module integrates the multi-modal environment perception of the fire scene by multiple sensors, and integrates the inputs from different sensors into a unified environment model through data fusion technology;

[0010] The deep learning environment recognition module processes the environmental data provided by the sensor based on the trained deep learning model to identify key features in the fire scene;

[0011] The adaptive route planning module combines the information provided by the environment recognition module and uses hierarchical planning and reinforcement learning algorithms to perform real-time route planning;

[0012] The intelligent task allocation and scheduling module intelligently allocates and schedules tasks for multiple drones based on real-time environmental data and task requirements;

[0013] The self-organizing communication and data transmission module establishes a self-organizing network between drones to achieve direct communication and data sharing between drones;

[0014] The intelligent obstacle avoidance and dynamic adjustment module combines deep learning and SLAM technology to realize the perception and prediction of dynamic obstacles, provide intelligent obstacle avoidance solutions, and adjust the route in real time during flight to avoid new obstacles or dangerous areas;

[0015] The AI ​​control and task execution module performs fine control of the drone based on the AI ​​algorithm, provides real-time task feedback, supports instant adjustment and optimization during task execution, and automatically starts the backup task plan or switches to other drones to continue the task when the task is interrupted;

[0016] The fault prediction and emergency handling module is based on a machine learning fault prediction model to identify and warn of drone faults in advance, provide a multi-level emergency handling mechanism, and conduct emergency rehearsals through virtual simulation technology to optimize emergency handling strategies.

[0017] Preferably, the multi-sensor fusion and environment perception module includes a sensor array unit, a data acquisition unit, and a data fusion unit; the sensor array unit is responsible for collecting various environmental data at the fire scene; the data acquisition unit is responsible for acquiring real-time data from each sensor and performing preliminary preprocessing; the data fusion unit integrates data from different sensors into a unified environmental model through a fusion algorithm to provide comprehensive environmental information;

[0018] The deep learning environment recognition module includes an image processing unit, a recognition model unit, and a feature extraction unit; the image processing unit is responsible for processing the image data collected by the sensor; the recognition model unit uses a pre-trained deep learning model to identify key features in the fire scene; the feature extraction unit extracts valuable feature information from the processed image or other data for analysis and decision-making by other modules;

[0019] The adaptive route planning module includes a global planning unit, a local adjustment unit, and a reinforcement learning unit; the global planning unit is responsible for global route planning according to the overall mission requirements and environmental information; the local adjustment unit dynamically adjusts the local flight path of the drone based on real-time environmental changes; the reinforcement learning unit uses an adaptive learning algorithm to allow the drone to optimize the route planning strategy based on flight experience;

[0020] The intelligent task allocation and scheduling module includes a task priority evaluation unit, a drone status monitoring unit, and a task scheduling unit; the task priority evaluation unit prioritizes different tasks according to task urgency, importance, and resource conditions, and determines the order of task execution; the drone status monitoring unit monitors the status of each drone in real time; the task scheduling unit intelligently allocates tasks according to task priority and drone status, and coordinates the task execution of multiple drones to avoid conflicts.

[0021] Preferably, the self-organizing communication and data transmission module includes a communication network unit, a data transmission optimization unit, and a 5G / satellite communication unit; the communication network unit establishes a self-organizing network between drones to support direct communication and data sharing between drones; the data transmission optimization unit is responsible for managing and optimizing the data transmission path; the 5G / satellite communication unit provides a high-bandwidth, low-latency communication link;

[0022] The intelligent obstacle avoidance and dynamic adjustment module includes a dynamic perception unit, an obstacle avoidance decision unit, and a dynamic adjustment unit; the dynamic perception unit monitors dynamic obstacles during flight in real time; the obstacle avoidance decision unit calculates the optimal obstacle avoidance path based on environmental perception and SLAM technology, and performs obstacle avoidance operations; the dynamic adjustment unit dynamically adjusts the flight path of the drone according to real-time environmental changes and mission requirements.

[0023] Preferably, the AI ​​control and task execution module includes a precision control unit, a real-time feedback unit, and a task recovery unit; the precision control unit is responsible for the precise control operation of the UAV; the real-time feedback unit collects and analyzes the feedback data generated during the task execution process, and provides it to the control system for immediate adjustment and optimization; if the task is interrupted for some reason, the task recovery unit automatically plans an alternative plan or switches to a backup UAV to continue the task;

[0024] The fault prediction and emergency handling module includes a fault monitoring unit, a prediction and analysis unit, and an emergency handling unit; the fault monitoring unit monitors various key parameters of the drone in real time and identifies potential faults; the prediction and analysis unit uses a machine learning model to analyze historical data and real-time monitoring data, predict faults and send out warning signals; the emergency handling unit starts the emergency handling procedure when a fault or abnormal situation is detected.

[0025] Preferably, the data acquisition unit uses Kalman filtering to process sensor noise when collecting data, so as to perform smoothing in the time series data:

[0026] State prediction formula:

[0027] x k|k-1 =Fx k-1|k-1 +Bu k ,

[0028] Covariance prediction:

[0029] P k|k-1 =FP k-1|k-1 F T +Q,

[0030] Kalman gain:

[0031] K k =P k|k-1 H T (HP k|k-1 H T +R) -1 ,

[0032] Status Update:

[0033] x k|k =x k|k-1 +K k (z k -Hx k|k-1 ),

[0034] Covariance update:

[0035] P k|k =(IK k H)P k|k-1 ,

[0036] Among them, x k|k-1 is the predicted state vector at time k, P k|k-1 is the prediction of the state covariance matrix, F is the state transfer matrix, H is the measurement matrix, K k is the Kalman gain, z k is the measurement value, Q and R are the process noise covariance and measurement noise covariance;

[0037] The data fusion unit uses extended Kalman filtering to fuse nonlinear systems in multi-sensor data fusion:

[0038] Nonlinear state equation:

[0039] x k =f(x k-1 ,u k )+w k ,

[0040] Nonlinear observation equation:

[0041] z k =h(x k )+v k ,

[0042] Among them, f(.) and h(.) are the nonlinear state transfer function and observation function of the system, w k and v k are process noise and observation noise.

[0043] Preferably, the image processing unit includes the following related formula:

[0044] Image convolution operation:

[0045]

[0046] Among them, I(x,y) is the intensity value of the original image at the pixel position (x,y), K(i,j) is the convolution kernel, and I filtered (x,y) is the intensity value of the filtered image at pixel location (x,y).

[0047] Histogram Equalization:

[0048]

[0049] Where CDF(i) is the cumulative distribution function, P(j) is the probability density function of gray level j, M×N is the total number of pixels in the image, L is the number of gray levels in the image, and I equalized (x,y) is the image after equalization;

[0050] In the recognition model unit, the deep learning model is usually a convolutional neural network, and its operation steps are as follows:

[0051] Convolutional Layer:

[0052] h i,j,k =f(∑ m,n,l W m,n,l,k ·x i+m,j+n,l +b k ),

[0053] Among them, x i,j,l is the pixel value of the input feature map, l represents the number of channels, and W m,n,l,k is the convolution kernel weight, k represents the output channel index, b k is the bias, f(.) is the activation function, h i,j,k is the pixel value of the output feature map;

[0054] h i,j,k =max{x 2i,2j,k ,x 2i+1,2j,k ,x 2i,2j+1,k ,x 2i+1,2j+1,k},

[0055] Reduce the size of the feature map through pooling operations while retaining important information;

[0056] Fully connected layer:

[0057] z i =f(∑ j W i,j ·h j +b i ),

[0058] Among them, h j is the output of the previous layer, W i,j is the weight matrix of the fully connected layer, b i is the bias, z i is the output of the fully connected layer.

[0059] Preferably, the local adjustment unit may use a gradient descent method to optimize the path when adjusting the flight path in real time, so as to avoid obstacles or adjust the flight altitude;

[0060] Gradient descent update rule:

[0061]

[0062] Among them, x k is the coordinate vector of the path point, α is the learning rate, is the gradient of the path cost function f(x) with respect to the path point;

[0063] Path cost function:

[0064]

[0065] The first term represents the smoothness of the path, minimizing the distance difference between path points, and the second term represents the distance cost to the obstacle, d(x j ,o j ) is the path point x j With obstacles j The distance between them is short, but the cost is high.

[0066] Preferably, in the 5G / satellite communication unit, power control and resource allocation are key issues. The following is a resource allocation algorithm based on Lagrangian duality theory:

[0067] Objective function: maximize the total rate of the system and meet the power and bandwidth constraints:

[0068]

[0069] Among them, P i is the transmission power of the ith user, |h i | 2 is the square of the modulus of the channel gain, σ 2 is the noise power, I i is the interference power;

[0070] Constraints:

[0071] a. Total power constraint:

[0072]

[0073] Among them, P total is the total power budget of the system;

[0074] b. Bandwidth allocation for each user:

[0075] B i ≤B max ,

[0076] Preferably, the dynamic adjustment unit uses model predictive control to adjust the trajectory of the drone in real time to ensure stable flight during obstacle avoidance;

[0077] State space model: The motion model of the drone can be expressed in discrete time state space form:

[0078] x k+1 =Ax k +Bu k ,

[0079] Among them, x k is the state vector at time k, u k is the control input at time k, the A and B system matrices;

[0080] Cost function: MPC optimizes the control input by minimizing the following cost function:

[0081]

[0082] Among them, Q and R are the weight matrices of state and control input, Qf is the weight matrix of the terminal state;

[0083] Constraints: During the optimization process, MPC usually has to meet the following constraints:

[0084] x min ≤x k ≤x max ,

[0085] u min ≤u k ≤u max ,

[0086] Among them, x min and x max are the upper and lower limits of the state, u min and u max are the upper and lower limits of the control input.

[0087] Preferably, the task recovery unit is based on task recovery of a partially observable Markov decision process, and the task recovery unit uses Markov to decide the optimal recovery strategy when task execution is interrupted;

[0088] Markov decision model: Markov is represented by state s, action a, and observation o, where state s is partially observable;

[0089] Bayesian updating: based on observations o k , update the posterior distribution of the state:

[0090] b k+1 (s′)=η·P(o k+1 |s′)∑ s∈S P(s′|s,a k )b k (s),

[0091] Among them, b k (s) is the state belief at time k, P(o k+1 |s′) is the observation model, P(s′|s,a k ) is a state transition model, that is, executing action a in state s k The probability of transferring to state s′, η is the normalization factor;

[0092] Value function: The value function is used to evaluate the optimal strategy under each belief state b, with the goal of maximizing long-term returns:

[0093]

[0094] Among them, R(b k ,a k ) is in belief state bk Next, perform action a k is the immediate return, and γ is the discount factor.

[0095] The present invention provides an automatic route planning system for firefighting and rescue UAVs based on AI recognition. It has the following beneficial effects:

[0096] 1. The present invention uses deep learning technology of convolutional neural network and principal component analysis to achieve high-precision recognition of key features in complex scenes at fire scenes. In addition, under complex lighting conditions and smoke-occluded environments, the deep learning model can automatically extract effective features and improve recognition accuracy. Compared with traditional recognition methods based on rules or simple machine learning, the deep learning environment recognition module significantly improves the perception ability of drones in changeable and complex environments, enabling drones to better perceive and understand the dynamic changes at the fire scene and make more accurate responses.

[0097] 2. The present invention realizes real-time adaptive planning of UAV routes by combining global planning, local adjustment and reinforcement learning algorithms, so that the UAV can continuously optimize its path selection strategy through autonomous learning and cope with dynamic changes in complex environments. Compared with traditional static or pre-set path planning methods, the adaptive route planning module can flexibly adjust the route according to real-time environmental information, avoid obstacles, flame spread and other emergencies, improve the efficiency and safety of mission execution, enhance the autonomy of the UAV, and enable it to have stronger adaptability and decision-making ability in changing rescue scenarios.

[0098] 3. The present invention adopts the advanced technologies of optical flow dynamic obstacle detection, potential field obstacle avoidance strategy and model predictive control to realize intelligent dynamic obstacle avoidance and path adjustment, ensuring that the UAV can perform optimal path adjustment in a dynamic environment and maintain flight stability and efficiency. Compared with traditional obstacle avoidance technology, it provides higher autonomous obstacle avoidance capability and path adjustment flexibility, can accurately avoid obstacles in complex dynamic environments, and adjust the flight path in real time to avoid possible dangers, thereby improving the safety of UAVs in actual rescue scenarios and the reliability of mission completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 is a system diagram of the present invention;

[0100] Figure 2 It is a schematic diagram of the multi-sensor fusion and environment perception module of the present invention;

[0101] Figure 3 A schematic diagram of a deep learning environment recognition module of the present invention;

[0102] Figure 4 It is a schematic diagram of an adaptive route planning module of the present invention;

[0103] Figure 5 This is a schematic diagram of the intelligent task allocation and scheduling module of the present invention;

[0104] Figure 6 It is a schematic diagram of the self-organizing communication and data transmission module of the present invention;

[0105] Figure 7 It is a schematic diagram of the intelligent obstacle avoidance and dynamic adjustment module of the present invention;

[0106] Figure 8 This is a schematic diagram of the AI ​​control and task execution module of the present invention;

[0107] Fig. 9 It is a schematic diagram of the fault prediction and emergency processing module of the present invention. DETAILED DESCRIPTION

[0108] In order to make the technical personnel in the technical field understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in combination with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a partial embodiment of the present invention, not a complete embodiment. Based on the embodiment of the present invention, other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0109] The present invention is described in detail below in conjunction with the accompanying drawings:

[0110] Example:

[0111] Please refer to the attached Figure 1 - Attachment Fig. 9The embodiment of the present invention provides an automatic route planning system for fire rescue drones based on AI recognition, including a multi-sensor fusion and environment perception module, a deep learning environment recognition module, an adaptive route planning module, an intelligent task allocation and scheduling module, a self-organizing communication and data transmission module, an intelligent obstacle avoidance and dynamic adjustment module, an AI control and task execution module, and a fault prediction and emergency processing module; the multi-sensor fusion and environment perception module integrates multi-sensors to perceive the multi-modal environment of the fire scene, and integrates the inputs from different sensors into a unified environmental model through data fusion technology; the deep learning environment recognition module processes the environmental data provided by the sensor based on the trained deep learning model and identifies the key features in the fire scene; the adaptive route planning module combines the information provided by the environment recognition module and uses hierarchical planning and reinforcement learning algorithms to perform real-time route planning; the intelligent task allocation and scheduling module performs real-time route planning based on the real-time environment The self-organizing communication and data transmission module establishes a self-organizing network between drones to achieve direct communication and data sharing between drones. The intelligent obstacle avoidance and dynamic adjustment module combines deep learning and SLAM technology to perceive and predict dynamic obstacles, provide intelligent obstacle avoidance solutions, and adjust the flight path in real time to avoid new obstacles or dangerous areas during flight. The AI ​​control and task execution module performs fine control of drones based on AI algorithms, provides real-time task feedback, supports instant adjustment and optimization during task execution, and automatically starts backup task plans or switches to other drones to continue the task when the task is interrupted. The fault prediction and emergency handling module uses a machine learning fault prediction model to identify and warn drone faults in advance, provide a multi-level emergency handling mechanism, and conduct emergency rehearsals through virtual simulation technology to optimize emergency handling strategies.

[0112] The multi-sensor fusion and environmental perception module can reduce the error caused by a single sensor, enhance the system's overall understanding of the environment and the accuracy of decision-making; the deep learning environment recognition module can adapt to the changing environment, identify and understand the dynamic changes of the fire scene in real time, and help the drone to respond more accurately; the adaptive route planning module can learn and optimize its own path selection strategy, improve its autonomous decision-making ability in complex environments, and reduce its dependence on human intervention; the intelligent task allocation and scheduling module can reasonably allocate the resources of the drone, avoid resource waste, and improve the overall efficiency of the system; the self-organizing communication and data transmission module can quickly transmit important information between drones, reduce communication delays, improve the efficiency and reliability of data transmission, and ensure the continuity of task execution; the intelligent obstacle avoidance and dynamic adjustment module can adjust the route in real time according to obstacles or dangerous areas to avoid collisions, ensure the safety and smooth completion of the task; the AI ​​control and task execution module can automatically start the backup plan or switch to other drones to continue the task, ensure the continuity and final completion of the task, and improve the reliability of the system; the fault prediction and emergency handling module can improve the system's response ability and handling efficiency in emergency situations.

[0113] The multi-sensor fusion and environmental perception module includes a sensor array unit, a data acquisition unit, and a data fusion unit; the sensor array unit is responsible for collecting various environmental data at the fire scene; the data acquisition unit is responsible for obtaining real-time data from each sensor and performing preliminary preprocessing; the data fusion unit integrates data from different sensors into a unified environmental model through a fusion algorithm to provide comprehensive environmental information; the deep learning environmental recognition module includes an image processing unit, a recognition model unit, and a feature extraction unit; the image processing unit is responsible for processing the image data collected by the sensor; the recognition model unit uses a pre-trained deep learning model to identify key features at the fire scene; the feature extraction unit extracts valuable features from the processed image or other data The module collects information for analysis and decision-making by other modules; the adaptive route planning module includes a global planning unit, a local adjustment unit, and a reinforcement learning unit; the global planning unit is responsible for global route planning based on overall mission requirements and environmental information; the local adjustment unit dynamically adjusts the local flight path of the drone based on real-time environmental changes; the reinforcement learning unit uses an adaptive learning algorithm to allow the drone to optimize the route planning strategy based on flight experience; the intelligent task allocation and scheduling module includes a task priority evaluation unit, a drone status monitoring unit, and a task scheduling unit; the task priority evaluation unit prioritizes different tasks according to the urgency, importance, and resource conditions of the task, and determines the order of task execution; the drone status monitoring unit monitors each drone in real time The task scheduling unit intelligently allocates tasks according to the task priority and the status of the drone, and coordinates the execution of tasks of multiple drones to avoid conflicts. The self-organizing communication and data transmission module includes a communication network unit, a data transmission optimization unit, and a 5G / satellite communication unit. The communication network unit establishes a self-organizing network between drones and supports direct communication and data sharing between drones. The data transmission optimization unit is responsible for managing and optimizing the data transmission path. The 5G / satellite communication unit provides a high-bandwidth, low-latency communication link. The intelligent obstacle avoidance and dynamic adjustment module includes a dynamic perception unit, an obstacle avoidance decision unit, and a dynamic adjustment unit. The dynamic perception unit monitors dynamic obstacles in real time during flight. The obstacle avoidance decision unit is based on environmental perception and SLAM technology. The system calculates the optimal obstacle avoidance path and performs obstacle avoidance operations; the dynamic adjustment unit dynamically adjusts the flight path of the drone according to real-time environmental changes and mission requirements; the AI ​​control and mission execution module includes a fine control unit, a real-time feedback unit, and a mission recovery unit; the fine control unit is responsible for the precise control operation of the drone; the real-time feedback unit collects and analyzes the feedback data generated during the mission execution process and provides it to the control system for immediate adjustment and optimization; if the mission is interrupted for some reason, the mission recovery unit automatically plans an alternative plan or switches to a backup drone to continue the mission; the fault prediction and emergency handling module includes a fault monitoring unit, a prediction analysis unit, and an emergency handling unit; the fault monitoring unit monitors the key parameters of the drone in real time and identifies potential faults;The prediction and analysis unit uses machine learning models to analyze historical data and real-time monitoring data, predict faults and send out warning signals; the emergency processing unit initiates the emergency processing procedure when a fault or abnormal situation is detected. ;

[0114] The benefits of the sensor array unit are to provide more comprehensive environmental perception and enhance the drone's perception of complex scenes; the benefits of the data acquisition unit are to ensure the quality and reliability of the data, provide high-quality input for subsequent data fusion and environmental recognition, and improve the overall performance of the system; the benefits of the data fusion unit are to improve the accuracy and comprehensiveness of environmental information and support more accurate decision-making and planning; the benefits of the image processing unit are to provide clear and optimized input for subsequent deep learning recognition, which helps to improve the accuracy and efficiency of recognition; the benefits of the recognition model unit are to provide accurate environmental analysis and support drones to make correct responses in complex environments; the benefits of the feature extraction unit enable the system to quickly obtain key data in complex environments and improve overall decision-making efficiency; the benefits of the global planning unit are to improve the efficiency and safety of the mission and reduce uncertainty during flight; the benefits of the local adjustment unit are to ensure that the drone can flexibly respond to emergencies and improve the adaptability of the system; the benefits of the reinforcement learning unit are to improve the autonomous decision-making ability and efficiency of the drone in complex environments , reduce manual intervention, and improve the intelligence level of the system; the benefits of the task priority evaluation unit are to improve the rationality and effectiveness of task scheduling, and maximize the success rate and efficiency of task completion; the benefits of the drone status monitoring unit are to ensure that task allocation is in line with the actual status of the drone, and to avoid task failures due to poor drone status; the benefits of the task scheduling unit are to avoid task conflicts and waste of resources, and improve the efficiency and coordination of overall task execution; the benefits of the communication network unit are to improve the robustness of the system and task continuity; the benefits of the data transmission optimization unit are to ensure the efficiency and reliability of data transmission, and reduce data loss and delay; the benefits of the 5G / satellite communication unit are to ensure smooth communication between the drone and the ground control center and other drones; the benefits of the dynamic perception unit are to ensure that the drone can quickly identify and respond in complex environments, and improve flight safety; the benefits of the obstacle avoidance decision unit are to ensure that the drone can fly safely in complex and dynamic environments, and avoid collisions or accidents; the benefits of the dynamic adjustment unit are to improve the execution efficiency and success rate of the task.

[0115] The data acquisition unit, when collecting data, uses Kalman filtering to deal with sensor noise to perform smoothing in time series data:

[0116] State prediction formula:

[0117] x k|k-1 =Fx k-1|k-1 +Bu k ,

[0118] Covariance prediction:

[0119] P k|k-1 =FP k-1|k-1 F T +Q,

[0120] Kalman gain:

[0121] K k =P k|k-1 H T (HP k|k-1 H T +R) -1 ,

[0122] Status Update:

[0123] x k|k =x k|k-1 +K k (z k -Hx k|k-1 ),

[0124] Covariance update:

[0125] P k|k =(IK k H)P k|k-1 ,

[0126] Among them, x k|k-1 is the predicted state vector at time k, P k|k-1 is the prediction of the state covariance matrix, F is the state transfer matrix, H is the measurement matrix, K k is the Kalman gain, z k is the measurement value, Q and R are the process noise covariance and measurement noise covariance;

[0127] The data fusion unit uses the extended Kalman filter to fuse nonlinear systems in multi-sensor data fusion:

[0128] Nonlinear state equation:

[0129] x k =f(x k-1 ,u k )+w k ,

[0130] Nonlinear observation equation:

[0131] z k =h(x k )+v k ,

[0132] Among them, f(.) and h(.) are the nonlinear state transfer function and observation function of the system, w k and v k are process noise and observation noise.

[0133] Kalman filtering smoothes the time series of sensor data through a series of prediction and update steps, reduces the impact of noise, and improves the accuracy and reliability of state estimation. It is used for linear systems.

[0134] The extended Kalman filter is used for nonlinear systems. It linearizes the nonlinear state and observation equations, combines the prediction and update processes, and fuses multi-sensor data to provide more accurate state estimation. It is an extended version of the standard Kalman filter and is used to deal with nonlinear situations in real systems.

[0135] The image processing unit includes the following related formulas:

[0136] Image convolution operation:

[0137]

[0138] Among them, I(x,y) is the intensity value of the original image at the pixel position (x,y), K(i,j) is the convolution kernel, and I filtered (x,y) is the intensity value of the filtered image at pixel location (x,y).

[0139] Histogram Equalization:

[0140]

[0141] Where CDF(i) is the cumulative distribution function, P(j) is the probability density function of gray level j, M×N is the total number of pixels in the image, L is the number of gray levels in the image, and I equalized (x,y) is the image after equalization;

[0142] In the recognition model unit, the deep learning model is usually a convolutional neural network, and its operation steps are as follows:

[0143] Convolutional Layer:

[0144] h i,j,k =f(∑ m,n,l W m,n,l,k ·x i+m,j+n,l +b k ),

[0145] Among them, x i,j,l is the pixel value of the input feature map, l represents the number of channels, and W m,n,l,k is the convolution kernel weight, k represents the output channel index, b k is the bias, f(.) is the activation function, h i,j,k is the pixel value of the output feature map;

[0146] h i,j,k =max{x 2i,2j,k,x 2i+1,2j,k ,x 2i,2j+1,k ,x 2i+1,2j+1,k},

[0147] Reduce the size of the feature map through pooling operations while retaining important information;

[0148] Fully connected layer:

[0149] z i =f(∑ j W i,j ·h j +b i ),

[0150] Among them, h j is the output of the previous layer, W i,j is the weight matrix of the fully connected layer, b i is the bias, z i is the output of the fully connected layer.

[0151] Image convolution operation and histogram equalization are key steps in image processing. They are used for feature extraction and image contrast enhancement. They play a vital role in image preprocessing to ensure the quality and usability of input images.

[0152] Convolutional layers, pooling layers, and fully connected layers are the basic components of convolutional neural networks, each responsible for feature extraction, feature simplification, and feature classification. Through the combination of layers, convolutional neural networks can automatically learn to extract multi-level feature information from input images, thereby achieving efficient image recognition and classification tasks.

[0153] The local adjustment unit can use the gradient descent method to optimize the path when adjusting the route in real time, so that it avoids obstacles or adjusts the flight altitude;

[0154] Gradient descent update rule:

[0155]

[0156] Among them, x k is the coordinate vector of the path point, α is the learning rate, is the gradient of the path cost function f(x) with respect to the path point;

[0157] Path cost function:

[0158]

[0159] The first term represents the smoothness of the path, minimizing the distance difference between path points, and the second term represents the distance cost to the obstacle, d(x j ,o j ) is the path point x j With obstaclesj The distance between them is short, but the cost is high.

[0160] The gradient descent update rule plays a key role in the path optimization process. It continuously iterates to improve the coordinates of the path points so that the path cost function is minimized and the flight path of the drone is optimized.

[0161] The path cost function provides a clear goal for path optimization, taking into account the smoothness of the path and ensuring a safe distance between the drone and obstacles. By minimizing the function, the system can adjust the path in real time to ensure the safety and efficiency of the drone flight.

[0162] In 5G / satellite communication units, power control and resource allocation are key issues. The following is a resource allocation algorithm based on Lagrangian duality theory:

[0163] Objective function: maximize the total rate of the system and meet the power and bandwidth constraints:

[0164]

[0165] Among them, P i is the transmission power of the ith user, |h i | 2 is the square of the modulus of the channel gain, σ 2 is the noise power, I i is the interference power;

[0166] Constraints:

[0167] a. Total power constraint:

[0168]

[0169] Among them, P total is the total power budget of the system;

[0170] b. Bandwidth allocation for each user:

[0171] B i ≤B max ,

[0172] The objective function is to maximize the total data transmission rate of the system, which is the core goal of the resource allocation algorithm and aims to optimize communication performance. The total power constraint ensures that the system's transmission power operates within the predetermined power budget to avoid power overload or interference with other systems. The bandwidth allocation constraint of each user ensures that the bandwidth allocated to each user does not exceed its maximum allowed value, ensuring the rationality of resource allocation and the overall stability of the system.

[0173] The dynamic adjustment unit uses model predictive control to adjust the trajectory of the drone in real time to ensure stable flight during obstacle avoidance;

[0174] State space model: The motion model of the drone can be expressed in discrete time state space form:

[0175] x k+1 =Ax k +Bu k ,

[0176] Among them, x k is the state vector at time k, u k is the control input at time k, the A and B system matrices;

[0177] Cost function: MPC optimizes the control input by minimizing the following cost function:

[0178]

[0179] Among them, Q and R are the weight matrices of state and control input, Q f is the weight matrix of the terminal state;

[0180] Constraints: During the optimization process, MPC usually has to meet the following constraints:

[0181] x min ≤x k ≤x max ,

[0182] u min ≤u k ≤u max ,

[0183] Among them, x min and x max are the upper and lower limits of the state, u min and u max are the upper and lower limits of the control input.

[0184] The state-space model describes the dynamic behavior of the UAV, and MPC uses the model to predict future states and optimize control inputs to adjust the flight trajectory;

[0185] The cost function is the core of MPC optimization. By minimizing the cost function, MPC can balance multiple objectives and achieve optimal control.

[0186] Constraints ensure that the state and control input of the drone are within a safe and acceptable range during the optimization process, preventing violations of operational restrictions or physical limitations and ensuring the safety and stability of the flight process.

[0187] The task recovery unit is based on the partially observable Markov decision process for task recovery. When the task execution is interrupted, the task recovery unit uses Markov to decide the optimal recovery strategy;

[0188] Markov decision model: Markov is represented by state s, action a, and observation o, where state s is partially observable;

[0189] Bayesian updating: based on observations o k , update the posterior distribution of the state:

[0190] b k+1 (s′)=η·P(o k+1 |s′)∑ s∈S P(s′|s,a k )b k (s),

[0191] Among them, b k (s) is the state belief at time k, P(o k+1 |s′) is the observation model, P(s′|s,a k ) is a state transition model, that is, executing action a in state s k The probability of transferring to state s′, η is the normalization factor;

[0192] Value function: The value function is used to evaluate the optimal strategy under each belief state b, with the goal of maximizing long-term returns:

[0193]

[0194] Among them, R(b k ,a k ) is in belief state b k Next, perform action a k is the immediate return, and γ is the discount factor.

[0195] The Markov decision model provides a framework to describe the state evolution and decision-making process of the UAV after the mission is interrupted. The Bayesian update continuously corrects the system's estimate of the state by combining observation and state transition models, ensuring accurate state inference under uncertain conditions and providing a basis for subsequent decision-making. The value function helps the system select the optimal recovery strategy under the current belief state by maximizing long-term returns, ensuring that the mission can be successfully recovered in uncertain and complex environments.

[0196] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An automatic route planning system for fire rescue drones based on AI recognition, characterized in that: It includes multi-sensor fusion and environmental perception module, deep learning environment recognition module, adaptive route planning module, intelligent task allocation and scheduling module, self-organizing communication and data transmission module, intelligent obstacle avoidance and dynamic adjustment module, AI control and task execution module, fault prediction and emergency handling module; The multi-sensor fusion and environment perception module integrates the multi-modal environment perception of the fire scene by multiple sensors, and integrates the inputs from different sensors into a unified environment model through data fusion technology; The deep learning environment recognition module processes the environmental data provided by the sensor based on the trained deep learning model to identify key features in the fire scene; The adaptive route planning module combines the information provided by the environment recognition module and uses hierarchical planning and reinforcement learning algorithms to perform real-time route planning; The intelligent task allocation and scheduling module intelligently allocates and schedules tasks for multiple drones based on real-time environmental data and task requirements; The self-organizing communication and data transmission module establishes a self-organizing network between drones to achieve direct communication and data sharing between drones; The intelligent obstacle avoidance and dynamic adjustment module combines deep learning and SLAM technology to realize the perception and prediction of dynamic obstacles, provide intelligent obstacle avoidance solutions, and adjust the route in real time during flight to avoid new obstacles or dangerous areas; The AI ​​control and task execution module performs fine control of the drone based on the AI ​​algorithm, provides real-time task feedback, supports instant adjustment and optimization during task execution, and automatically starts the backup task plan or switches to other drones to continue the task when the task is interrupted; The fault prediction and emergency handling module is based on a machine learning fault prediction model to identify and warn of drone faults in advance, provide a multi-level emergency handling mechanism, and conduct emergency rehearsals through virtual simulation technology to optimize emergency handling strategies.

2. According to the AI ​​recognition-based firefighting and rescue UAV route automatic planning system according to claim 1, it is characterized in that: The multi-sensor fusion and environmental perception module includes a sensor array unit, a data acquisition unit, and a data fusion unit; the sensor array unit is responsible for collecting various environmental data at the fire scene; the data acquisition unit is responsible for acquiring real-time data from each sensor and performing preliminary preprocessing; the data fusion unit integrates data from different sensors into a unified environmental model through a fusion algorithm to provide comprehensive environmental information; The deep learning environment recognition module includes an image processing unit, a recognition model unit, and a feature extraction unit; the image processing unit is responsible for processing the image data collected by the sensor; the recognition model unit uses a pre-trained deep learning model to identify key features in the fire scene; the feature extraction unit extracts valuable feature information from the processed image or other data for analysis and decision-making by other modules; The adaptive route planning module includes a global planning unit, a local adjustment unit, and a reinforcement learning unit; the global planning unit is responsible for global route planning according to the overall mission requirements and environmental information; the local adjustment unit dynamically adjusts the local flight path of the drone based on real-time environmental changes; the reinforcement learning unit uses an adaptive learning algorithm to allow the drone to optimize the route planning strategy based on flight experience; The intelligent task allocation and scheduling module includes a task priority evaluation unit, a drone status monitoring unit, and a task scheduling unit; the task priority evaluation unit prioritizes different tasks according to the urgency, importance, and resource conditions of the tasks and determines the order of task execution; the drone status monitoring unit monitors the status of each drone in real time; The task scheduling unit intelligently allocates tasks according to task priorities and drone status, and coordinates the execution of tasks of multiple drones to avoid conflicts.

3. According to the AI ​​recognition-based firefighting and rescue UAV route automatic planning system of claim 1, it is characterized in that: The self-organizing communication and data transmission module includes a communication network unit, a data transmission optimization unit, and a 5G / satellite communication unit; the communication network unit establishes a self-organizing network between drones to support direct communication and data sharing between drones; the data transmission optimization unit is responsible for managing and optimizing the data transmission path; the 5G / satellite communication unit provides a high-bandwidth, low-latency communication link; The intelligent obstacle avoidance and dynamic adjustment module includes a dynamic perception unit, an obstacle avoidance decision unit, and a dynamic adjustment unit; the dynamic perception unit monitors dynamic obstacles during flight in real time; the obstacle avoidance decision unit calculates the optimal obstacle avoidance path based on environmental perception and SLAM technology, and performs obstacle avoidance operations; the dynamic adjustment unit dynamically adjusts the flight path of the drone according to real-time environmental changes and mission requirements.

4. According to the AI-based firefighting and rescue UAV route automatic planning system of claim 2, it is characterized in that: The AI ​​control and task execution module includes a precision control unit, a real-time feedback unit, and a task recovery unit; the precision control unit is responsible for the precise control operation of the drone; the real-time feedback unit collects and analyzes the feedback data generated during the task execution process and provides it to the control system for immediate adjustment and optimization; if the task is interrupted for some reason, the task recovery unit automatically plans an alternative plan or switches to a backup drone to continue the task; The fault prediction and emergency handling module includes a fault monitoring unit, a prediction and analysis unit, and an emergency handling unit; the fault monitoring unit monitors various key parameters of the drone in real time and identifies potential faults; the prediction and analysis unit uses a machine learning model to analyze historical data and real-time monitoring data, predict faults and send out warning signals; the emergency handling unit starts the emergency handling procedure when a fault or abnormal situation is detected.

5. According to the AI ​​recognition-based fire rescue drone route automatic planning system of claim 2, it is characterized in that: The data acquisition unit, when collecting data, uses Kalman filtering to process sensor noise so as to perform smoothing in the time series data: State prediction formula: x k|k-1 =Fx k-1|k-1 +Bu k , Covariance prediction: P k|k-1 =FP k-1|k-1 F T +Q, Kalman gain: K k =P k|k-1 H T (HP k|k-1 H T +R) -1 , Status Update: x k|k =x k|k-1 +K k (z k -Hx k|k-1 ), Covariance update: P k|k =(I-K k H)P k|k-1 , Among them, x k|k-1 is the predicted state vector at time k, P k|k-1 is the prediction of the state covariance matrix, F is the state transfer matrix, H is the measurement matrix, K k is the Kalman gain, z k is the measurement value, Q and R are the process noise covariance and measurement noise covariance; The data fusion unit uses extended Kalman filtering to fuse nonlinear systems in multi-sensor data fusion: Nonlinear state equation: x k =f(x k-1 ,u k )+w k , Nonlinear observation equation: z k =h(x k )+V k , Among them, f(.) and h(.) are the nonlinear state transfer function and observation function of the system, w k and v k are process noise and observation noise.

6. The automatic route planning system for fire rescue drones based on AI recognition according to claim 2 is characterized in that: The image processing unit includes the following related formulas: Image convolution operation: Among them, I(x, y) is the intensity value of the original image at the pixel position (x, y), K(i, j) is the convolution kernel, and I filtered (x, y) is the intensity value of the filtered image at pixel location (x, y). Histogram Equalization: Where CDF(i) is the cumulative distribution function, P(j) is the probability density function of gray level j, M×N is the total number of pixels in the image, L is the number of gray levels in the image, and I equalized (x, y) is the image after equalization; In the recognition model unit, the deep learning model is usually a convolutional neural network, and its operation steps are as follows: Convolutional Layer: h i,j,k =f(∑ m,n,l W m,n,l,k ·x i+m,j+n,l +b k ), Among them, x i,j,l is the pixel value of the input feature map, I represents the number of channels, and W m,n,l,k is the convolution kernel weight, k represents the output channel index, b k is the bias, f(.) is the activation function, h i,j,k is the pixel value of the output feature map; h i,j,k =max{x 2i,2j,k ,x 2i+1,2j,k ,x 2i,2j+1,k ,x 2i+1,2j+1,k }, Reduce the size of the feature map through pooling operations while retaining important information; Fully connected layer: z i =f(∑ j W i,j ·h j +b i ), Among them, h j is the output of the previous layer, W i,j is the weight matrix of the fully connected layer, b i is the bias, z i is the output of the fully connected layer.

7. The automatic route planning system for fire rescue UAV based on AI recognition according to claim 2 is characterized in that: The local adjustment unit may use a gradient descent method to optimize the path when adjusting the flight path in real time, so as to avoid new obstacles or adjust the flight altitude; Gradient descent update rule: Among them, x k is the coordinate vector of the path point, α is the learning rate, is the gradient of the path cost function f(x) with respect to the path point; Path cost function: The first term represents the smoothness of the path, minimizing the distance difference between path points, and the second term represents the distance cost to the obstacle, d(x j , o j ) is the path point x j With obstacles j The distance between them is short, but the cost is high.

8. The automatic route planning system for firefighting and rescue UAV based on AI recognition according to claim 3 is characterized in that: In the 5G / satellite communication unit, power control and resource allocation are key issues. The following is a resource allocation algorithm based on Lagrangian duality theory: Objective function: maximize the total rate of the system and meet the power and bandwidth constraints: Among them, P i is the transmission power of the ith user, |h i | 2 is the square of the modulus of the channel gain, σ 2 is the noise power, I i is the interference power; Constraints: a. Total power constraint: Among them, P total is the total power budget of the system; b. Bandwidth allocation for each user:

9. The automatic route planning system for firefighting and rescue UAV based on AI recognition according to claim 3 is characterized in that: The dynamic adjustment unit uses model predictive control to adjust the trajectory of the UAV in real time to ensure stable flight during obstacle avoidance; State space model: The motion model of the drone can be expressed in discrete time state space form: x k+1 =Ax k +Bu k , Among them, x k is the state vector at time k, u k is the control input at time k, the A and B system matrices; Cost function: MPC optimizes the control input by minimizing the following cost function: Among them, Q and R are the weight matrices of state and control input, Q f is the weight matrix of the terminal state; Constraints: During the optimization process, MPC usually has to meet the following constraints: x min ≤x k ≤x max , in min in k in max , Among them, x min and x max are the upper and lower limits of the state, u min and u max are the upper and lower limits of the control input.

10. The automatic route planning system for firefighting and rescue UAV based on AI recognition according to claim 3 is characterized in that: The task recovery unit is based on the task recovery of the partially observable Markov decision process. When the task execution is interrupted, the task recovery unit uses Markov to decide the optimal recovery strategy; Markov decision model: Markov is represented by state s, action a, and observation o, where state s is partially observable; Bayesian updating: based on observations o k , update the posterior distribution of the state: b k+1 (s′)6η·P(o k+1 |s′)∑ s∈S P(s′|s,a k )b k (s), Among them, b k (s) is the state belief at time k, P(o k+1 |s′) is the observation model, P(s′|s, a k ) is a state transition model, that is, executing action a in state s k The probability of transferring to state s′, η is the normalization factor; Value function: The value function is used to evaluate the optimal strategy under each belief state b, with the goal of maximizing long-term returns: Among them, R(b k ,a k ) is in belief state b k Next, perform action a k is the immediate return, and γ is the discount factor.

Citation Information

Patent Citations

  • Unmanned surface vehicle path planning method based on rapid scanning method

    CN111665846A

  • Unmanned aerial vehicle electric power inspection system

    CN113554775A

  • Unmanned aerial vehicle emergency fire control forest patrol command system

    CN113625773A

  • Method for multi-agent formation dynamic path planning and storage medium

    CN113687657A

  • Multi-sensor fusion early warning prediction method based on deep learning

    CN114664041A

Cited By

  • Air-ground unmanned equipment management and control system and method based on multi-agent collaborative learning

    CN119247864A

  • Fire trapped person positioning method and system

    CN119322312A

  • Self-adaptive planning unmanned navigation path planning system and method

    CN119356369A

  • Unmanned aerial vehicle task automatic planning and situation display system based on artificial intelligence

    CN119512164A

  • Artificial Intelligence-Based Unmanned Aerial Vehicle (UAV) Mission Automated Planning and Situation Display System

    CN119512164B