A method and system for object identification to determine the flight path of a drone

Abstract

The present invention discloses a method and system for determining the flight path of a drone by object identification, and relates to the field of data encryption technology, including collecting environmental data through multiple sensors and performing preprocessing, extracting features from the preprocessed environmental data, and performing feature fusion to obtain a comprehensive feature vector, generating a three-dimensional semantic map based on the comprehensive feature vector by using a semantic segmentation technology, dividing an area into a safe area and a dangerous area, and performing object identification on obstacles in combination with the area division results, predicting the motion trajectory of dynamic obstacles, and planning the optimal flight path of the drone in combination with the object identification, semantic segmentation and motion trajectory prediction results.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous flight and path planning of unmanned aerial vehicles, and in particular to a method and system for determining the flight path of an unmanned aerial vehicle by object identification. Background Art

[0002] In recent years, with the rapid development of UAV technology and its wide application in military, logistics, agriculture, surveying and mapping, disaster relief and other fields, higher requirements have been put forward for the autonomous flight capability of UAVs, especially in complex dynamic environments. UAVs need to be able to perceive the surrounding environment in real time, identify key objects, predict the motion trajectory of dynamic targets, and perform autonomous path planning based on the above information to ensure the safety of flight and the efficient completion of tasks. Traditional path planning methods are mostly based on rule algorithms or heuristic algorithms. These methods perform well in static and regular environments, but in dynamic and highly uncertain environments, their ability to process complex environmental information is obviously insufficient. In addition, the rapid development of deep learning technology in recent years has provided new solutions for the intelligentization of UAV tasks. By combining perception data and deep learning methods for semantic analysis and decision-making, UAVs can accurately and efficiently cope with complex environments, which has become a research hotspot.

[0003] However, the existing UAV path planning technology still faces multiple challenges. Traditional environmental perception technology usually relies on a single sensor and has deficiencies in multi-sensor fusion and multi-modal data analysis, resulting in incomplete acquisition of environmental information or untimely processing, which affects the accuracy of subsequent path planning. The existing technology has weak prediction capabilities for the motion trajectory of dynamic targets and cannot accurately predict the future position change trend of the target, especially in complex dynamic scenes, which can easily lead to the disconnection between the path planning of the UAV and the actual environment. The generation and application of three-dimensional semantic maps still have certain technical bottlenecks. The existing semantic segmentation methods are mostly limited to two-dimensional planes and cannot make full use of three-dimensional environmental information, resulting in the UAV's division of safe areas and dangerous areas in the environment is not accurate enough, thus affecting the planning effect of the flight path. In summary, the existing technology still has deficiencies in multi-sensor fusion, dynamic target trajectory prediction, and three-dimensional environment semantic segmentation. A solution with higher environmental understanding and decision-making capabilities is needed. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method for object identification to determine the flight path of a drone to solve the problem that the generation and application of three-dimensional semantic maps still have certain technical bottlenecks. Most of the existing semantic segmentation methods are limited to two-dimensional planes and cannot make full use of three-dimensional environmental information, resulting in the drone's division of safe areas and dangerous areas in the environment is not accurate enough, thus affecting the planning effect of the flight path.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for determining a flight path of a drone by object identification, comprising:

[0008] Collect environmental data through various sensors and perform pre-processing;

[0009] Extract features from the preprocessed environmental data and perform feature fusion to obtain a comprehensive feature vector;

[0010] Based on the comprehensive feature vector, a 3D semantic map is generated through semantic segmentation technology to divide the area into safe area and dangerous area;

[0011] Based on the area division results, obstacles are identified and the motion trajectory of dynamic obstacles is predicted;

[0012] Combine object identification, semantic segmentation and motion trajectory prediction results to plan the optimal flight path of the UAV.

[0013] As a preferred solution of the method for determining the flight path of a drone by object identification according to the present invention, the environmental data is collected by a variety of sensors and preprocessed, and the specific steps are as follows:

[0014] Install RGB cameras, LiDAR, infrared sensors IR, inertial measurement units IMU and global positioning systems GPS on the drone to collect visible light image data, three-dimensional point cloud data, thermal radiation image data, motion status data, and geographic location data;

[0015] Use timestamp technology to synchronize the environmental data collected by different sensors, and use Kalman filter to filter the noise in the environmental data;

[0016] The preprocessed environmental data is integrated into the environmental data set, and the environmental data set is set to x. The expression is:

[0017] x={x1,x2,x3,x4,x5};

[0018] Among them, x1 represents visible light image data, x2 represents three-dimensional point cloud data, x3 represents thermal radiation image data, x4 represents motion status data, and x5 represents geographic location data.

[0019] As a preferred solution of the method for determining the flight path of a drone by object identification according to the present invention, the steps of extracting features from the preprocessed environmental data and fusing features to obtain a comprehensive feature vector are as follows:

[0020] Extract the feature vector f1 containing image texture and color of visible light image data through convolutional neural network CNN;

[0021] Use PointNet to extract the spatial geometric feature vector f2 of 3D point cloud data;

[0022] Extract the heat intensity distribution feature vector f3 of the thermal radiation image data through a thermal image dedicated network;

[0023] The time series feature vector f4 of the motion state data is extracted through the long short-term memory network LSTM;

[0024] The spatial location feature vector f5 of the geographic location data is extracted through the fully connected layer analysis;

[0025] The feature vector f1 containing image texture and color, the spatial geometry feature vector f2, the thermal intensity distribution feature vector f3, the time series feature vector f4, and the spatial position feature vector f5 are fused into a comprehensive feature vector, which is expressed as:

[0026] F = [f1,f2,f3,f4,f5];

[0027] Where F represents the comprehensive feature vector.

[0028] As a preferred solution of the method for determining the flight path of a drone by object identification according to the present invention, wherein: based on the comprehensive feature vector, a three-dimensional semantic map is generated by semantic segmentation technology to divide the environment into a safe area and a dangerous area, and the specific steps are:

[0029] The three-dimensional feature field T is constructed by mapping function G(F), and the expression is:

[0030]

[0031] Among them, i represents the index of the eigenvector, e represents the natural number base, and f i represents the i-th eigenvector;

[0032] Assuming the semantic category label of the three-dimensional space is l, the corresponding probability distribution expression is:

[0033]

[0034] Among them, P(l|T) represents the probability distribution of the semantic category l of the three-dimensional space point in the feature field T, exp represents the exponential function, μ l represents the mean of the semantic category label l, represents the standard deviation of the semantic category label l, k represents the index of the safe area and the dangerous area, μ k represents the mean of the safe area and the dangerous area, Indicates the standard deviation of safe and dangerous areas;

[0035] For each 3D point q m , determine its semantic label L according to the maximum probability principle.

[0036] As a preferred solution of the method for determining the flight path of a drone by object identification according to the present invention, wherein: for each three-dimensional point, its semantic label L is determined according to the maximum probability principle, and the expression is:

[0037]

[0038] Among them, argmax represents the maximum value function;

[0039] Map the semantic label L to the three-dimensional space to generate a three-dimensional semantic map, which is expressed as:

[0040] M = L;

[0041] Where M represents a three-dimensional semantic map.

[0042] As a preferred solution of the method for determining the flight path of a drone by object identification according to the present invention, the method combines the area division result, identifies the obstacles, and predicts the motion trajectory of the dynamic obstacles. The specific steps are as follows:

[0043] An ultrasonic anemometer is installed on the drone to collect environmental forces A(t) including wind speed and wind direction;

[0044] For dynamic obstacles, the state transfer equation of the dynamic target is constructed based on the Markov decision process MDP, and the expression is:

[0045] S(t+1)=F s (S(t),A(t))+∈(t);

[0046] Among them, S(t+1) represents the state vector of the target at the next moment, S(t) represents the state vector of the target at time t, A(t) represents the environmental force, ∈(t) represents the random noise term, which is obtained using Gaussian noise;

[0047] Among them, F s Represents the state transfer function, expressed as:

[0048]

[0049] Among them, G(M) represents the feature mapping function of the semantic region, represents the gradient of the target state, W1 represents the weight coefficient of the state vector of the target at time t, W2 represents the weight coefficient of the feature mapping function of the semantic area, and W3 represents the weight coefficient of the gradient of the target state;

[0050] Based on the state vector S(t) of the target at time t, the motion trajectory C(t) of the dynamic target is generated, which is expressed as:

[0051]

[0052] Where T represents the prediction time window, S x (t) represents the spatial position of the target at time t, represents the speed of the target at time t, Φ represents the priority weight of the semantic area where the target is located, α1 represents the weighting coefficient of the spatial position of the target at time t, α2 represents the weighting coefficient of the speed of the target at time t, and α3 represents the weighting coefficient of the priority weight of the semantic area where the target is located.

[0053] As a preferred solution of the method for determining the flight path of a drone by object identification according to the present invention, the optimal flight path of the drone is planned by combining the object identification, semantic segmentation and motion trajectory prediction results, and the specific steps are as follows:

[0054] Based on the results of semantic segmentation, object identification and motion prediction, a path set Q is dynamically generated on the three-dimensional semantic map M, which is expressed as:

[0055] Q={q1,q2,...,q n};

[0056] Among them, q represents a path to be selected, and n represents the number of paths;

[0057] Based on the results of semantic segmentation, object identification and motion prediction, a path cost function is constructed, which is expressed as:

[0058] J(Q)=∫ Q [w1·d(U,C(t))+w2·Ψ(M)]dl;

[0059] Where Q represents the path set, J(Q) represents the path cost function, d(U,C(t)) represents the distance from the current position U of the UAV to the predicted motion trajectory C(t), Ψ(M) represents the regional cost function, w1 represents the weight coefficient of the distance from the current position U of the UAV to the predicted motion trajectory C(t), w2 represents the weight coefficient of the regional cost function, and dl represents the path differential element;

[0060] Use the A* algorithm to search for the path that minimizes J(Q) on the three-dimensional semantic map and obtain the optimal path D;

[0061] The UAV performs flight inspection along the optimal path D.

[0062] In a second aspect, the present invention provides a system for determining the flight path of a UAV by object identification, comprising an environmental data acquisition module, a data preprocessing module, a feature fusion module, a three-dimensional semantic map generation module, a motion trajectory prediction module, an object identification module and a flight path planning module.

[0063] The environmental data acquisition module collects environmental data in real time through a variety of sensors and provides raw data for subsequent processing;

[0064] The data preprocessing module is used to clean, filter and normalize the collected environmental data to improve the data quality;

[0065] The feature fusion module uses a deep learning model to extract and fuse features of the preprocessed environmental data to generate a comprehensive feature vector;

[0066] The three-dimensional semantic map generation module generates a three-dimensional semantic map by semantic segmentation based on the comprehensive feature vector, and divides the environment into a safe area and a dangerous area;

[0067] The motion trajectory prediction module: predicts the motion trajectory of the dynamic target based on the environmental segmentation result and analyzes the change trend of the target position;

[0068] The object identification module identifies and classifies dynamic and static objects in the environment based on semantic segmentation results and target features;

[0069] The flight path planning module plans the optimal flight path of the UAV according to the object identification, semantic segmentation and motion trajectory prediction results to ensure safety and mission efficiency.

[0070] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the method for determining the flight path of a drone by object identification as described in the first aspect of the present invention.

[0071] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the method for determining the flight path of a drone by object identification as described in the first aspect of the present invention.

[0072] The beneficial effects of the present invention are: through multi-sensor collaborative perception, deep feature extraction and fusion, semantic segmentation to generate three-dimensional semantic maps, dynamic obstacle recognition and motion trajectory prediction, and optimal path planning based on semantic and dynamic prediction, a comprehensive, intelligent and efficient UAV flight path planning method is constructed. This technical solution not only realizes comprehensive perception and understanding of the environment, but also gives UAVs the ability to dynamically adapt to complex scenes, and can plan safe and efficient flight paths in real time, which significantly improves the safety, reliability and task completion efficiency of UAVs in complex dynamic environments. Compared with the existing technology, it has significant technical progress and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0074] Figure 1 This is a flowchart of the method for determining the flight path of a drone by object identification in Example 1.

[0075] Figure 2 Schematic diagram of the system for object identification to determine the flight path of a drone in Example 1. DETAILED DESCRIPTION

[0076] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0077] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0078] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0079] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a method for determining a flight path of a drone by object identification, comprising the following steps:

[0080] Collect environmental data through various sensors and perform pre-processing;

[0081] Extract features from the preprocessed environmental data and perform feature fusion to obtain a comprehensive feature vector;

[0082] Based on the comprehensive feature vector, a 3D semantic map is generated through semantic segmentation technology to divide the area into safe area and dangerous area;

[0083] Based on the area division results, obstacles are identified and the motion trajectory of dynamic obstacles is predicted;

[0084] Combine object identification, semantic segmentation and motion trajectory prediction results to plan the optimal flight path of the UAV.

[0085] This embodiment also provides a system for determining a flight path of a drone by object identification, including:

[0086] S1 collects environmental data through various sensors and performs preprocessing;

[0087] Install RGB cameras, LiDAR, infrared sensors IR, inertial measurement units IMU and global positioning systems GPS on the drone to collect visible light image data, three-dimensional point cloud data, thermal radiation image data, motion status data, and geographic location data;

[0088] It solves the problem of limited sensing range of traditional single sensors, especially in complex environments with low light, dynamic obstacles, and diverse terrain. The collaboration of multiple sensors can significantly improve the diversity and accuracy of environmental perception.

[0089] The visible light image data includes image resolution, RGB pixel values, and visible light image frame rate;

[0090] The 3D point cloud data includes the 3D coordinates, intensity information, and point cloud density of the points;

[0091] The thermal radiation image data includes thermal image resolution, temperature value range, and thermal radiation image frame rate;

[0092] The motion state data includes motion speed, acceleration, angular velocity, and attitude angle;

[0093] Geolocation data includes longitude, latitude, altitude, speed, direction, and timestamp;

[0094] Use timestamp technology to synchronize the environmental data collected by different sensors, and use Kalman filter to filter the noise in the environmental data;

[0095] The preprocessed environmental data is integrated into the environmental data set, and the environmental data set is set to x. The expression is:

[0096] x={x1,x2,x3,x4,x5};

[0097] Among them, x1 represents visible light image data, x2 represents three-dimensional point cloud data, x3 represents thermal radiation image data, x4 represents motion state data, and x5 represents geographic location data;

[0098] Through this step, the UAV achieves comprehensive perception of complex environmental information and ensures the accuracy and consistency of the data through synchronization and preprocessing. Compared with traditional methods, this step not only expands the perception range, but also enhances the reliability of environmental data, providing high-quality basic data for subsequent feature extraction and fusion.

[0099] S2 extracts features from the preprocessed environmental data and performs feature fusion to obtain a comprehensive feature vector;

[0100] Extract the feature vector f1 containing image texture and color of visible light image data through convolutional neural network CNN;

[0101] Use PointNet to extract the spatial geometric feature vector f2 of 3D point cloud data;

[0102] Extract the heat intensity distribution feature vector f3 of the thermal radiation image data through a thermal image dedicated network;

[0103] The time series feature vector f4 of the motion state data is extracted through the long short-term memory network LSTM;

[0104] The spatial location feature vector f5 of the geographic location data is extracted through the fully connected layer analysis;

[0105] It solves the problem that a single feature model cannot fully describe a complex environment, and at the same time improves the accuracy and robustness of feature extraction through deep learning technology;

[0106] The feature vector f1 containing image texture and color, the spatial geometry feature vector f2, the thermal intensity distribution feature vector f3, the time series feature vector f4, and the spatial position feature vector f5 are fused into a comprehensive feature vector, which is expressed as:

[0107] F = [f1,f2,f3,f4,f5];

[0108] Where F represents the comprehensive feature vector;

[0109] Through this step, the drone obtains a comprehensive feature vector that integrates multi-dimensional perception information, significantly enhancing its ability to describe complex environments. Compared with the traditional single feature extraction method, this step not only achieves the deep fusion of multimodal data, but also improves the integrity and accuracy of environmental information, providing more accurate input for subsequent semantic segmentation and path planning.

[0110] Based on the comprehensive feature vector, S3 generates a three-dimensional semantic map through semantic segmentation technology, dividing the area into safe and dangerous areas;

[0111] The problem that traditional geometric maps can only provide spatial location information is solved. Semantic maps further give safety semantics to environmental areas, enabling drones to perceive and understand the environment more intelligently.

[0112] The three-dimensional feature field T is constructed by mapping function G(F), and the expression is:

[0113]

[0114] Among them, i represents the index of the eigenvector, e represents the natural number base, and f i represents the i-th eigenvector;

[0115] Assume that the semantic category label of the three-dimensional space is l, and the value range of l is 0 and 1, representing the safe area and the dangerous area respectively. The corresponding probability distribution expression is:

[0116]

[0117] Among them, P(l|T) represents the probability distribution of the semantic category l of the three-dimensional space point in the feature field T, exp represents the exponential function, μ l represents the mean of the semantic category label l, represents the standard deviation of the semantic category label l, k represents the index of the safe area and the dangerous area, μ k represents the mean of the safe area and the dangerous area, Indicates the standard deviation of safe and dangerous areas;

[0118] For each 3D point q m , determine its semantic label L according to the maximum probability principle, the expression is:

[0119]

[0120] Among them, argmax represents the maximum value function;

[0121] When l = 0, it indicates a safe area, and when l = 1, it indicates a dangerous area;

[0122] Compared with traditional methods, this step significantly improves the drone's ability to understand the environment and provides an efficient decision-making basis for subsequent path planning;

[0123] Map the semantic label L to the three-dimensional space to generate a three-dimensional semantic map, which is expressed as:

[0124] M = L;

[0125] Where M represents a three-dimensional semantic map;

[0126] The real-time generation capability of 3D semantic maps ensures the adaptability and decision-making accuracy of drones in dynamic scenarios;

[0127] Through this step, the drone can quickly identify safe and dangerous areas in the flight environment based on the three-dimensional semantic map generated in real time.

[0128] S4 combines the area division results to identify obstacles and predict the motion trajectory of dynamic obstacles;

[0129] When the speed in the motion state data is 0, it means that the obstacle is stationary and is marked as a static obstacle;

[0130] When the speed in the motion state data is greater than 0, it means that the obstacle is moving and is marked as a dynamic obstacle;

[0131] It solves the problem of unpredictable obstacle movement in dynamic environments. By predicting the motion trajectory, the drone can perceive the potential collision risk in advance.

[0132] An ultrasonic anemometer is installed on the drone to collect environmental forces A(t) including wind speed and wind direction;

[0133] For dynamic obstacles, the state transfer equation of the dynamic target is constructed based on the Markov decision process MDP, and the expression is:

[0134] S(t+1)=F s (S(t),A(t))+∈(t);

[0135] Among them, S(t+1) represents the state vector of the target at the next moment, S(t) represents the state vector of the target at time t, A(t) represents the environmental force, ∈(t) represents the random noise term, which is obtained using Gaussian noise;

[0136] Among them, F s Represents the state transfer function, expressed as:

[0137]

[0138] Among them, G(M) represents the feature mapping function of the semantic region, represents the gradient of the target state, W1 represents the weight coefficient of the state vector of the target at time t, W2 represents the weight coefficient of the feature mapping function of the semantic area, and W3 represents the weight coefficient of the gradient of the target state;

[0139] Based on the state vector S(t) of the target at time t, the motion trajectory C(t) of the dynamic target is generated, which is expressed as:

[0140]

[0141] Where T represents the prediction time window, S x (t) represents the spatial position of the target at time t, represents the speed of the target at time t, Φ represents the priority weight of the semantic area where the target is located, α1 represents the weighting coefficient of the spatial position of the target at time t, α2 represents the weighting coefficient of the speed of the target at time t, and α3 represents the weighting coefficient of the priority weight of the semantic area where the target is located;

[0142] Through this step, the UAV can accurately identify and predict the motion of dynamic obstacles, significantly improving its obstacle avoidance capabilities in complex dynamic environments. Compared with traditional static map methods, this step gives the UAV the ability to adapt to dynamic environments, ensuring the safety and stability of flight.

[0143] S5 combines object identification, semantic segmentation and motion trajectory prediction results to plan the optimal flight path of the drone;

[0144] Combining semantic information and dynamic prediction, optimal path planning that takes into account both safety and efficiency is achieved;

[0145] Based on the results of semantic segmentation, object identification and motion prediction, a path set Q is dynamically generated on the three-dimensional semantic map M, which is expressed as:

[0146] Q={q1,q2,...,q n};

[0147] Among them, q represents a path to be selected, and n represents the number of paths;

[0148] Based on the results of semantic segmentation, object identification and motion prediction, a path cost function is constructed, which is expressed as:

[0149] J(Q)=∫ Q [w1·d(U,C(t))+w2·Ψ(M)]dl;

[0150] Where Q represents the path set, J(Q) represents the path cost function, d(U,C(t)) represents the distance from the current position U of the UAV to the predicted motion trajectory C(t), Ψ(M) represents the regional cost function, w1 represents the weight coefficient of the distance from the current position U of the UAV to the predicted motion trajectory C(t), w2 represents the weight coefficient of the regional cost function, and dl represents the path differential element;

[0151] By optimizing the path cost function, we ensure that the path planning results take into account both safety and efficiency;

[0152] Use the A* algorithm to search for the path that minimizes J(Q) on the three-dimensional semantic map and obtain the optimal path D;

[0153] Using the dynamic programming algorithm (A*), the real-time adaptability of the UAV in complex environments is achieved;

[0154] The UAV performs flight inspection along the optimal path D.

[0155] This embodiment also provides a computer device, which is suitable for the method of determining the flight path of a drone by object identification, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the method of determining the flight path of a drone by object identification as proposed in the above embodiment.

[0156] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0157] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for determining the flight path of a drone by object identification as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0158] In summary, the present invention constructs a comprehensive, intelligent and efficient UAV flight path planning method through: multi-sensor collaborative perception, deep feature extraction and fusion, semantic segmentation to generate three-dimensional semantic maps, dynamic obstacle recognition and motion trajectory prediction, and optimal path planning based on semantic and dynamic prediction. This technical solution not only realizes the comprehensive perception and understanding of the environment, but also gives the UAV the ability to dynamically adapt to complex scenes, and can plan safe and efficient flight paths in real time, which significantly improves the safety, reliability and task completion efficiency of UAVs in complex dynamic environments. Compared with the existing technology, it has significant technical progress and application value.

[0159] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for determining the flight path of a drone by object identification, characterized in that: include, Collect environmental data through various sensors and perform pre-processing; Extract features from the preprocessed environmental data and perform feature fusion to obtain a comprehensive feature vector; Based on the comprehensive feature vector, a 3D semantic map is generated through semantic segmentation technology to divide the area into safe area and dangerous area; Based on the area division results, obstacles are identified and the motion trajectory of dynamic obstacles is predicted; Combine object identification, semantic segmentation and motion trajectory prediction results to plan the optimal flight path of the UAV.

2. The method for determining the flight path of a drone by object identification according to claim 1, characterized in that: The environmental data is collected through a variety of sensors and preprocessed, and the specific steps are as follows: Install RGB cameras, LiDAR, infrared sensors IR, inertial measurement units IMU and global positioning systems GPS on the drone to collect visible light image data, three-dimensional point cloud data, thermal radiation image data, motion status data, and geographic location data; Use timestamp technology to synchronize the environmental data collected by different sensors, and use Kalman filter to filter the noise in the environmental data; The preprocessed environmental data is integrated into the environmental data set, and the environmental data set is set to x. The expression is: x={x1,x2,x3,x4,x5}; Among them, x1 represents visible light image data, x2 represents three-dimensional point cloud data, x3 represents thermal radiation image data, x4 represents motion status data, and x5 represents geographic location data.

3. The method for determining the flight path of a drone by object identification as claimed in claim 2, characterized in that: The process of extracting features from the preprocessed environmental data and fusing the features to obtain a comprehensive feature vector is as follows: Extract the feature vector f1 containing image texture and color of visible light image data through convolutional neural network CNN; Use PointNet to extract the spatial geometric feature vector f2 of 3D point cloud data; Extract the heat intensity distribution feature vector f3 of the thermal radiation image data through a thermal image dedicated network; The time series feature vector f4 of the motion state data is extracted through the long short-term memory network LSTM; The spatial location feature vector f5 of the geographic location data is extracted through the fully connected layer analysis; The feature vector f1 containing image texture and color, the spatial geometry feature vector f2, the thermal intensity distribution feature vector f3, the time series feature vector f4, and the spatial position feature vector f5 are fused into a comprehensive feature vector, which is expressed as: F = [f1,f2,f3,f4,f5]; Where F represents the comprehensive feature vector.

4. The method for determining the flight path of a drone by object identification as claimed in claim 3, characterized in that: The method generates a three-dimensional semantic map based on the comprehensive feature vector by using semantic segmentation technology to divide the environment into a safe area and a dangerous area. The specific steps are as follows: The three-dimensional feature field T is constructed by mapping function G(F), and the expression is: Among them, i represents the index of the eigenvector, e represents the natural number base, and f i represents the i-th eigenvector; Assuming the semantic category label of the three-dimensional space is l, the corresponding probability distribution expression is: Among them, P(l|T) represents the probability distribution of the semantic category l of the three-dimensional space point in the feature field T, exp represents the exponential function, μ l represents the mean of the semantic category label l, represents the standard deviation of the semantic category label l, k represents the index of the safe area and the dangerous area, μ k represents the mean of the safe area and the dangerous area, Indicates the standard deviation of safe and dangerous areas; For each 3D point q m , determine its semantic label L according to the maximum probability principle.

5. The method for determining the flight path of a drone by representing an object identifier according to claim 4, characterized in that: For each 3D point, its semantic label L is determined according to the maximum probability principle, and the expression is: Among them, argmax represents the maximum value function; Map the semantic label L to the three-dimensional space to generate a three-dimensional semantic map, which is expressed as: M = L; Where M represents a three-dimensional semantic map.

6. The method for determining the flight path of a drone by object identification as claimed in claim 5, characterized in that: The specific steps of combining the area division results, identifying obstacles, and predicting the motion trajectory of dynamic obstacles are as follows: An ultrasonic anemometer is installed on the drone to collect environmental forces A(t) including wind speed and wind direction; For dynamic obstacles, the state transfer equation of the dynamic target is constructed based on the Markov decision process MDP, and the expression is: S(t+1)=F s (S(t),A(t))+∈(t); Among them, S(t+1) represents the state vector of the target at the next moment, S(t) represents the state vector of the target at time t, A(t) represents the environmental force, ∈(t) represents the random noise term, which is obtained using Gaussian noise; Among them, F s Represents the state transfer function, expressed as: Among them, G(M) represents the feature mapping function of the semantic region, represents the gradient of the target state, W1 represents the weight coefficient of the state vector of the target at time t, W2 represents the weight coefficient of the feature mapping function of the semantic area, and W3 represents the weight coefficient of the gradient of the target state; Based on the state vector S(t) of the target at time t, the motion trajectory C(t) of the dynamic target is generated, which is expressed as: Where T represents the prediction time window, S x (t) represents the spatial position of the target at time t, represents the speed of the target at time t, Φ represents the priority weight of the semantic area where the target is located, α1 represents the weighting coefficient of the spatial position of the target at time t, α2 represents the weighting coefficient of the speed of the target at time t, and α3 represents the weighting coefficient of the priority weight of the semantic area where the target is located.

7. The method for determining the flight path of a drone by object identification according to claim 6, characterized in that: The optimal flight path of the UAV is planned by combining the object identification, semantic segmentation and motion trajectory prediction results. The specific steps are as follows: Based on the results of semantic segmentation, object identification and motion prediction, a path set Q is dynamically generated on the three-dimensional semantic map M, which is expressed as: Q={q1,q2,...,q n }; Among them, q represents a path to be selected, and n represents the number of paths; Based on the results of semantic segmentation, object identification and motion prediction, a path cost function is constructed, which is expressed as: J(Q)=∫ Q [w1·d(U,C(t))+w2·Ψ(M)]dl; Where Q represents the path set, J(Q) represents the path cost function, d(U,C(t)) represents the distance from the current position U of the UAV to the predicted motion trajectory C(t), Ψ(M) represents the regional cost function, w1 represents the weight coefficient of the distance from the current position U of the UAV to the predicted motion trajectory C(t), w2 represents the weight coefficient of the regional cost function, and dl represents the path differential element; Use the A* algorithm to search for the path that minimizes J(Q) on the three-dimensional semantic map and obtain the optimal path D; The UAV performs flight inspection along the optimal path D.

8. A system for determining the flight path of a drone by object identification, based on the method for determining the flight path of a drone by object identification according to any one of claims 1 to 7, characterized in that: It includes environmental data acquisition module, data preprocessing module, feature fusion module, 3D semantic map generation module, motion trajectory prediction module, object identification module and flight path planning module. The environmental data acquisition module collects environmental data in real time through a variety of sensors and provides raw data for subsequent processing; The data preprocessing module is used to clean, filter and normalize the collected environmental data to improve the data quality; The feature fusion module uses a deep learning model to extract and fuse features of the preprocessed environmental data to generate a comprehensive feature vector; The three-dimensional semantic map generation module generates a three-dimensional semantic map by semantic segmentation based on the comprehensive feature vector, and divides the environment into a safe area and a dangerous area; The motion trajectory prediction module: predicts the motion trajectory of the dynamic target based on the environmental segmentation result and analyzes the change trend of the target position; The object identification module identifies and classifies dynamic and static objects in the environment based on semantic segmentation results and target features; The flight path planning module plans the optimal flight path of the UAV according to the object identification, semantic segmentation and motion trajectory prediction results to ensure safety and mission efficiency.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for determining the flight path of a drone by object identification as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for determining the flight path of a drone by object identification as described in any one of claims 1 to 7 are implemented.

Images