A Multi-Source Fusion Positioning Method and System for Low-Altitude UAVs

Abstract

The present invention discloses a multi-source fusion positioning method and system for low-altitude unmanned aerial vehicles, which relates to the technical field of unmanned aerial vehicle positioning. It solves the technical problem that it is difficult to identify an object by combining various information data of the object, which affects the accuracy of target positioning. By combining the data of infrared images and high-definition images, the information of both is fully utilized to improve the accuracy of target detection and recognition. Infrared imaging provides information about the surface temperature of an object, which is very important for identifying certain types of target objects. Through the temperature distribution, the state or behavior of the target object can be better understood. Combining the temperature information with the visual information helps to distinguish objects with similar appearances but different thermal characteristics in a complex environment, thus improving the classification effect. By mapping the extracted temperature estimation data to the corresponding pixel points in the high-definition image, more accurate position positioning is achieved, providing a reliable basis for subsequent tasks.

Description

Technical Field

[0001] The present invention belongs to the field of UAV positioning, and specifically relates to a multi-source fusion positioning method and system for low-altitude UAVs. Background Art

[0002] With the rapid development of UAV technology, low-altitude UAVs are increasingly widely used in fields such as agricultural monitoring, environmental protection, urban management, and disaster relief. However, the low-altitude flight environment is usually complex and dynamic. For example, buildings, trees, and other obstacles may pose challenges to the positioning accuracy of UAVs. Traditional single positioning methods often face problems such as weak signals, large drifts, or complete failures in these scenarios. Therefore, a method that can integrate multiple sensor data is needed to improve the positioning accuracy and reliability of UAVs in complex environments. In recent years, multi-source data fusion technology has gradually become an important means to solve this problem. By fusing data from different sensors, such as GPS, IMU, lidar, vision sensors, etc., the limitations of single sensors can be effectively overcome, and more accurate and stable positioning can be achieved. This not only helps to improve the autonomous flight ability of UAVs but also enhances their execution efficiency in specific tasks.

[0003] Most of the existing multi-source fusion positioning schemes for low-altitude UAVs perform target positioning by identifying objects in high-definition images, and it is difficult to combine various information data of the objects to identify the objects, which affects the accuracy of target positioning. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; for this purpose, the present invention proposes a multi-source fusion positioning method and system for low-altitude UAVs to solve the technical problem that it is difficult to combine various information data of the object to identify the object, which affects the accuracy of target positioning.

[0005] To solve the above problems, the first aspect of the present invention provides a multi-source fusion positioning method for low-altitude UAVs, including:

[0006] Mark the target positioning area on the map, divide the marked target positioning area into blocks, allocate at least one UAV to each search block for target search, and set a specific flight trajectory for each UAV. The UAVs fly low over specific positions in the search blocks according to the flight trajectories to collect image data of the search blocks;

[0007] Collect the real-time position data, attitude data of the UAVs, and three-dimensional point cloud data in front of the UAVs. At the same time, collect infrared image data and high-definition image data in multiple directions of the UAVs;

[0008] Estimate the temperature based on the infrared image data of the drone, map the temperature estimation data extracted from the infrared image data of the drone to the corresponding pixel points of the high-definition image data, combine the contour information of the target object, and analyze the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel points;

[0009] Screen the similar contour information with the actual deviation coefficient less than the preset threshold, and establish a target object tracking list according to the actual deviation coefficient of the screened similar contour information;

[0010] According to the target object tracking list, move to the similar contour position one by one, establish a tracking rule for the similar contour position, track the target object according to the tracking rule of the similar contour position, and calculate the target object position information according to the position data, attitude data of the drone detecting the target object and the three-dimensional point cloud data in front of the drone.

[0011] As a further solution of the present invention: collect the attitude data of the drone and the three-dimensional point cloud data in front of the drone. At the same time, collect the infrared image data and high-definition image data in multiple directions of the drone, including the following steps:

[0012] Obtain the attitude data of the drone in real time through the IMU inertial measurement unit, collect the infrared image data in four directions of the front, below, left, and right through the infrared thermal imaging camera, and collect the high-definition image data in the corresponding direction of the infrared thermal imaging camera through the RGB camera;

[0013] Scan the front of the drone in real time through the lidar to collect the three-dimensional point cloud data in front of the drone.

[0014] As a further solution of the present invention: estimate the temperature based on the infrared image data of the drone, and map the temperature estimation data extracted from the infrared image data of the drone to the corresponding pixel points of the high-definition image data, including the following steps:

[0015] Align the infrared images and high-definition image data collected by the infrared thermal imaging camera and the RGB camera in the same direction at the same time, and map several pixel points in the corresponding high-definition image data to the corresponding pixel points in the infrared image;

[0016] Estimate the temperature of each pixel point of the infrared images in four directions of the drone obtained, and associate the obtained temperature estimation value with the pixel point of the high-definition image corresponding to the pixel point of the infrared image.

[0017] As a further solution of the present invention: estimate the temperature of each pixel point of the infrared images in four directions of the drone obtained, including the following steps:

[0018] Collect a large number of infrared images of the same kind of target objects under the same conditions in the target positioning area through an infrared camera, measure the temperatures at different positions of the same kind of target objects corresponding to the infrared images through an infrared thermometer, and record the corresponding temperature values;

[0019] Map the acquisition positions where temperature measurements are taken to the pixel points of the infrared images of the same kind of target objects, and use the recorded corresponding temperature values as the labels for the corresponding pixel points;

[0020] Train a convolutional neural network model with the infrared images of the same kind of target objects after adding labels;

[0021] Input the infrared images of the four directions of the drone collected in real time into the trained convolutional neural network model to estimate the temperature of each pixel point of the infrared images.

[0022] As a further solution of the present invention: Analyze the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel points, including the following steps:

[0023] Collect the temperature information of a large number of the same kind of target objects under the same conditions in the target positioning area, set the temperature range of the target object according to the collected temperature information, and set the standard temperature value of the target object according to the average value of the collected temperature information;

[0024] Collect the high-definition image data of a large number of the same kind of target objects under the same conditions in the target positioning area, and mark the contours of the same kind of target objects;

[0025] Train a deep learning model with the marked high-definition image data to identify the contour information of the target object;

[0026] Input the high-definition image data collected in real time into the trained deep learning model to identify the similar contour of the target object;

[0027] Annotate the similar contour part of the target object detected in the high-definition image data, and analyze the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel points.

[0028] As a further solution of the present invention: Analyze the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel points, including the following steps:

[0029] Perform gray-scale processing on the high-definition image of the contour of the same kind of target objects with labels, and set the gray-scale value range of the target object according to the maximum and minimum gray-scale values of each pixel point in the contour part;

[0030] Extract the contour part of the high-definition image of the same-kind substances of the target object after grayscale processing, scale it to the same size, and move all the contours to the same position in a unified coordinate system for image fusion;

[0031] Statistically calculate the grayscale value of each pixel point in the contour part of the image in the coordinates, and divide it by the total number of contours to obtain the average value as the standard grayscale value;

[0032] Scale the contour part of the similar contour of the target object retrieved from the high-definition image data to the same size;

[0033] Statistically calculate the grayscale value of the similar contour part of the retrieved target object, and based on the temperature estimation value corresponding to each pixel point of the similar contour part of the target object, analyze the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel point, using the following formula:

[0034]

[0035] where μ is the actual deviation coefficient, G i is the grayscale value of the i-th pixel point in the detected similar contour information, G 0 is the standard grayscale value of the i-th pixel point in the contour part of the image in the coordinates, T i is the temperature estimation value of the i-th pixel point in the detected similar contour information, T 0 is the standard temperature value of the target object, and i ∈ (1, 2,..., n).

[0036] As a further solution of the present invention: According to the target object tracking list, move towards the similar contour positions one by one to establish a similar contour position tracking rule, including the following steps:

[0037] According to the target object tracking list, the unmanned aerial vehicle leaves the flight trajectory and moves towards the similar contour positions one by one, and establishes the following similar contour position tracking rule:

[0038] Based on the attitude data of the unmanned aerial vehicle and the three-dimensional point cloud data in front of the unmanned aerial vehicle, detect the distance between the similar contour position and the unmanned aerial vehicle. If the distance exceeds the preset threshold, give up moving towards the corresponding similar contour position and delete it from the target object tracking list; otherwise, move towards the similar contour position;

[0039] When the unmanned aerial vehicle moves towards the corresponding similar contour position and analyzes the actual deviation coefficient between the detected similar contour position and the target object in real time according to the collected data. If the actual deviation coefficient is less than the minimum similarity threshold, stop moving towards the corresponding similar contour position and delete it from the target object tracking list; otherwise, continue to move towards the similar contour position;

[0040] If the distance between the detected similar contour position and the UAV is less than the preset detection distance, and at the same time, the actual deviation coefficient is greater than the maximum similarity threshold, it is determined that the target object has been locked; otherwise, the movement towards the corresponding similar contour position is abandoned and deleted from the target tracking list.

[0041] If it is determined that the target object has been locked, other UAVs are immediately stopped from searching for the target.

[0042] As a further solution of the present invention: Track the target object according to the similar contour position tracking rule, and calculate the target object position information based on the position data, attitude data of the UAV that detects the target object, and the three-dimensional point cloud data in front of the UAV, including the following steps:

[0043] Track the target object according to the similar contour position tracking rule, the UAV moves towards the similar contour position, and at the same time, detect the three-dimensional point cloud data in the direction of the similar contour position in front of the UAV.

[0044] Obtain the high-definition image of the similar contour of the marked target object collected in front of the UAV, map the similar contour of the marked target object into the three-dimensional point cloud data in front of the UAV, and calculate the target object position information according to the position data and attitude data of the UAV that detects the target object, and the mapping relationship of the similar contour in the three-dimensional point cloud data in front of the UAV.

[0045] As a further solution of the present invention: Calculate the target object position information according to the position data and attitude data of the UAV that detects the target object, and the mapping relationship of the similar contour in the three-dimensional point cloud data in front of the UAV, including the following steps:

[0046] Convert the coordinate system of the UAV itself to the world coordinate system according to the GPS position and attitude data of the UAV.

[0047] Convert the obtained three-dimensional point cloud data from the lidar coordinate system to the world coordinate system.

[0048] According to the internal parameter data of the RGB camera, map the contour part coordinates of the high-definition image of the similar contour of the marked target object to the world coordinate system, and calculate the depth value of each contour point in the contour part.

[0049] According to the depth value of each contour point in the contour part, calculate the average value of the distances from each contour point to the position of each contour point as the distance between the detected target object and the UAV, and obtain the target object position information according to the GPS position, attitude data of the UAV and the distance between the detected target object and the UAV.

[0050] As another aspect of the present invention: A low-altitude UAV multi-source fusion positioning system, which uses a low-altitude UAV multi-source fusion positioning method as described above to achieve low-altitude UAV multi-source fusion positioning.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] By combining the data of infrared images and high-definition images, the present invention makes full use of the information of both to improve the accuracy of target detection and recognition. Infrared imaging provides information about the surface temperature of an object, which is very important for identifying certain types of target objects; through the temperature distribution, the state or behavior of the target object can be better understood. Combining the temperature information with the visual information helps to distinguish objects with similar appearances but different thermal characteristics in a complex environment, thereby improving the classification effect. By mapping the extracted temperature estimation data to the corresponding pixel points in the high-definition image, more accurate position positioning is achieved, providing a reliable basis for subsequent tasks.

[0053] According to the target tracking list, the present invention moves towards the similar contour positions one by one, establishes the tracking rules for the similar contour positions, tracks the target objects according to the tracking rules for the similar contour positions, and calculates the target object position information based on the position data, attitude data of the drone detecting the target object, and the three-dimensional point cloud data in front of the drone; by combining the position information, attitude information and three-dimensional point cloud data of the drone, it is convenient to improve the estimation accuracy of the position of the target object, thereby achieving more accurate positioning. By dynamically acquiring and processing multi-source data, the position estimation of the target object can be adjusted in real time, improving the response speed; by establishing the tracking rules for the similar contour positions, the drone can flexibly respond to environmental changes or the movement of the target object, achieving continuous and effective tracking. According to the tracking rules, the optimal flight route is planned, enabling the drone to efficiently cover all relevant areas, thereby saving energy and time. Concentrating resources on the confirmed targets helps to improve the overall task efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0055] Figure 1 It is a schematic flowchart of the method of the present invention;

[0056] Figure 2 It is a flowchart of the method for mapping the temperature estimation data of the present invention to the corresponding pixel points of the high-definition image data. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work belong to the scope of protection of the present invention.

[0058] Please refer to Figure 1 - Figure 2 , an embodiment of the first aspect of the present invention provides a multi-source fusion positioning method for low-altitude unmanned aerial vehicles, including:

[0059] Mark the target positioning area on the map, divide the marked target positioning area into blocks, allocate at least one unmanned aerial vehicle to each search block for target search, and set a specific flight trajectory for each unmanned aerial vehicle. The unmanned aerial vehicle flies low over specific positions in the search block according to the flight trajectory to collect image data of the search block.

[0060] Collect the real-time position data, attitude data of the unmanned aerial vehicle, and three-dimensional point cloud data in front of the unmanned aerial vehicle. At the same time, collect infrared image data and high-definition image data in multiple directions of the unmanned aerial vehicle.

[0061] Estimate the temperature according to the infrared image data of the unmanned aerial vehicle, map the temperature estimation data extracted from the infrared image data of the unmanned aerial vehicle to the corresponding pixel points of the high-definition image data, and combine the target object contour information. Analyze the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel points.

[0062] Screen the similar contour information with an actual deviation coefficient less than the preset threshold, and establish a target object tracking list according to the actual deviation coefficient of the screened similar contour information.

[0063] According to the target object tracking list, move to the similar contour position one by one, establish a tracking rule for the similar contour position, track the target object according to the tracking rule of the similar contour position, and calculate the target object position information according to the position data, attitude data of the unmanned aerial vehicle that detects the target object, and the three-dimensional point cloud data in front of the unmanned aerial vehicle.

[0064] Specifically, in this embodiment, the GIS software, QGIS map tool or ArcGIS map tool is used to manually draw the target positioning area that needs to be searched.

[0065] Mark the selected target area on the map, draw the boundary box or polygon of the area according to the actual situation, and save its coordinate information.

[0066] Divide the marked target positioning area into blocks:

[0067] Grid division: Divide the entire target area into certain sizes. For example, divide it into grids of 100 meters × 100 meters. This function can be implemented using the numpy library in Python or GIS software.

[0068] According to the number of blocks and the number of available drones, assign each drone to each block.

[0069] Set to fly gradually inward from an arbitrary point on the edge of a target area to form a spiral path, and set specific flight trajectories for each drone to cover the search blocks it is responsible for.

[0070] In this embodiment, temperature estimation is performed based on the infrared image data of the drone. The temperature estimation data extracted from the infrared image data of the drone is mapped to the corresponding pixel points of the high-definition image data. Combining the target object contour information, through the detected similar contour information and the temperature information of the corresponding pixel points, analyze the actual deviation coefficient between the detected similar contour of the target object and the actual target object;

[0071] By combining the data of infrared images and high-definition images, make full use of the information of both to improve the accuracy of target detection and recognition. Infrared imaging provides information about the surface temperature of an object, which is very important for identifying certain types of target objects, such as mechanical equipment, animals, etc. Through the temperature distribution, better understand the state or behavior of the target object. Combining temperature information with visual information helps to distinguish objects with similar appearances but different thermal characteristics in a complex environment, thus improving the classification effect. By mapping the extracted temperature estimation data to the corresponding pixel points in the high-definition image, more accurate position positioning can be achieved, providing a reliable basis for subsequent tasks.

[0072] Screen the similar contour information with an actual deviation coefficient less than the preset threshold, and establish a target object tracking list according to the actual deviation coefficient of the screened similar contour information; by setting the deviation coefficient threshold, it is convenient to effectively exclude the contours that do not match the characteristics of the target object, thereby reducing the false alarm rate and improving the accuracy of tracking. Screen out the targets with high credibility for tracking, which is convenient for reducing the computational amount and storage requirements for subsequent processing and improving the system efficiency. Establish a target object tracking list based on reliable data sources, making subsequent analysis, decision-making, and operations more intuitive and easy to manage.

[0073] Move to the similar contour positions one by one according to the target tracking list, establish the tracking rules for the similar contour positions, track the target according to the tracking rules for the similar contour positions, and calculate the target position information based on the position data, attitude data of the drone that detects the target, and the three-dimensional point cloud data in front of the drone; By combining the position information, attitude information and three-dimensional point cloud data of the drone, it is convenient to improve the estimation accuracy of the target object's position, so as to achieve more accurate positioning. By dynamically acquiring and processing multi-source data, the position estimation of the target object can be adjusted in real time, and the response speed can be improved; By establishing the tracking rules for the similar contour positions, the drone can flexibly respond to environmental changes or the movement of the target object, and achieve continuous and effective tracking. According to the tracking rules, plan the optimal flight route so that the drone can efficiently cover all relevant areas, thus saving energy and time. Concentrating resources on the confirmed targets helps to improve the overall mission efficiency.

[0074] Estimate the temperature according to the infrared image data of the drone, map the temperature estimation data extracted from the infrared image data of the drone to the corresponding pixels of the high-definition image data, and combine the target object contour information. Through the detected similar contour information and the temperature information of the corresponding pixels, analyze the actual deviation coefficient between the detected similar contour of the target object and the actual target object;

[0075] Screen the similar contour information with the actual deviation coefficient less than the preset threshold, and establish a target tracking list according to the actual deviation coefficient of the screened similar contour information;

[0076] Move to the similar contour positions one by one according to the target tracking list, establish the tracking rules for the similar contour positions, track the target according to the tracking rules for the similar contour positions, and calculate the target position information based on the position data, attitude data of the drone that detects the target, and the three-dimensional point cloud data in front of the drone.

[0077] In one embodiment of the present invention, collect the attitude data of the drone and the three-dimensional point cloud data in front of the drone. At the same time, collect the infrared image data and high-definition image data in multiple directions of the drone, including the following steps:

[0078] Obtain the attitude data of the drone in real time through the IMU inertial measurement unit, collect the infrared image data in the front, below, left and right directions through the infrared thermal imaging camera, and collect the high-definition image data in the corresponding directions of the infrared thermal imaging camera through the RGB camera;

[0079] Scan the front of the drone in real time through the lidar to collect the three-dimensional point cloud data in front of the drone.

[0080] In one embodiment of the present invention, temperature estimation is performed based on the infrared image data of the drone, and the temperature estimation data extracted from the infrared image data of the drone is mapped to the corresponding pixel points of the high-definition image data, including the following steps:

[0081] Align the infrared images and high-definition image data collected simultaneously by the infrared thermal imaging camera and the RGB camera in the same direction, and map several pixel points in the corresponding high-definition image data to the corresponding pixel points in the infrared image;

[0082] Estimate the temperature of each pixel point of the infrared images in four directions of the drone obtained, and associate the obtained temperature estimation values to the pixel points of the high-definition image corresponding to the pixel points of the infrared image.

[0083] In one embodiment of the present invention, estimating the temperature of each pixel point of the infrared images in four directions of the drone obtained includes the following steps:

[0084] Collect a large number of infrared images of the same kind of substances of the target object under the same conditions in the target positioning area through an infrared camera, measure the temperature of different positions of the same kind of substances of the corresponding target object through an infrared thermometer, and record the corresponding temperature values;

[0085] Map the acquisition positions where the temperature is measured to the pixel points of the infrared image of the same kind of substances of the target object, and use the recorded corresponding temperature values as the labels of the corresponding pixel points;

[0086] Train a convolutional neural network model through the infrared image of the same kind of substances of the target object after adding labels;

[0087] Input the infrared images in four directions of the drone collected in real time into the trained convolutional neural network model to estimate the temperature of each pixel point of the infrared image.

[0088] In one embodiment of the present invention, analyzing the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel points includes the following steps:

[0089] Collect the temperature information of a large number of the same kind of substances of the target object under the same conditions in the target positioning area, set the temperature range of the target object according to the collected temperature information, and set the standard temperature value of the target object according to the mean value of the collected temperature information;

[0090] Collect a large number of high-definition image data of the same kind of substances of the target object under the same conditions in the target positioning area, and mark the contours of the same kind of substances of the target object;

[0091] Train a deep learning model through the marked high-definition image data to identify the contour information of the target object;

[0092] Input the high-definition image data collected in real time into the trained deep learning model to recognize the similar contour of the target object;

[0093] Annotate the similar contour part of the target object detected in the high-definition image data, and analyze the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel points.

[0094] In one embodiment of the present invention, analyzing the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel points includes the following steps:

[0095] Perform grayscale processing on the high-definition image of the contour of the same-kind substance of the marked target object, and set the grayscale value range of the target object according to the maximum and minimum grayscale values of each pixel point in the contour part.

[0096] Extract the contour part of the high-definition image of the same-kind substance of the target object after grayscale processing, scale it to the same size, and move all the contours to the same position in the unified coordinate system for image fusion.

[0097] Statistically calculate the grayscale value of each pixel point in the contour part image in the coordinate, and divide it by the total number of contours to obtain the average value as the standard grayscale value.

[0098] Scale the contour part of the similar contour of the target object retrieved from the high-definition image data to the same size;

[0099] Statistically calculate the grayscale value of the retrieved similar contour part of the target object, and according to the temperature estimated value corresponding to each pixel point of the similar contour part of the target object, analyze the actual deviation coefficient between the detected similar contour of the target object and the target object through the detected similar contour information and the temperature information of the corresponding pixel points, through the following formula:

[0100]

[0101] where, μ is the actual deviation coefficient, G i is the grayscale value of the i-th pixel point in the detected similar contour information, G 0 is the standard grayscale value of the i-th pixel point in the contour part image in the coordinate, T i is the temperature estimated value of the i-th pixel point in the detected similar contour information, T 0 is the standard temperature value of the target object, i ∈ (1, 2,..., n).

[0102] In one embodiment of the present invention, according to the target object tracking list, move to the similar contour position one by one to establish the similar contour position tracking rule, including the following steps:

[0103] According to the target tracking list, the UAV leaves the flight trajectory and moves towards the similar contour positions one by one, and the following similar contour position tracking rules are established:

[0104] According to the UAV attitude data and the three-dimensional point cloud data in front of the UAV, detect the distance between the similar contour position and the UAV. If the distance exceeds the preset threshold, give up moving towards the corresponding similar contour position and delete it from the target tracking list. Otherwise, move towards the similar contour position;

[0105] When the UAV is moving towards the corresponding similar contour position, analyze the actual deviation coefficient between the detected similar contour position and the target in real time according to the collected data. If the actual deviation coefficient is less than the minimum similarity threshold, stop moving towards the corresponding similar contour position and delete it from the target tracking list. Otherwise, continue to move towards the similar contour position;

[0106] If it is detected that the distance between the similar contour position and the UAV is less than the preset detection distance, and at the same time, the actual deviation coefficient is greater than the maximum similarity threshold, it is determined that the target has been locked. Otherwise, give up moving towards the corresponding similar contour position and delete it from the target tracking list;

[0107] If it is determined that the target has been locked, immediately stop other UAVs from searching for the target.

[0108] In one embodiment of the present invention, the target is tracked according to the similar contour position tracking rules, and the target position information is calculated according to the position data, attitude data of the UAV that detects the target, and the three-dimensional point cloud data in front of the UAV, including the following steps:

[0109] Track the target according to the similar contour position tracking rules, the UAV moves towards the similar contour position, and at the same time, detect the three-dimensional point cloud data in the direction of the similar contour position in front of the UAV;

[0110] Obtain the high-definition image of the similar contour of the marked target object collected in front of the UAV, map the similar contour of the marked target object into the three-dimensional point cloud data in front of the UAV, and calculate the target position information according to the position data and attitude data of the UAV that detects the target, and the mapping relationship between the similar contour and the three-dimensional point cloud data in front of the UAV.

[0111] In one embodiment of the present invention, the target position information is calculated according to the position data and attitude data of the UAV that detects the target, and the mapping relationship between the similar contour and the three-dimensional point cloud data in front of the UAV, including the following steps:

[0112] According to the GPS position and attitude data of the UAV, convert the coordinate system of the UAV itself into the world coordinate system;

[0113] Convert the acquired three-dimensional point cloud data from the lidar coordinate system to the world coordinate system;

[0114] According to the internal parameter data of the RGB camera, map the contour part coordinates of the high-definition image of the marked target object's similar contour to the world coordinate system, and calculate the depth value of each contour point in the contour part;

[0115] According to the depth value of each contour point in the contour part, calculate the average value of the distance from each contour point to the position of each contour point as the distance between the detected target object and the drone. Based on the GPS position, attitude data of the drone, and the distance between the detected target object and the drone, obtain the target object position information.

[0116] Specifically, in this embodiment, taking the current position of the drone as the origin, a local coordinate system is established. In this coordinate system, the flight direction of the drone can be defined as the X-axis, vertically downward as the Z-axis, and the Y-axis is perpendicular to the XZ plane.

[0117] According to the attitude data of the drone, including: roll angle, pitch angle, and yaw angle, calculate the attitude transformation matrix, and use the matrix to transform the position of the target object in the local coordinate system to the position in the global geodetic coordinate system.

[0118] In the local coordinate system, represent the position of the target object as a distance vector, whose length is equal to the distance measured by the sensor, and the direction is jointly determined by the orientation of the drone and the direction of the target object relative to the drone.

[0119] Use the attitude transformation matrix to transform the distance vector from the local coordinate system to the global geodetic coordinate system, and then add the GPS position of the drone to obtain the position of the target object in the global geodetic coordinate system.

[0120] As an embodiment of another aspect of the present invention, a low-altitude drone multi-source fusion positioning system is provided. This system uses a low-altitude drone multi-source fusion positioning method as described above to achieve low-altitude drone multi-source fusion positioning.

[0121] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical method of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A low-altitude UAV multi-source fusion positioning method, characterized in that: include: The target positioning area is marked on the map, and the marked target positioning area is divided into blocks. At least one UAV is assigned to each search block for target search, and a preset flight trajectory is set for each UAV. The UAV flies low over the preset position of the search block according to the flight trajectory to collect image data of the search block; Collect the real-time position data, attitude data and 3D point cloud data in front of the drone, and collect infrared image data and high-definition image data from multiple directions of the drone; The temperature is estimated based on the infrared image data of the drone. The temperature estimation data extracted from the infrared image data of the drone is mapped to the corresponding pixel points of the high-definition image data. Combined with the contour information of the target object, the detected similar contour information and the temperature information of the corresponding pixel points are used to analyze the deviation coefficient between the detected similar contour of the target object and the actual deviation coefficient of the target object. Filter similar profile information whose actual deviation coefficient is less than a preset threshold, and establish a target object tracking list according to the actual deviation coefficient of the filtered similar profile information; According to the target tracking list, move to the similar contour position one by one, establish the similar contour position tracking rules, track the target according to the similar contour position tracking rules, and calculate the target position information based on the position data, posture data of the drone that detects the target and the three-dimensional point cloud data in front of the drone.

2. A low-altitude UAV multi-source fusion positioning method according to claim 1, characterized in that: Collect the attitude data of the drone and the three-dimensional point cloud data in front of the drone. At the same time, collect the infrared image data and high-definition image data of the drone in multiple directions, including the following steps: The IMU inertial measurement unit is used to obtain the drone's attitude data in real time. The infrared thermal imaging camera is used to collect infrared image data in the four directions of front, bottom, left and right. The RGB camera is used to collect high-definition image data in the corresponding direction of the infrared thermal imaging camera. The front of the drone is scanned in real time through the laser radar to collect three-dimensional point cloud data in front of the drone.

3. The low-altitude UAV multi-source fusion positioning method according to claim 1 is characterized in that: The temperature is estimated according to the infrared image data of the drone, and the temperature estimation data extracted from the infrared image data of the drone is mapped to the corresponding pixel points of the high-definition image data, including the following steps: Align the infrared image and high-definition image data collected by the infrared thermal imaging camera and RGB camera in the same direction at the same time, and map several pixels in the corresponding high-definition image data to corresponding pixels in the infrared image; The temperature of each pixel of the infrared images acquired from four directions of the drone is estimated, and the obtained temperature estimate is associated with the pixel of the high-definition image mapped corresponding to the pixel of the infrared image.

4. The low-altitude UAV multi-source fusion positioning method according to claim 2 is characterized in that: The temperature of each pixel point of the infrared images acquired from the drone in four directions is estimated, including the following steps: Collect a large number of infrared images of the same type of target objects under the same conditions in the target positioning area through an infrared camera, measure the temperature of the same type of target objects at different positions through an infrared thermometer, and record the corresponding temperature values; Map the acquisition position for temperature measurement to the pixel point of the infrared image of the target object, and use the recorded corresponding temperature value as the label of the corresponding pixel point; The convolutional neural network model is trained by adding labeled infrared images of similar substances to the target object; The infrared images collected in four directions by the drone in real time are input into the trained convolutional neural network model to estimate the temperature of each pixel in the infrared image.

5. The low-altitude UAV multi-source fusion positioning method according to claim 1 is characterized in that: Analyzing the deviation coefficient between the detected similar contour information and the actual deviation coefficient between the detected similar contour and the target object by using the detected similar contour information and the temperature information of the corresponding pixel point, including the following steps: Collect temperature information of a large number of target objects of the same type under the same conditions in the target positioning area, set the target object temperature range according to the collected temperature information, and set the target object standard temperature value according to the average value of the collected temperature information; Collect a large amount of high-definition image data of similar target objects under the same conditions in the target positioning area, and mark the contours of similar target objects; The deep learning model is trained through labeled high-definition image data to identify the contour information of the target object; Input the high-definition image data collected in real time into the trained deep learning model to identify similar contours of the target object; The similar contour parts of the target object detected in the high-definition image data are marked, and the deviation coefficient between the detected similar contour of the target object and the actual target object is analyzed through the detected similar contour information and the temperature information of the corresponding pixel points.

6. The multi-source fusion positioning method for low-altitude UAV according to claim 5 is characterized in that: Analyzing the deviation coefficient between the detected similar contour information and the actual deviation coefficient between the detected similar contour and the target object by using the detected similar contour information and the temperature information of the corresponding pixel point, including the following steps: Grayscale processing is performed on the high-definition image of the outline of the target object of the same substance as the marked object, and the grayscale value range of the target object is set according to the maximum and minimum grayscale values ​​of each pixel point of the outline part; Extract the contour part of the high-definition image of the target object of the same substance after grayscale processing, scale it to the same size, and move all contours to the same position in the unified coordinate system for image fusion; Count the grayscale value of each pixel of the contour image in the coordinates, divide it by the total number of contours, and get the average value as the standard grayscale value; Rescaling contour portions of similar contours of the target object retrieved from the high-definition image data to the same size; The grayscale value of the similar contour part of the retrieved target object is counted, and according to the temperature estimation value corresponding to each pixel point of the similar contour part of the target object, the detected similar contour information and the temperature information of the corresponding pixel point are used to analyze the deviation coefficient between the detected similar contour of the target object and the actual target object, which is performed by the following formula: Among them, μ is the actual deviation coefficient, G i is the gray value of the i-th pixel in the detected contour-like information, G0 is the standard gray value of the i-th pixel in the contour part image in the coordinate, T i is the temperature estimation value of the i-th pixel in the detected similar contour information, T0 is the standard temperature value of the target object, i∈(1, 2, …, n).

7. The low-altitude UAV multi-source fusion positioning method according to claim 1 is characterized in that: According to the target tracking list, move to similar contour positions one by one, and establish similar contour position tracking rules, including the following steps: According to the target tracking list, the drone leaves the flight track and moves to similar contour positions one by one, and establishes the following similar contour position tracking rules: According to the drone attitude data and the 3D point cloud data in front of the drone, the distance between the similar contour position and the drone is detected. If the distance exceeds the preset threshold, the movement to the corresponding similar contour position is abandoned and deleted from the target tracking list. Otherwise, the movement to the similar contour position is performed. The drone moves toward the corresponding similar contour position and analyzes the actual deviation coefficient between the detected similar contour position and the target object in real time based on the collected data. If the actual deviation coefficient is less than the minimum similarity threshold, the drone stops moving toward the corresponding similar contour position and deletes it from the target tracking list. Otherwise, it continues moving toward the similar contour position. If the distance between the detected similar contour position and the drone is less than the preset detection distance, and the actual deviation coefficient is greater than the maximum similarity threshold, it is determined that the target has been locked. Otherwise, it abandons moving to the corresponding similar contour position and deletes it from the target tracking list. If it is determined that the target has been locked, other drones will be stopped immediately to stop the target search.

8. The low-altitude UAV multi-source fusion positioning method according to claim 1 is characterized in that: Tracking the target object according to the similar contour position tracking rule, and calculating the target object position information according to the position data, attitude data of the drone detecting the target object and the three-dimensional point cloud data in front of the drone, including the following steps: According to the similar contour position tracking rule, the target object is tracked, and the UAV moves to the similar contour position. At the same time, the three-dimensional point cloud data of the similar contour position direction in front of the UAV is detected; Obtain a high-definition image of a similar outline of a marked target object collected in front of the drone, map the similar outline of the marked target object to the three-dimensional point cloud data in front of the drone, and calculate the target object position information based on the position data and posture data of the drone that detects the target object and the mapping relationship of the similar outline in the three-dimensional point cloud data in front of the drone.

9. A low-altitude UAV multi-source fusion positioning method according to claim 8, characterized in that: According to the position data and attitude data of the drone that detects the target object, and the mapping relationship of the three-dimensional point cloud data of the similar contour in front of the drone, the target object position information is calculated, including the following steps: According to the GPS position and attitude data of the drone, the drone's own coordinate system is converted into the world coordinate system; Convert the acquired 3D point cloud data from the LiDAR coordinate system to the world coordinate system; According to the internal reference data of the RGB camera, the coordinates of the contour part of the high-definition image similar to the contour of the marked target object are mapped to the world coordinate system, and the depth value of each contour point of the contour part is calculated; According to the depth value of each contour point in the contour part, the mean distance from each contour point to each contour point position is calculated as the distance between the detected target and the UAV. According to the GPS position and attitude data of the UAV and the distance between the detected target and the UAV, the target position information is obtained.

10. A low-altitude UAV multi-source fusion positioning system, characterized in that: The system adopts a low-altitude UAV multi-source fusion positioning method as described in any one of claims 1-9 to achieve low-altitude UAV multi-source fusion positioning.

Images