Unmanned aerial vehicle edge computing system and method for object detection and visual localization

Abstract

The application relates to a UAV edge computing system and method for target detection and visual positioning, which is characterized in that a gimbal and an onboard computer are carried on a UAV, a YOLOX-based aerial target detection model and a single-computer passive visual positioning algorithm are deployed on the onboard computer, the onboard computer performs real-time processing on data information obtained by the gimbal, transmits target position information and a target key frame detected by the onboard computer back to a ground control station, and reduces network burden and rear information processing pressure. Meanwhile, the processed information can be saved on the onboard computing platform for subsequent viewing. The system can be applied to the fields of fire rescue and military reconnaissance, and can greatly improve rescue and reconnaissance efficiency, speed up decision-making, and better protect the safety of personnel.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) inspection technology, and in particular to an UAV edge computing system and method for target detection and visual positioning. Background Technology

[0002] In recent years, with the rapid advancements in manufacturing, materials, and computer information technologies, the intelligence level of unmanned aerial vehicles (UAVs) has developed at an unprecedented pace. UAVs are characterized by high maneuverability, low operational barriers, and strong autonomous control capabilities, enabling them to adapt to various complex terrains and climates, and have found widespread application in both military and civilian fields. Target detection using UAV aerial images has also become a research hotspot in the field of computer vision. UAVs equipped with visible light and infrared gimbals can collect large amounts of high-quality image data from different angles and altitudes, providing more useful information for military reconnaissance and civilian fire and rescue operations. For a long time, most scholars have treated UAVs as data acquisition devices, focusing primarily on the analysis and processing of images acquired by UAVs. With the development of artificial intelligence, the academic community has proposed more and more deep learning object detection models, such as Faster R-CNN and the YOLO series. These algorithms offer increasingly higher detection accuracy and target richness, but simultaneously, the network structures of these models are becoming more complex, requiring significant storage space and computational power. The time required for data transmission from the UAV back to the ground for processing and analysis before feedback to the user is long and inefficient. This method is limited by communication transmission capabilities, resulting in low utilization of UAV data. The interaction and feedback efficiency between the ground station and the UAV after information extraction is low, significantly impacting the accuracy of detection, tracking, and positioning algorithms that provide decision-making references, and easily missing valuable rescue and reconnaissance opportunities. Furthermore, target positioning relies on the target pixel coordinates provided by the target detection algorithm and the UAV's own pose. Therefore, equipment performance and data accuracy are crucial prerequisites for achieving high-precision target positioning. Due to limitations in communication transmission capabilities and reliability, uniformly transmitting data back to the ground station can easily lead to data loss, data mismatch, and low accuracy, greatly limiting the application scope of UAV target detection and positioning.

[0003] Currently, existing airborne processing systems developed for real-time data processing suffer from limited scalability and computational capabilities. They are also unfriendly to the detection speed and deployment of larger detection models, hindering the development of real-time artificial intelligence, image processing, and other applications on drones. To improve onboard data processing capabilities and response speed, and to help developers customize professional solutions for complex business applications, companies and research teams are continuously launching airborne computers suitable for various drone platforms. In 2019, DJI launched the Manifold 2 airborne computer specifically for drones. Equipped with an NVIDIA Jetson TX2 processor, the Manifold 2G can quickly process large amounts of onboard data, enabling image processing, target recognition, and comprehensive analysis of sensor data. However, due to its highly integrated system, the Manifold 2 only supports Ubuntu 16.04 and is now discontinued, making it unable to support higher versions and more complex deep learning models. This significantly limits the deployment of higher-performance algorithms on its platform.

[0004] In terms of algorithm design, many high-performance algorithm models for drone aerial target detection are large in size and take a long inference time, making it impossible to achieve real-time image target detection. When the detection results are sent back to the ground, a large delay will occur, resulting in low efficiency of the information feedback function of the detection results. Summary of the Invention

[0005] Therefore, it is necessary to address the technical problems of low utilization rate of aerial photography data, data transmission latency, and low scalability of existing airborne computing platforms by providing a UAV edge computing system and method for target detection and visual positioning. This system is a scalable, programmable real-time data acquisition, transmission, and processing system. Real-time target detection and visual positioning based on this system can effectively expand the intelligent applications of UAVs in various industries. Furthermore, in practical use, it can be programmed and expanded as needed, greatly freeing up manpower, reducing costs, and improving industry efficiency.

[0006] An edge computing system for unmanned aerial vehicles (UAVs) oriented towards target detection and visual positioning, the system comprising: a UAV, a gimbal, an onboard computer, an information transmission module, a data and image transmission communication module, an onboard power supply module, and a ground station.

[0007] The gimbal is mounted on the drone and is used for capturing and storing aerial images.

[0008] The information transmission module is installed on the drone and is used to expand the OSDK interface provided on the drone into a USB interface, a UART interface, and an XT30 power output interface.

[0009] The onboard computer is installed on the UAV and communicates with the UAV through the UART interface and acquires image data captured by the gimbal through the USB interface. It is used to deploy a trained YOLOX-based aerial target detection model and subscribe to the aerial image data of the UAV gimbal through the installed ROS operating system. It adopts a single-machine passive visual positioning algorithm and calculates the true position of each target in the geographic coordinate system through coordinate system transformation. The aerial target detection model is used to detect ten types of targets in the gimbal aerial images in real time.

[0010] The data transmission communication module includes a mobile terminal and a ground terminal. The mobile terminal is connected to the airborne computer, and the ground terminal is connected to the ground station via a network. The ground terminal is powered by a lithium battery.

[0011] The ground station is a laptop computer used to receive and view real-time tracking footage to assist the command center in making rapid decisions.

[0012] The airborne power supply module is installed on the UAV and is used to supply power to the airborne computer and the mobile terminal for data transmission.

[0013] A UAV edge computing method for target detection and visual positioning is disclosed. The method is applied to the onboard computer of any of the aforementioned UAV edge computing systems for target detection and visual positioning, enabling real-time detection and positioning of aerial targets for ground photography by the UAV onboard platform. The method includes:

[0014] Open Nomochine to remotely connect to the onboard computer and confirm that the onboard computer and the drone are communicating normally.

[0015] Obtain the position coordinates of the drone at the takeoff location after it has been prepared for takeoff.

[0016] Run the positioning detection program and subscribe to the video stream from the drone pod camera and the drone's pose data.

[0017] The video stream from the subscribed drone pod camera is converted to a different format. The resulting images are then input into a trained YOLOX-based aerial target detection model to detect targets in each image, thus obtaining the target category and pixel position of each target in each frame.

[0018] Based on the pixel position of each target in the image, a single-machine passive visual positioning algorithm is used to detect the true position of each target in the geographic coordinate system through coordinate system transformation.

[0019] Each image is labeled according to the target category and the actual location of each target in the geographic coordinate system, and the labeled image with the detection and positioning information is transmitted to the ground station.

[0020] The aforementioned edge computing system and method for UAVs, oriented towards target detection and visual positioning, involves an onboard UAV equipped with a gimbal and an onboard computer. The onboard computer deploys a YOLOX-based aerial target detection model and a single-unit passive visual positioning algorithm. The onboard computer processes data acquired by the gimbal in real time, transmitting detected target location information and keyframes back to the ground control station, thus reducing network load and the pressure on rear intelligence processing. Simultaneously, the processed information can be saved to the onboard computing platform for later review. This system can be applied in fields such as fire rescue and military reconnaissance, significantly improving rescue and reconnaissance efficiency, accelerating decision-making, and better protecting personnel safety. Attached Figure Description

[0021] Figure 1 This is a physical diagram of an unmanned aerial vehicle (UAV) edge computing system for target detection and visual localization in one embodiment.

[0022] Figure 2 This is a hardware configuration diagram of an unmanned aerial vehicle (UAV) edge computing system for target detection and visual localization in one embodiment.

[0023] Figure 3 This is a flowchart illustrating a UAV edge computing method for target detection and visual localization in one embodiment.

[0024] Figure 4 This is a schematic diagram of an edge computing method for a UAV oriented to visual positioning in another embodiment;

[0025] Figure 5 This is a flowchart illustrating a UAV edge computing method for target detection and visual localization in another embodiment;

[0026] Figure 6 This is the detection effect under noon light conditions in another embodiment;

[0027] Figure 7 This is the detection effect under nighttime lighting conditions in another embodiment;

[0028] Figure 8 For another embodiment, the detection effect under foggy weather conditions

[0029] Figure 9 This is an example of the detection effect under rainy weather conditions;

[0030] Figure 10 This is the detection effect under a dense number of targets in another embodiment;

[0031] Figure 11 This is the detection performance with a sparse number of targets in another embodiment;

[0032] Figure 12 This is the detection effect against the background of a bridge deck in another embodiment;

[0033] Figure 13 This is an example of detection performance against a sports field background.

[0034] Figure 14 In another embodiment, the real-time detection positioning effect is shown. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0036] In one embodiment, such as Figure 1 , Figure 2 As shown, an edge computing system for UAVs oriented towards target detection and visual positioning is provided. The system includes: UAV 1, gimbal 2, onboard computer 3, information transmission module 4, data and image transmission communication module 5, onboard power supply module 6, and ground station 7.

[0037] The gimbal 2 is mounted on the drone 1 and is used for capturing and storing aerial images.

[0038] Information transmission module 4 is installed on UAV 1 to expand the OSDK interface provided on UAV 1 into a USB interface, a UART interface, and an XT30 power output interface. Preferably, information transmission module 4 uses an OSDK expansion component, which expands the OSDK interface provided on the M300 RTK UAV into three standard interfaces: USB Type-A, UART serial port, and XT30 power output interface. Onboard computer 3 communicates with UAV 1 via the UART interface and acquires image data captured by the gimbal mounted on UAV 1 via the USB interface.

[0039] An airborne computer 3 is installed on the drone 1. The airborne computer 3 communicates with the drone 1 through a UART interface and acquires image data captured by the gimbal 2 through a USB interface. The airborne computer 3 is used to deploy a trained YOLOX-based aerial target detection model and subscribes to the aerial image data of the drone gimbal through the installed ROS operating system. It adopts a single-machine passive visual positioning algorithm and calculates the true position of each target in the geographic coordinate system through coordinate system transformation. The aerial target detection model is used to detect ten types of targets in the aerial images captured by the gimbal in real time.

[0040] Specifically, due to the limited computing power of the onboard computer, the YOLOX-s detection network was chosen to balance detection performance and speed. The inference model used was trained on the VisDrone aerial detection dataset, supporting the detection of ten target categories: pedestrians, people, cars, vans, buses, trucks, motorcycles, bicycles, awning tricycles, and tricycles. To achieve real-time detection, the trained high-performance detection model was converted into a high-performance inference model supported by the TensorRT inference engine and deployed on the onboard computer module. By setting the confidence level and NMS (non-maximum suppression) value, the inference model can detect targets and draw bounding boxes and label categories for targets in each frame, enabling real-time detection of ten target categories in aerial images captured by the UAV camera.

[0041] The onboard computer is also used to perform target localization tasks using a single-machine passive visual localization algorithm based on the target detection results. This algorithm calculates the true location of each detected target in the geographic coordinate system through coordinate system transformation. Before takeoff, the latitude, longitude, and absolute altitude of the UAV's takeoff origin are obtained. After a target is detected, the UAV's position and attitude information at the time of the image frame containing the target, along with the target's pixel position in the image, are used to calculate the target's true geographical location.

[0042] The data transmission communication module 5 includes a mobile terminal 501 and a ground terminal 502. The mobile terminal 501 is connected to the airborne computer, and the ground terminal 502 is connected to the ground station via a network. The airborne computer 1 and the ground station 7 are configured to be on the same local area network. The ground station 1 can receive the real-time image from the airborne computer 1 by opening Nomochine for remote connection. The ground terminal of the data transmission is powered by a lithium battery.

[0043] Ground station 7 is a laptop computer used for real-time reception and viewing of tracking footage to assist the command center in making rapid decisions. Preferably, ground station 7 uses a ThinkPad T470 laptop computer.

[0044] The airborne power supply module 6 is installed on the UAV 1 to supply power to the airborne computer 3 and the mobile terminal 501 for data transmission.

[0045] This system is a scalable edge computing real-time detection and tracking system. Compared to traditional methods that transmit data to a ground station for processing, this system offers superior real-time performance. It can transmit real-time calculation results to the ground and save detection results to the onboard computer for later retrieval. Compared to transmitting data to the ground for detection, the data is more secure and complete, the algorithm accuracy is better, it is unaffected by the resolution of the transmission link, and the detection accuracy is higher. Combining target detection technology and spatial 3D passive visual positioning technology, it achieves real-time and accurate target detection and positioning, transmitting target point categories and real-time locations back to the ground station to assist the ground command center in making rapid and efficient decisions. This system enables real-time detection and positioning of aerial targets by an airborne platform, completes the counting and statistical tasks of detected targets within the field of view, and maintains a positioning accuracy within 3 meters when the UAV is flying at an altitude of 100m.

[0046] The aforementioned UAV edge computing system for target detection and visual positioning integrates a gimbal and an onboard computer. The onboard computer deploys a YOLOX-based aerial target detection model and a single-unit passive visual positioning algorithm. The onboard computer processes data acquired by the gimbal in real time, transmitting detected target location information and keyframes back to the ground control station, reducing network load and the pressure on rear intelligence processing. Simultaneously, the processed information can be saved to the onboard computing platform for later review. This system can be applied in fields such as fire rescue and military reconnaissance, significantly improving rescue and reconnaissance efficiency, accelerating decision-making, and better protecting personnel safety.

[0047] Furthermore, the drone is a DJI M300RTK drone.

[0048] Specifically, the DJI M300RTK drone is a drone with six-way positioning and obstacle avoidance and a visual flight assistance interface. It meets the IP45 protection level, supports an ultra-long flight time of 55 minutes and a maximum image transmission distance of 15 kilometers. It can support 3 payloads at the same time and has rich SDK open interfaces.

[0049] Furthermore, the gimbal is the Zenmuse H20T gimbal.

[0050] Specifically, the Zenmuse H20 gimbal integrates three sensors: a 20-megapixel zoom camera, a 12-megapixel wide-angle camera, and a laser rangefinder with a detection range of up to 1200 meters, providing intuitive visible light and thermal imaging. When paired with the DJI 300RTK drone, it enables intelligent functions such as large-spot positioning, intelligent tracking, repeat shooting, and ultra-high-definition matrix photography.

[0051] Furthermore, the onboard computer is an edge computing device, the NVIDIA Jetson AGX Orin.

[0052] Specifically, the onboard computer is NVIDIA Jetson AGXOrin, NVIDIA's latest edge computing device, featuring an NVIDIA Ampere architecture with 2048 NVIDIA CUDA cores and 64 Tensor Cores, boasting a computing power of up to 275 TOPS and a power configuration ranging from 15W to 60W. It runs JetPack 5.0, including a root file system based on Ubuntu 20.04, and features the latest computing stack of CUDA 11.4, TensorRT 8.4, and cuDNN 8.3. This is used to deploy a trained deep learning model for object detection and to subscribe to drone camera data via the installed ROS operating system to perform real-time inference detection and localization of gimbal-captured aerial image data.

[0053] Furthermore, the data transmission communication module serves as the mobile and ground terminals for ZGET Homer data transmission.

[0054] Specifically, ZGET Homer digital image transmission is used for long-distance communication between UAVs and ground stations, with a communication distance of 2 to 5 kilometers.

[0055] Furthermore, the airborne power supply module is a DJI multi-function power module and a 24V to 12V voltage regulator module.

[0056] In one embodiment, such as Figure 3 As shown, a UAV edge computing method for target detection and visual positioning is provided. This method is applied to the onboard computer of any of the aforementioned UAV edge computing systems for target detection and visual positioning, enabling real-time detection and positioning of aerial targets for ground photography by the UAV onboard platform. The method includes:

[0057] Step 300: Open Nomochine to remotely connect to the onboard computer and confirm that the onboard computer and the drone are communicating normally.

[0058] Specifically, the onboard computer, NVIDIA Jetson AGX Orin, and the ground station ThinkPad T470 laptop are configured to be on the same local area network. The T470 laptop uses Nomochine software to connect to the remote desktop of the onboard computer.

[0059] Edit the .launch configuration file in OSDK ROS, set the DJI OSDK user ID, key, and custom baud rate, and check the serial port connection status to confirm that communication between the onboard computer and the drone has been successfully established.

[0060] Step 302: Obtain the position coordinates of the drone at the takeoff location after it has been prepared for takeoff.

[0061] Specifically, the pre-flight preparations for the drone include: connecting the drone handle to the network and setting up the network RTK service (using the carrier phase dynamic real-time differential positioning method, the service provider can provide the three-dimensional positioning results of the observation point in the specified coordinate system in real time and quickly, and achieve centimeter-level accuracy).

[0062] After the UAV is ready for takeoff, obtain the WGS84 coordinate data (B, L, H) of the UAV takeoff point O, which is (0, 0, 0) in the northeast-northeast coordinate system.

[0063] Step 304: Run the positioning detection program and subscribe to the drone pod camera video stream and drone pose data.

[0064] Specifically, in the ROS environment, open a terminal, run the OSDK ROS master node, and publish drone aerial image messages, drone WGS84 location messages, and gimbal attitude messages; in the ROS environment, open another terminal and run the target detection and localization node; subscribe to the drone aerial image messages, drone WGS84 location messages, and gimbal attitude message data published by the master node, and use the ApproximateTimeSynchronizer method in ROS to synchronize the time of the three types of messages; pass the synchronized subscribed messages to the callback function.

[0065] Step 306: Convert the format of the subscribed drone pod camera video stream, input the resulting image into the trained YOLOX-based aerial target detection model, detect the target in each image, and obtain the target category and the pixel position of the target in each frame.

[0066] Step 308: Based on the pixel position of each target in the image, a single-machine passive visual positioning algorithm is used to detect the true position of each target in the geographic coordinate system through coordinate system transformation.

[0067] Step 310: Based on the target category and the actual location of each target in the geographic coordinate system in each frame of the image, label each image and transmit the labeled image with the detection and positioning information to the ground station.

[0068] Specifically, using the target prediction results from step 306, with the top left corner of the target as the origin, the position of each target is clearly defined in the image by drawing a bounding box. The target category and the true relative geographical location of the target predicted in step 308 are marked in the top left corner of the target box. The image with the marked detection and positioning information is then transmitted back to the ground station.

[0069] In one embodiment, the YOLOX-based aerial target detection model is a PyTorch detection model trained on the VisDrone detection dataset by YOLOX-s, targeting ten types of targets: pedestrians, people, cars, vans, buses, trucks, motorcycles, bicycles, awning-tricycles, and tricycles. The model is converted to ONNX format and accelerated into an engine weight model using TensorRT.

[0070] The subscribed drone pod camera video stream is format-converted, and the resulting images are input into a trained YOLOX-based aerial target detection model. Targets in each image are detected, and the target category and pixel location in each frame are obtained. Step 306 includes: converting the subscribed drone pod camera video stream from ROS format to CV format using the cv_bridge library; inputting the format-converted images into the trained YOLOX-based aerial target detection model; extracting features at different scales in the Backbone layer; fusing these extracted features in the Neck layer; and finally inferring the target location and category in the Head layer; setting prediction confidence and non-maximum suppression thresholds; sorting all predicted boxes by probability; retaining the predicted box with the highest probability if it meets a preset condition compared to other boxes; iterating until the target box with the lowest probability is reached; and using all retained predicted boxes as target boxes. The preset condition is:

[0071] IOU(bbox_max, bbox_cur) > NMS

[0072] Where bbox_max is the predicted bounding box with the highest probability, bbox_cur is the other bounding boxes, and NMS is the non-maximum suppression threshold.

[0073] Calculate the confidence score for all bounding boxes. If the confidence score is greater than the prediction confidence, the bounding box is taken as the final prediction result, thus completing the object detection task.

[0074] In one embodiment, step 308 includes: obtaining the position (x, y) of the center point of each target in the image pixels, and obtaining the WGS84 coordinate position information (B) of the UAV in the current frame. S L S H S The system obtains gimbal pitch, yaw, and roll angle information (roll, yaw, pitch) and gimbal camera focal length f and pixel size pixdis information. Based on the current frame image information and the pixel position information of each target, a single-machine passive visual positioning algorithm is used to detect the true position of each target in the geographic coordinate system through coordinate system transformation.

[0075] Specifically, the algorithm obtains the pixel position of the center point of each target in the image, the WGS84 coordinate position information of the UAV in the current frame, the pitch, yaw and roll angle information of the gimbal, and the focal length and pixel size information of the gimbal camera; the image information of the current frame and the pixel position information of each target are input into the single-machine passive vision positioning algorithm, and the true position of each target is obtained by combining the takeoff point coordinate data, thus completing the target positioning task.

[0076] In one embodiment, locating a target in a drone aerial image essentially involves determining the transformation relationship between the onboard camera coordinate system and the world coordinate system. For the drone target localization problem, multiple coordinate systems are established and their transformation relationships are determined. A schematic diagram of the single-machine passive visual localization algorithm is shown below. Figure 4 As shown. The stand-alone passive vision positioning algorithm specifically includes: converting the coordinate unit from pixels to the corresponding physical distance using a coordinate unit conversion formula; the coordinate unit conversion formula is:

[0077]

[0078] Where (x, y) represents the position of the pixel in the image, (x', y') represents the physical coordinates of the pixel in a coordinate system with the camera center as the origin, and φ x φ y These represent the pixel size in the x and y directions of the image, respectively.

[0079] Calculate the northeast coordinates of the image point N of target point M in the camera coordinate system. The formula for calculating the northeast coordinates of the image point of the target point in the camera coordinate system is:

[0080]

[0081] Where (roll, yaw, pitch) represent the gimbal attitude information: roll angle, yaw angle, and pitch angle, and R... t R X R Y R Z This is the coordinate transformation matrix.

[0082] The coordinates of the UAV's position S at time T on the geodetic coordinate system (B S L S H S Convert to Northeast Geodetic Coordinate System The conversion formula from the geodetic coordinate system to the northeast geodetic coordinate system is as follows:

[0083]

[0084]

[0085] Where, a is the semi-major axis of the WGS-84 Earth ellipse, a = 6378137 m, (B, L, H) are the coordinates in the geodetic coordinate system, and e is the eccentricity. 2 =0.00669437999013, (X, Y, Z) are the coordinates in the spatial rectangular coordinate system.

[0086] Calculate the northeast coordinates of target point M, i.e., the actual location of the target point; the expression for calculating the actual location of the target point is:

[0087]

[0088] in, This represents the actual location of the target point M.

[0089] In one embodiment, such as Figure 5 As shown, an edge computing method for UAVs oriented towards target detection and visual localization is provided, and the specific steps include:

[0090] The first step is to open Nomochine and remotely connect to the onboard computer NVIDIA Jetson AGX Orin to confirm that the Orin is communicating normally with the drone.

[0091] (1) Configure the onboard computer NVIDIA Jetson AGX Orin and the ThinkPad T470 laptop to be on the same local area network. The T470 laptop opens the Nomochine software to connect to the Orin remote desktop.

[0092] (2) Edit the .launch configuration file in OSDK ROS, set the DJI OSDK user ID, key and the baud rate configured by yourself, and check the serial port connection status to confirm that Orin and the drone have successfully established communication.

[0093] The second step is for the drone to acquire various data from the takeoff point after it has completed its pre-takeoff preparations.

[0094] (1) Connect the drone handle to the network and set up the network RTK service to receive network RTK signals;

[0095] (2) Obtain the WGS84 coordinate data of point O, the takeoff location of the UAV;

[0096] The third step is to run the detection and positioning program and subscribe to the video stream from the drone pod camera and the drone's pose data.

[0097] (1) Open the terminal in the ROS environment, run the OSDK ROS master node, and publish drone aerial image messages, drone WGS84 location messages and gimbal attitude messages.

[0098] (2) Open a new terminal in the ROS environment and run the target detection and localization node;

[0099] (3) Subscribe to the drone aerial image messages, drone WGS84 location messages and gimbal attitude message data published by the master node, and use the .ApproximateTimeSynchronizer method in ROS to synchronize the three types of messages in time;

[0100] (4) Pass the synchronized subscription message to the callback function;

[0101] The fourth step is to detect ten types of targets in each aerial image to obtain the target category and pixel position in each frame.

[0102] (1) Load the trained YOLOX-based aerial target detection model. This model is a PyTorch detection model trained by YOLOX-s on the VisDrone detection dataset for ten target categories: pedestrians, people, cars, vans, buses, trucks, motorcycles, bicycles, awnings-tricycles, and tricycles. Convert the model to ONNX format and use TensorRT to accelerate the conversion to an .engine weighted model.

[0103] (2) Convert the subscribed image message from ROS format to CV format, read the image and call the aerial target detection model, extract features of different scales of the image in the Backbone, fuse these extracted features in the Neck layer, and finally infer the location and category of the target contained in the image in the Head part.

[0104] (3) Set two values: prediction confidence and non-maximum suppression, to obtain the final prediction result of the target in the frame image and complete the target detection task.

[0105] The fifth step involves inputting the pixel positions of each target in the image predicted in the fourth step into the visual localization algorithm to calculate the true geographical location of each target.

[0106] (1) Obtain the position of the center point of each target in the image pixel, and obtain the WGS84 coordinate position information of the UAV in the current frame, the pitch, yaw and roll angle information of the gimbal, and the focal length and pixel size information of the gimbal camera.

[0107] (2) Input the current frame image information and the position information of each target pixel into the single-machine passive vision positioning algorithm, and combine the take-off point coordinate data to obtain the real position of each target and complete the target positioning task.

[0108] The sixth step is to display the obtained detection and positioning results in real-time aerial images and transmit them to the ground station.

[0109] (1) Using the target results predicted in step four, draw a frame in the image to define the position of each target, with the upper left corner of the target as the origin. Mark the target category and the true relative geographical location of the target predicted in step five in the upper left corner of the target frame.

[0110] (4) Transmit the image with the detection and positioning information marked back to the ground station.

[0111] It should be understood that, although Figure 3 and Figure 5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 3 and Figure 5 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0112] In a verification embodiment, the UAV edge computing method for target detection and visual positioning was experimentally verified based on the built UAV edge computing system under different environments and weather conditions.

[0113] (1) Target detection experimental results

[0114] The target detection model trained by this invention can identify ten types of targets, including pedestrians, people, cars, vans, buses, trucks, motorcycles, bicycles, awning tricycles, and tricycles. It is robust in detecting and counting targets under different density levels, different environmental backgrounds, and different lighting conditions.

[0115] Figures 6-14 This embodiment demonstrates the detection and counting performance of the target-oriented detection UAV edge computing method for detecting targets of interest on the ground in different scenarios. Figure 6 , Figure 7 The test results were shown under different lighting conditions at noon and at night. Figure 8 , Figure 9 The test results were shown under different weather conditions, namely foggy and rainy days. Figure 10 , Figure 11 The detection performance is shown for different numbers of dense and sparse targets. Figure 12 , Figure 13 The test results were shown on the bridge surface and the sports field under different backgrounds.

[0116] Table 1 shows the detection accuracy of the edge computing method for target detection UAV designed in this invention for each of ten target categories, including automobiles:

[0117] Table 1 Detection accuracy of various targets

[0118]

[0119]

[0120] The target detection results and detection accuracy data show that the edge computing method for UAVs designed in this invention can effectively achieve target detection tasks from the perspective of the UAV. It has good detection results for ground targets captured by the UAV in various scenarios. The target detection bounding box and image positioning results are accurate, laying the foundation for accurate tracking and positioning in the future.

[0121] (2) Experimental Results of Target Detection and Localization

[0122] This invention selects a vehicle target for positioning accuracy testing. The attitude (roll, yaw, pitch) of the UAV is acquired at eight altitudes: 25m, 50m, 75m, 100m, 125m, 150m, 175m, and 200m. The geodetic coordinates (B, L, H) of the UAV at each altitude are also acquired. After each image is input into the target detection model, the center point position of the white vehicle target is obtained. The true coordinates of the vehicle are calculated by a visual positioning algorithm. The true position of the vehicle is obtained by the UAV RTK measurement.

[0123]

[0124] Table 2 shows the localization RMSE error of visual positioning on images at various altitudes in the edge computing method for UAVs that calculates target detection and visual positioning.

[0125] Table 2 Positioning RMSE Error at Different Heights

[0126] Height (m) 25m 50m 75m 100m 125m 150m 175m 200m RMSE 0.675 1.436 2.413 2.984 3.219 3.799 4.294 5.141

[0127] The system's real-time detection and positioning effect is as follows: Figure 14 As shown, this invention uses a single-machine passive visual positioning method, which ensures real-time positioning of the detected target while achieving a positioning error within 3m under the condition that the UAV flies at an altitude of 100m.

[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A UAV edge computing method for target detection and visual localization, characterized in that, The method includes: Open Nomochine to remotely connect to the onboard computer and confirm that the onboard computer and the drone are communicating normally. Obtain the position coordinates of the drone at its takeoff location after it has been prepared for takeoff. Run the positioning detection program and subscribe to the video stream from the drone pod camera and the drone's pose data; The video stream from the subscribed drone pod camera is converted to a different format. The resulting images are then input into a trained YOLOX-based aerial target detection model to detect targets in each image, thus obtaining the target category and pixel position of each target in each frame. Based on the pixel position of each target in the image, a single-machine passive visual positioning algorithm is used to detect the true position of each target in the geographic coordinate system through coordinate system transformation. Based on the target category and the actual location of each target in the geographic coordinate system in each frame of the image, each image is labeled, and the labeled image with the detection and positioning information is transmitted to the ground station; Specifically, the single-machine passive visual positioning algorithm includes: The coordinate unit is converted from pixels to the corresponding physical distance in meters using a coordinate unit conversion formula; the coordinate unit conversion formula is as follows: in, Indicates the position of a pixel on the image. This represents the physical coordinates of the pixel in a coordinate system with the camera center as the origin. Representing the image x direction and y Pixel size in orientation; Calculate the northeast coordinates of the image point N of target point M in the camera coordinate system. The formula for calculating the northeast coordinates of the image point of the target point in the camera coordinate system is: in, For gimbal attitude information, roll angle, yaw angle, and pitch angle. This is the coordinate transformation matrix; The coordinates of the UAV's position point S in the geodetic coordinate system at time T. Convert to Northeast Geodetic Coordinate System The conversion formula from the geodetic coordinate system to the northeast geodetic coordinate system is: in, a This refers to the semi-major axis of the Earth's ellipse as defined by WGS-84. e For eccentricity, These are coordinates in the geodetic coordinate system. The coordinates are in a spatial rectangular coordinate system; Calculate the northeast coordinates of target point M, i.e., the actual location of the target point; the expression for calculating the actual location of the target point is: in, This represents the actual location of the target point M.

2. The method according to claim 1, characterized in that, The aerial target detection model based on YOLOX is a PyTorch detection model trained on a pre-set detection dataset by YOLOX-s, targeting ten types of targets: pedestrians, people, cars, vans, buses, trucks, motorcycles, bicycles, sunshade tricycles, and tricycles. The model is converted to ONNX format and accelerated by TensorRT to be converted into an engine weight model. The video stream from the subscribed drone pod camera is converted to a different format. The resulting images are then input into a trained YOLOX-based aerial target detection model. This model detects targets in each image, obtaining the target category and pixel location for each frame, including: Convert the subscribed drone pod camera video stream from ROS format to CV format using the cv_bridge library; The converted image is input into a trained YOLOX-based aerial target detection model. Features at different scales of the image are extracted in the Backbone, these extracted features are fused in the Neck layer, and finally the target location and category in the Head part are inferred. Set the prediction confidence level and the non-maximum suppression threshold; All predicted bounding boxes are sorted by probability. For the predicted target box with the highest probability, it is compared with other boxes. If a preset condition is met, the predicted bounding box is retained. This process is repeated until the target box with the lowest probability is reached. All retained predicted bounding boxes are then used as the target boxes. The preset condition is: IOU(bbox_max, bbox_cur)>NMS Where bbox_max is the predicted bounding box with the highest probability, bbox_cur is other boxes, and NMS is the non-maximum suppression threshold; Calculate the confidence score for all bounding boxes. If the confidence score is greater than the prediction confidence, the bounding box is taken as the final prediction result; thus completing the object detection task.

3. The method according to claim 1, characterized in that, Based on the pixel positions of each target in the image, a single-machine passive visual localization algorithm is used to detect the true position of each target in the geographic coordinate system through coordinate system transformation, including: Obtain the pixel position of the center point of each target in the image. x , y ), obtain the WGS84 coordinate position information of the drone in the current frame (B S L S H S Gimbal pitch, yaw, and roll angle information and the focal length of the gimbal camera f Pixel size information (pixdis); Based on the current frame image information and the position information of each target pixel, a single-machine passive visual positioning algorithm is used to detect the true position of each target in the geographic coordinate system through coordinate system transformation.

4. A UAV edge computing system for target detection and visual positioning, characterized in that, The airborne computer in the system adopts the UAV edge computing method for target detection and visual positioning as described in any one of claims 1-3 to realize real-time detection and positioning of ground aerial targets by the UAV airborne platform. The system includes: UAV, gimbal, airborne computer, information transmission module, data and image transmission communication module, airborne power supply module and ground station. The gimbal is mounted on the drone and is used for capturing and storing aerial images; The information transmission module is installed on the drone and is used to expand the OSDK interface provided on the drone into a USB interface, a UART interface and an XT30 power output interface. The onboard computer is installed on the UAV and communicates with the UAV via the UART interface. It acquires image data captured by the gimbal via the USB interface. The computer is used to deploy a pre-trained YOLOX-based aerial target detection model and subscribes to aerial image data from the UAV gimbal through the installed ROS operating system. It employs a single-machine passive visual positioning algorithm and calculates the true location of each target in the geographic coordinate system through coordinate system transformation. The aerial target detection model is used to perform real-time detection of ten types of targets in the gimbal's aerial images. The data transmission communication module includes a mobile terminal and a ground terminal for data transmission. The mobile terminal is connected to the airborne computer, and the ground terminal for data transmission is connected to the ground station via a network. The ground terminal for data transmission is powered by a lithium battery. The ground station is a laptop computer used to receive and view real-time tracking images to assist the command center in making rapid decisions. The airborne power supply module is installed on the UAV and is used to supply power to the airborne computer and the mobile terminal for data transmission.

5. The system according to claim 4, characterized in that, The drone in question is a DJI M300RTK drone.

6. The system according to claim 4, characterized in that, The gimbal in question is the Zenmuse H20T gimbal.

7. The system according to claim 4, characterized in that, The onboard computer is an edge computing device, NVIDIA Jetson AGX Orin.

8. The system according to claim 4, characterized in that, The data transmission communication module is the mobile terminal and ground terminal of ZGET Homer data transmission.

9. The system according to claim 4, characterized in that, The airborne power supply module is a DJI multi-function power module and a 24V to 12V voltage regulator module.

Images