A Distributed Visual Guidance System for Unmanned Aerial Vehicles and Its Image Processing Method

Abstract

This application discloses a distributed visual guidance system for unmanned aerial vehicles (UAVs) and its image processing method, belonging to the field of UAV technology. The distributed visual guidance system consists of multiple front-end image processing units and a central fusion processing unit connected via a system bus. The front-end units are responsible for image acquisition and lightweight target detection, and upload the structured encapsulation of the recognition results. The central unit is responsible for time synchronization, spatial alignment, and fusion of data from multiple front-end units, and performs complex multi-target tracking. This disclosure significantly reduces system bandwidth requirements and central computing load through distributed collaborative computing, enabling real-time, efficient, and intelligent processing of multiple high-definition video streams on resource-constrained UAV platforms.

Description

Technical Field

[0001] This application belongs to the field of unmanned aerial vehicle (UAV) technology, and in particular relates to a distributed vision guidance system for UAVs and its image processing method. Background Technology

[0002] With the development of drone technology, the requirements for visual guidance systems are becoming increasingly stringent, necessitating the ability to process multiple high-definition video streams in real time to achieve rapid identification and stable tracking of large-scale, multi-target applications. Traditional solutions primarily employ two architectures: 1. Centralized Processing Architecture: The raw video streams captured by all cameras are directly transmitted to a central processing unit (such as a high-performance onboard computer) for processing. The disadvantages of this architecture are: extremely high data bandwidth requirements, with multiple high-definition video streams putting enormous pressure on bus bandwidth and the central processing unit interface; excessive computational load on the central processing unit, as it simultaneously processes detection and tracking algorithms for multiple video streams, making it difficult to guarantee real-time performance, and resulting in high power consumption and heat generation.

[0003] 2. Fully Edge Processing Architecture: All target detection and tracking calculations are performed at each camera node, and the recognition results are then uploaded. While this architecture reduces the burden on the central processing unit, it places extremely high demands on the computing power of each edge node. The cost, power consumption, and size of edge nodes are difficult to control, and it is also difficult to achieve effective collaboration and fusion of data between multiple nodes.

[0004] Therefore, existing technologies cannot efficiently complete the real-time intelligent processing of multiple high-definition videos under the constraints of limited airborne resources (power consumption, weight, computing power). It is necessary to provide a new UAV distributed vision guidance system and its image processing method to solve the above-mentioned technical problems. Summary of the Invention

[0005] The purpose of this disclosure is to provide a distributed vision guidance system for unmanned aerial vehicles and its image processing method in order to solve the above-mentioned problems.

[0006] This disclosure achieves the above objectives through the following technical solutions: A distributed vision guidance system for unmanned aerial vehicles (UAVs) includes a central fusion processing unit and multiple front-end image processing units connected to it via a system bus. The front-end image processing unit is connected to an image sensor for acquiring raw image data and performing target recognition on the raw image data. The recognition results are then encapsulated in a structured manner to form a structured data packet, which is then sent to the central fusion processing unit via the system bus. The central fusion processing unit is used to receive and parse each structured data packet, perform time synchronization and data fusion on the parsed data, and run a multi-target tracking algorithm to output the target's tracking trajectory.

[0007] As a further optimization of this disclosure, the front-end image processing unit includes a SoC chip integrating a lightweight object detection model, which is used to perform object recognition on the original image data to obtain recognition results.

[0008] As a further optimization of this disclosure, the identification result includes one or more of the following: target location information, category information, and feature vector.

[0009] As a further optimization of this disclosure, the lightweight object detection model is a deep learning model that has undergone quantization or pruning.

[0010] As a further optimization of this disclosure, the central fusion processing unit is also used to convert the target position information identified by each front-end image processing unit into a unified global coordinate system based on the installation parameters of each image sensor, thereby completing spatial alignment.

[0011] An image processing method applied to the aforementioned distributed vision guidance system for unmanned aerial vehicles includes the following steps: The front-end image processing unit collects raw image data, performs target detection to obtain recognition results, encapsulates the recognition results in a structured manner to form a structured data packet, and sends it to the central fusion processing unit. The central fusion processing unit receives all the structured data packets, performs time synchronization and data fusion, and runs a multi-target tracking algorithm to output the tracking results.

[0012] As a further optimization of this disclosure, structured encapsulation refers to packaging the identification results together with the timestamp and device identifier into a structured data packet.

[0013] As a further optimization of this disclosure, the data fusion includes: synchronizing all identification results based on timestamps, and unifying the coordinates of the target location based on preset spatial transformation parameters.

[0014] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to execute the program stored in the memory to implement the image processing method described above.

[0015] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method.

[0016] The beneficial effects of this disclosure are as follows: (1) Significantly reduce system bandwidth requirements: The front-end unit only uploads structured metadata (such as coordinates, categories, feature vectors), which reduces the amount of data by several orders of magnitude compared to the original video stream, thus completely solving the bandwidth bottleneck problem of multi-channel high-definition video transmission.

[0017] (2) Optimize the allocation of computing resources: Distribute the computationally intensive detection tasks to multiple low-power front-end units, and concentrate the more complex tracking and fusion tasks to the central unit, thereby achieving efficient collaboration of computing power and effectively controlling the overall system power consumption and cost.

[0018] (3) Improve system real-time performance: Since the central unit does not need to process the original image, it only needs to process the lightweight structured data, enabling it to complete multi-target tracking and decision-making more quickly, and the system response delay is lower.

[0019] (4) Enhanced system scalability: The architecture allows the visual perception range to be expanded by adding front-end units without significantly modifying the central processing logic, resulting in strong system scalability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a system structure block diagram of an embodiment of this disclosure; Figure 2 This is a general flowchart of the image processing method in the embodiments of this disclosure; Figure 3 This is a detailed flowchart of the internal workings of the central fusion processing unit in an embodiment of this disclosure; Figure 4 This is a structural block diagram of an embodiment of this disclosure. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] like Figure 1As shown, a distributed vision guidance system for unmanned aerial vehicles includes: At least two front-end image processing units, one central fusion processing unit, and a system bus; The front-end image processing unit is connected to an image sensor to acquire raw image data and run a lightweight target detection model to perform target recognition on the acquired images. The front-end image processing unit is configured to: not upload the raw image data, but encapsulate the recognition results in a structured manner to generate a data packet containing one or more of the target location information, category information, and feature vectors, and upload it to the central fusion processing unit through the system bus. The central fusion processing unit is connected to the front-end image processing unit via a system bus. It is used to receive structured data packets uploaded by all front-end image processing units and to perform time synchronization, spatial alignment, and data fusion on all structured data packets. The central fusion processing unit runs complex target tracking algorithms and completes multi-target cross-view continuous tracking and global positioning based on the fused data.

[0024] The drone platform provides support and power to the system, while the central fusion processing unit outputs the final target tracking results.

[0025] The data packet structure is as follows:

[0026] A structured data packet is a structured collection that contains: Packet header: contains timestamp and device ID.

[0027] Payload: A list containing identification information for one or more targets. Information for each target consists of the following fields: Target ID (optional): Used for front-end association.

[0028] Bounding box coordinates: The position of the target in the image (e.g., x, y, width, height).

[0029] Category label and confidence level: The identified target category and its confidence level.

[0030] Appearance feature vector (optional): Feature descriptor (e.g., 128-dimensional vector) extracted from the target region.

[0031] This encapsulation method greatly reduces the amount of data transmitted, conveying only key information rather than the entire image.

[0032] The front-end image processing unit uses a SoC chip with an integrated lightweight NPU, and the lightweight object detection model is a deep learning model that has been quantized and pruned; the central fusion processing unit uses a SoC chip with an integrated high-performance NPU and CPU.

[0033] The system bus is either a USB bus or a Gigabit Ethernet bus.

[0034] An image processing method based on the above system includes the following steps: Front-end processing steps: executed by each front-end image processing unit, including: S1. Acquire the raw video stream from the image sensor; S2. Preprocess and downsample the original video stream; S3. Input the processed image into a lightweight object detection model to obtain the recognition results of the objects in the image. The recognition results include bounding box coordinates, class confidence, and class label. S4. Optionally, extract appearance feature vectors from the identified target regions; S5. Encapsulate the identification result and / or feature vector together with the timestamp and device ID into a data packet; S6. Send the encapsulated data packet to the central fusion processing unit via the system bus; Central processing steps: executed by the central fusion processing unit, including: S7. Receive data packets from all front-end image processing units; S8. Based on the timestamps in each data packet, synchronize all recognition results in time to form a global perception frame; S9. Perform spatial alignment and data fusion on the multi-source recognition results after time synchronization; S10. Based on the fused data, run a multi-target tracking algorithm and output the continuous trajectory, global coordinates, and unique identity ID of each target.

[0035] The spatial alignment in step S9 is based on the fixed installation position and angle parameters of each image sensor and the UAV platform. The target position identified by each front-end unit is uniformly transformed into the global coordinate system of the UAV through coordinate transformation.

[0036] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] like Figure 1 As shown, this system includes multiple front-end image processing units (RV1126B chip), a central fusion processing unit (RK3588 chip), and a USB 3.0 system bus.

[0038] Front-end processing (such as) Figure 2 As shown): Each front-end image processing unit connects to a 4K image sensor. The front-end image processing unit runs a YOLOv8n model quantized with INT8. Its workflow is as follows: acquire 4K images → downsample to 1080p using the RGA module → send the 1080p images to the NPU to run the YOLOv8n model for object detection → obtain the bounding box coordinates, category, and confidence of the target → use ROIAlign to crop the target region from the original image and extract a 128-dimensional feature vector → encapsulate [timestamp, device ID, target information list (coordinates, category, feature vector)] into a JSON format data packet → send it to the central unit via the USB 3.0 bus.

[0039] Central processing (such as) Figure 2 and Figure 3 As shown): The central fusion processing unit receives all structured data packets via the USB host interface. First, based on the timestamps in the data packets (based on the PTP protocol), all data is aligned to the same time point. Then, based on the pre-calibrated extrinsic parameters (position and angle) of each camera, the target coordinates detected by each front-end unit are uniformly transformed to a global coordinate system with the UAV as the origin. Finally, the fused global target list is received, the motion trajectory is predicted using Kalman filtering, and data association is performed using appearance feature vectors, ultimately outputting the global coordinates, velocity, and unique ID of each target throughout the process. Specifically, data packets from all front-ends are received. First, time synchronization is performed, aligning data arriving at different times to the same time point. Next, spatial alignment is performed, using camera calibration parameters to uniformly transform the target positions in each camera coordinate system to the UAV's global coordinate system. Then, data fusion is performed, determining whether the detection results from different cameras belong to the same physical target and merging them. Finally, the multi-target tracking core process is entered: predicting the new target position based on the existing trajectory; associating and matching the new detection result with the predicted position; updating the existing trajectory based on the matching result, or creating a new trajectory or deleting lost trajectories. The final output is a list containing all global state information for all targets.

[0040] See Figure 4 The electronic device provided in the embodiments of this disclosure includes a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140. Memory 1130 is used to store computer programs; The processor 1110, when executing the program stored in the memory 1130, implements the image processing method described above.

[0041] The aforementioned communication bus 1140 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0042] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.

[0043] The memory 1130 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1130 may also be at least one storage device located remotely from the aforementioned processor 1110.

[0044] Embodiments of this disclosure also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the image processing method described above.

[0045] The embodiments described above are merely examples of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these modifications and improvements all fall within the protection scope of this disclosure.

Claims

1. A distributed visual guidance system for unmanned aerial vehicles (UAVs), characterized in that, It includes a central fusion processing unit and multiple front-end image processing units connected to it via a system bus; The front-end image processing unit is connected to an image sensor for acquiring raw image data and performing target recognition on the raw image data. The recognition results are then encapsulated in a structured manner to form a structured data packet, which is then sent to the central fusion processing unit via the system bus. The central fusion processing unit is used to receive and parse each structured data packet, perform time synchronization and data fusion on the parsed data, and run a multi-target tracking algorithm to output the target's tracking trajectory.

2. The distributed visual guidance system for unmanned aerial vehicles according to claim 1, characterized in that, The front-end image processing unit includes a SoC chip that integrates a lightweight target detection model. The lightweight target detection model is used to perform target recognition on the original image data to obtain recognition results.

3. The distributed visual guidance system for unmanned aerial vehicles according to claim 1, characterized in that, The identification result includes one or more of the target's location information, category information, and feature vector.

4. A distributed visual guidance system for unmanned aerial vehicles according to claim 2, characterized in that, The lightweight object detection model is a deep learning model that has undergone quantization or pruning.

5. A distributed visual guidance system for unmanned aerial vehicles according to claim 1, characterized in that, The central fusion processing unit is also used to convert the target position information identified by each front-end image processing unit into a unified global coordinate system based on the installation parameters of each image sensor, thereby completing spatial alignment.

6. An image processing method applied to a distributed vision guidance system for an unmanned aerial vehicle (UAV) according to any one of claims 1-5, characterized in that, Includes the following steps: The front-end image processing unit collects raw image data, performs target detection to obtain recognition results, encapsulates the recognition results in a structured manner to form a structured data packet, and sends it to the central fusion processing unit. The central fusion processing unit receives all the structured data packets, performs time synchronization and data fusion, and runs a multi-target tracking algorithm to output the tracking results.

7. The image processing method according to claim 6, characterized in that, Structured encapsulation refers to packaging the identification results together with the timestamp and device identifier into a structured data packet.

8. The image processing method according to claim 6, characterized in that, The data fusion includes: synchronizing all recognition results based on timestamps, and unifying the coordinates of the target location based on preset spatial transformation parameters.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor for executing a program stored in a memory to implement the image processing method according to any one of claims 6-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method according to any one of claims 6-8.

Images