Airborne intelligent identification system and method based on edge calculation

Abstract

The invention discloses an airborne intelligent identification system and method based on edge calculation. The airborne intelligent identification system comprises edge calculation and intelligent identification application for airborne constrained resources. The edge computing system is based on a domestic high-performance intelligent reconfigurable chip, integrates a heterogeneous multi-core processor, programmable logic, an intelligent computing accelerator, a unique general architecture and reconfigurable characteristics, and supports multi-board expansion to realize computing power and storage upgrading. According to the intelligent identification application method, a processor runs a data flow control program, a programmable logic runs an image acquisition processing program, an intelligent computing accelerator deploys an edge intelligent algorithm processing architecture of a target identification model, and efficient data interaction and telemetry data communication are achieved through an SRIO bus, a high-speed serial interface and a radio frequency transceiver. According to the method, the unmanned aerial vehicle is converted from a mobile sensor to an intelligent node capable of performing local processing, and the accuracy of a target identification system is improved by using an adaptive image processing reconstruction method while the power consumption and the memory resource consumption are not improved.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent computing for unmanned aerial vehicles (UAVs), and in particular to an airborne intelligent identification system and method based on edge computing. Background Technology

[0002] With the rapid development of drone technology, applications requiring complex computations, such as artificial intelligence (AI) and image processing, are being deployed at the edge, leading to an explosive increase in the number of devices connected to ground stations and the amount of data generated at the edge. Affected by communication bottlenecks, security, and privacy issues, intelligent drone applications have shifted from cloud computing to edge computing in an effort to alleviate the communication burden of high-resolution video streams and provide flight assistance, automatic landing, and identification and tracking.

[0003] Environmental visual perception is crucial for drone flight and is central to many related applications. Focusing on drone target detection and recognition, the increasing resolution and frame rate of airborne image sensors place higher demands on the speed and accuracy of image processing and target recognition. Traditional solutions rely on airborne sensors to collect data, transmit it, and then process it, significantly consuming network and energy resources, reducing real-time performance, and limiting the effectiveness of intelligent algorithms. Furthermore, traditional airborne systems based on single computing resources cannot be equipped with sufficient hardware computing power to process data, and compared to cloud computing, the limitations of fixed hardware functionality greatly reduce operational flexibility, making it difficult to meet the demands.

[0004] Therefore, constructing a high-performance airborne target recognition system under the resource constraints of UAV platforms is a problem of significant research value. Summary of the Invention

[0005] This invention provides an airborne intelligent identification system and method based on edge computing, which enables the deployment of intelligent computing on edge devices.

[0006] In a first aspect, an airborne intelligent identification system based on edge computing is provided. The system is an edge computing platform based on an intelligent reconfigurable chip, including an intelligent reconfigurable chip and an interface module.

[0007] The intelligent reconfigurable chip comprises a multi-core processor, programmable logic, and an intelligent computing accelerator. The multi-core main processor is responsible for the parallel execution of functional tasks and the deployment of intelligent algorithms. The programmable logic is used to implement image acquisition and output, storage of raw images, intermediate results, and processed image data, as well as intelligent preprocessing of image data. The intelligent computing accelerator, based on hardware optimization and flexible resource configuration, accelerates intelligent algorithms for convolutional neural networks. The programmable logic and the intelligent computing accelerator work together under the control of the multi-core main processor to complete the entire intelligent target recognition process.

[0008] The interface module includes an image acquisition interface module, an intelligent image processing and transmission interface module, a flight control data interaction module, and a telemetry and remote sensing data transmission module; the image acquisition interface and image output interface modules are connected to the programmable logic; the telemetry and remote sensing data transmission module and the flight control data interaction module are connected to the multi-core processor; the intelligent image processing and transmission interface module transmits the image data processed by the programmable logic to the intelligent computing accelerator and receives the results of intelligent computing.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the multi-core processor is used for general computing, responsible for control tasks such as data transmission and task scheduling, as well as temporary storage of intermediate computed values ​​of network data and weight data in artificial intelligence computing;

[0010] The programmable logic deploys image processing algorithms, which use the programmable properties of the programmable logic to obtain input image data from the interface module. After preprocessing, the image is adjusted to a data format that conforms to the operation of the intelligent computing accelerator. The processed image data is then postprocessed to obtain the final expected structure, which is then output through the data transmission interface.

[0011] Intelligent computing accelerators are used to support a variety of neural network operators and algorithms, perform convolution and pooling operations, and accelerate intelligent algorithm computation through parallel processing and hardware optimization.

[0012] In conjunction with the first aspect, in certain implementations of the first aspect, the system satisfies at least one of the following:

[0013] The multi-core main processor is an octa-core processor based on SPARC V8;

[0014] The programmable logic is an FPGA based on BQVU3P;

[0015] The neural network operators supported by the intelligent computing accelerator include: convolutions of size 1x1 to 7x7, depthwise convolutions of size 1x1 to 7x7, ReLU, Leaky ReLU, Pooling, and Concat;

[0016] The intelligent computing accelerator supports neural network algorithms including image classification and object recognition network models based on VGG, ResNet, and YOLO V3 / V4 / V5;

[0017] The intelligent computing accelerator supports PyTorch, Keras, Tensorflow, and ONNX frameworks;

[0018] The image acquisition interface and image output interface modules are connected to the programmable logic via MIPI and HDMI type interfaces;

[0019] The telemetry and remote sensing data transmission module and the flight control data interaction module are connected to the multi-core processor through LVDS type interface and VPX bus interface;

[0020] The intelligent image processing transmission interface module connects to programmable logic and intelligent computing accelerator via an SRIO type interface.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the intelligent recognition method executed by the system includes an image adaptive reconstruction processing method. The image adaptive reconstruction processing method utilizes the reconfigurable characteristics of the intelligent reconfigurable chip to adaptively reconstruct the real-time image processing algorithm in response to different scenarios encountered by the UAV during flight, and switches the image processing algorithm to adaptively reconstruct the programmable logic.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the image adaptive reconstruction processing method includes:

[0023] Programmable logic acquires video data, extracts image frames, and stores the data.

[0024] The programmable logic scales the raw image data horizontally and vertically before transmitting it to a multi-core processor.

[0025] The multi-core processor performs noise calculations on the scaled image data and adaptively outputs reconstruction instructions based on the calculation results. It controls the programmable logic in real time to implement image processing algorithms for reconstruction, including median filtering and haze removal.

[0026] The programmable logic responds to reconfiguration instructions and uses the reconfigured graphics processing algorithm to process the image data so that the noise of the image data meets the requirements.

[0027] The multi-core processor transmits image data with noise levels meeting the requirements to an intelligent computing accelerator equipped with a target recognition model via an AXI bus;

[0028] The intelligent computing accelerator completes image inference calculations and transmits the feature information of target recognition to the multi-core processor;

[0029] Multi-core processors output bounding box coordinates and classification information;

[0030] The programmable logic parses the detection box coordinates and classification information, and draws the detection box and label on the stored image frame.

[0031] In conjunction with the first aspect, in some implementations of the first aspect, the intelligent recognition method executed by the system includes a reasoning calculation method, which includes preprocessing, postprocessing, and model reasoning;

[0032] Preprocessing is implemented using programmable logic, including image size consistency adjustment, pixel value normalization, RGB channel conversion, and data type conversion, transforming the original image into a standard input acceptable to the model.

[0033] Post-processing is implemented using programmable logic, including bounding box adjustment, class probability filtering, and multi-scale fusion.

[0034] Model inference relies on intelligent computing accelerators to deploy intelligent network models on edge platforms through model transformation, model quantization, model compilation, and model deployment.

[0035] In conjunction with the first aspect, in some implementations of the first aspect, preprocessing includes:

[0036] Image adjustment: Adjust the size of the input image to match the pixel input size of the model;

[0037] Normalization operation: Standardizes pixel values ​​to a range between 0 and 1;

[0038] Padding: Add pixels around the image to ensure that the image dimensions match the model's expected input dimensions;

[0039] Channel conversion: Adjusts the channel order of the image to the RGB order that meets the requirements of the intelligent recognition algorithm;

[0040] Data type conversion: Convert the image representation into a fixed-point data structure suitable for accelerator acceleration.

[0041] In conjunction with the first aspect, in some implementations of the first aspect, post-processing includes:

[0042] Bounding box adjustment: Non-maximum suppression is used to fine-tune the detection results, retaining the rectangles describing the target location with the highest confidence.

[0043] Category probability filtering: By setting a probability threshold, detection results with low probabilities corresponding to the target category are filtered out to ensure that the final detection result is based on a target classification with high confidence.

[0044] Multi-scale fusion: The coordinates of detection boxes from different scales are scaled and adjusted, and then fused to enable accurate detection of targets in different scenes and target sizes.

[0045] In conjunction with the first aspect, in some implementations of the first aspect, model reasoning includes:

[0046] The model was developed using the PyTorch deep learning framework and converted into an ONNX floating-point model for intelligent computing accelerators using a model converter.

[0047] The ONNX floating-point model is converted into a fixed-point quantized neural network model using a dynamic fixed-point algorithm using a model quantization tool. The quantized model and the original floating-point model are then compared for inference and loss evaluation, and the error analysis results are output.

[0048] The compiler converts the fixed-point quantized neural network model into a binary file that can be recognized by intelligent computing accelerators;

[0049] The firmware parses the binary file generated by the compiler and deploys the computing tasks and corresponding data on the corresponding hardware nodes according to the computing flow information.

[0050] Secondly, an airborne intelligent identification method based on edge computing is provided, the method being applied to a system as described in any of the implementations of the first aspect above, the method comprising:

[0051] The image adaptive reconstruction processing method utilizes the reconfigurable characteristics of intelligent reconfigurable chips to adaptively reconstruct real-time image processing algorithms in response to different scenarios encountered by UAVs during flight, and switches image processing algorithms to adaptively reconstruct programmable logic.

[0052] The inference computation method includes preprocessing, postprocessing, and model inference. Preprocessing is implemented using programmable logic and includes image size consistency adjustment, pixel value normalization, RGB channel conversion, and data type conversion to transform the original image into a standard input acceptable to the model. Postprocessing is also implemented using programmable logic and includes bounding box adjustment, class probability filtering, and multi-scale fusion. Model inference is implemented using an intelligent computing accelerator, which deploys the intelligent network model on the edge platform through model conversion, model quantization, model compilation, and model deployment.

[0053] Compared with the prior art, the solution provided by the present invention has at least the following beneficial technical effects:

[0054] 1. This invention realizes an intelligent target recognition system for airborne edge intelligent recognition needs. Unlike the traditional design method of combining specific single machines by designing functional modules, the edge computing platform based on intelligent reconfigurable chips supports the completion of required functions by adapting different external interfaces through software and hardware reconfiguration. Its independent high-efficiency computing engine and rich peripheral interfaces support distributed processing. Through high-speed switching and software-defined topology, it can realize arbitrary configuration of multiple platforms and arbitrary access to data, which can effectively solve the current problems of real-time performance, network constraints, resource consumption and privacy protection in cloud computing.

[0055] 2. This invention proposes an image adaptive reconstruction processing method. Utilizing the intelligent reconstructability of this platform, it enables the UAV to perform real-time image processing algorithm reconstruction in different scenarios encountered during flight without increasing computation and power consumption, thereby greatly improving the quality of acquired images and the accuracy of target recognition.

[0056] 3. This invention proposes an edge computing platform architecture based on domestically produced intelligent reconfigurable chips and a heterogeneous intelligent algorithm processing architecture based on multi-core processors, programmable logic, and intelligent computing accelerators. This architecture deploys computational problems within the chip's computing power range to the side closer to the data source, significantly improving the intelligence level of the airborne computing platform and the adaptability of intelligent applications.

[0057] 4. This invention adopts the YOLOv5s lightweight target detection and recognition network, which is most suitable for deployment on edge devices. Based on domestically produced intelligent reconfigurable chips, the target detection and recognition algorithm is successfully deployed, realizing the rapid integration of intelligent recognition applications at the edge, while the accuracy can meet the needs of UAV application scenarios. Attached Figure Description

[0058] Figure 1 This is a diagram illustrating the functions and relationships of an airborne intelligent identification system based on edge computing according to the present invention within an airborne system.

[0059] Figure 2 This is a hardware architecture and interface module diagram of the edge computing platform based on the intelligent reconfigurable chip of the present invention.

[0060] Figure 3 This is a functional architecture diagram of an airborne intelligent identification system based on edge computing according to the present invention.

[0061] Figure 4 This is a data flow diagram of the airborne intelligent identification system of the present invention.

[0062] Figure 5 This is a flowchart of the model reasoning method for target recognition in this invention.

[0063] Figure 6 This is a flowchart of the compilation process of the intelligent network model in an embodiment of the present invention.

[0064] Figure 7 This is a diagram showing the test and verification results of an airborne intelligent recognition system based on edge computing, according to an embodiment of the present invention. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0066] Currently, China's chip design and manufacturing capabilities are continuously improving, endowing intelligent reconfigurable chips with both accelerated computing and flexible processing capabilities, demonstrating their application potential in UAV edge computing. This invention focuses on the system design and methodology research of airborne image processing and target recognition based on intelligent reconfigurable chips, giving full play to the high performance of multi-core processors in general computing, the high computing power of intelligent computing accelerators in artificial intelligence algorithms, and the flexibility of programmable logic devices (FPGAs) in image processing, to meet the intelligent requirements of airborne edge computing systems.

[0067] This invention addresses edge computing-based intelligent applications for unmanned aerial vehicles (UAVs) and proposes an airborne intelligent recognition system and method. It deploys computational tasks within the chip's computing power range to the side closer to the data source. Based on the concept of edge computing, it deploys intelligent recognition algorithms on UAVs using domestically produced intelligent reconfigurable chips, enabling rapid integration of image processing and target recognition applications at the edge.

[0068] like Figure 1 As shown, the present invention provides an airborne intelligent recognition system based on edge computing, including an edge computing platform based on an intelligent reconfigurable chip. The edge computing platform is used to execute the intelligent recognition method provided by the present invention, which includes image processing and target recognition.

[0069] The edge computing platform based on intelligent reconfigurable chips addresses the hardware deployment challenges of intelligent algorithms for image processing and target recognition under conditions of limited hardware resources, insufficient intelligent computing power, limited communication bandwidth, and restricted high-speed data transmission capabilities at the airborne edge. It transforms drones from mobile sensors into intelligent nodes capable of local processing. The edge computing platform based on intelligent reconfigurable chips includes a computing processing module and an interface module.

[0070] like Figure 2 As shown, the interface module includes a power module, a debugging and downloading module, an image acquisition interface module, an intelligent processing image transmission interface module, a flight control data interaction module, and a telemetry and remote sensing data transmission module. The image acquisition and image output interface modules connect to the intelligent reconfigurable chip via MIPI and HDMI interfaces. The telemetry and remote sensing data transmission module and the flight control data interaction module connect to the intelligent reconfigurable chip via LVDS and VPX bus interfaces (UART, GPIO, CAN). The intelligent processing image transmission interface module transmits the programmable logic-processed image data to the intelligent computing accelerator and receives the results of intelligent calculations via an SRIO interface. The power module and debugging and downloading module connect to the embedded intelligent algorithm processing architecture via A / D, EthernetMAC, and JTAG interfaces. The intelligent recognition system also supports USB, eMMC, and PCIe peripheral interfaces.

[0071] like Figure 3As shown, the computing module integrates a multi-core processor, programmable logic, and an intelligent computing accelerator based on an intelligent reconfigurable chip. The multi-core main processor is an eight-core SPARC V8 processor, responsible for the parallel execution of functional tasks and the deployment of intelligent algorithms. The programmable logic, based on the BQVU3P, implements image acquisition and output, storage of raw images, intermediate results, processed image data, and intelligent preprocessing of image data. The intelligent computing accelerator accelerates intelligent algorithms for convolutional neural networks based on hardware optimization and flexible resource configuration. Under the control of the main processor, the programmable logic and the intelligent computing accelerator work together to complete the entire intelligent target recognition process.

[0072] Multi-core processors are primarily used for general-purpose computing and are not focused on deep learning tasks. In the overall system architecture, they are responsible for data transmission and task scheduling control, as well as the temporary storage of intermediate computational values ​​of network data and weight data in artificial intelligence computing.

[0073] As a flexible hardware accelerator, the programmable logic unit (PLU) deploys image processing algorithms in the system. It uses the programmable properties of the PLU to obtain input image data from the interface module, performs preprocessing operations to adjust the image to a data format that conforms to the operation of the intelligent computing accelerator, and performs postprocessing operations on the processed image data to obtain the final expected structure, which is then output through the data transmission interface.

[0074] The intelligent computing accelerator is primarily used to support specific types of computational tasks. Supported neural network operators include convolutions of sizes 1x1 to 7x7, depthwise convolutions of sizes 1x1 to 7x7, ReLU, Leaky ReLU, Pooling, and Concat. Supported neural network algorithms include mainstream image classification and object recognition network models such as VGG, ResNet, and YOLO V3 / V4 / V5, supporting PyTorch, Keras, Tensorflow, and ONNX frameworks. In this invention, the intelligent object recognition algorithm is a deep learning algorithm, containing a large number of convolution and pooling operations. The intelligent computing accelerator accelerates the computation of the intelligent algorithm through parallel processing and hardware optimization.

[0075] Unlike traditional design methods that combine functional modules to create specific standalone machines, edge computing platforms based on intelligent reconfigurable chips support adaptation to different external interfaces through hardware and software reconfiguration to complete the required functions. Their proprietary high-efficiency computing engine and rich peripheral interfaces support distributed processing. Through high-speed switching and software-defined topology, multiple platforms can be arbitrarily configured and data can be accessed freely. Each platform is a general-purpose functional unit, supporting expansion to provide computing power or storage support. Online reconfiguration enables modular and software-defined on-machine processing, supporting the rapid construction and expansion of more complex on-machine processing systems.

[0076] The intelligent recognition methods implemented by this edge computing platform include image adaptive reconstruction processing and inference computing methods.

[0077] The image adaptive reconstruction processing method utilizes the intelligent reconstructability of this platform, enabling the UAV to adaptively reconstruct real-time image processing algorithms in different scenarios encountered during flight without increasing computation and power consumption. It switches between algorithms such as median filtering and fog removal, thereby achieving more accurate capture of the target to be identified in complex environments and improving the target recognition accuracy of subsequent inference calculations.

[0078] The inference computation method includes preprocessing, postprocessing, and model inference. Preprocessing is a crucial step in ensuring the model can efficiently and accurately identify and locate target objects. This includes image size consistency adjustment, pixel value normalization, RGB channel conversion, and data type conversion, transforming the raw image into a standard input acceptable to the model. Postprocessing optimizes and corrects the model's output through fine-tuning to obtain accurate and reliable object detection results. This includes bounding box adjustment, class probability filtering, and multi-scale fusion, improving the performance and adaptability of intelligent recognition. The model inference method uses the lightweight YOLOv5s network model. Through model conversion, model quantization, model compilation, and model deployment, the intelligent network model is deployed on an edge platform.

[0079] The following section further elaborates on the overall workflow of the intelligent recognition system based on the data flow diagram, such as... Figure 4 As shown, this includes an image adaptive reconstruction processing method and an inference computation method. The image adaptive reconstruction processing method enables the UAV to adaptively reconstruct and compute the image data acquired by the UAV at the edge during flight, when facing different environments (such as noise interference and fog). This optimizes the data and achieves more accurate capture of the target in complex environments, improving the accuracy of subsequent target recognition inference computation. The specific steps are as follows.

[0080] Step 1: The programmable logic acquires 1920×1080 video data through the HDMI interface, and performs image frame extraction and data storage.

[0081] Step 2: The programmable logic scales the original image data horizontally and vertically, converting it into a 640×640 image, which is then transmitted to the multi-core processor via a two-channel SRIO high-speed bus.

[0082] Step 3: The multi-core processor performs noise calculation on the 640×640 image data and adaptively outputs reconstruction instructions based on the calculation results. The programmable logic is controlled in real time through the third SRIO bus to realize image processing algorithms such as median filtering and haze removal for reconstruction.

[0083] Step 4: The programmable logic responds to the reconstruction instruction and uses the reconstructed graphics processing algorithm to process the image data so that the noise in the image data meets the requirements.

[0084] Step 5: The multi-core processor transmits the compliant 640×640 image data to the intelligent computing accelerator, which has deployed the target recognition model, via the AXI bus.

[0085] Step Six: The intelligent computing accelerator completes the image inference calculation and transmits the feature information to the multi-core processor via the AXI bus.

[0086] Step 7: The multi-core processor outputs the detection box coordinates and classification information through the 4th SRIO output.

[0087] Step 8: The programmable logic parses the coordinate information and draws detection boxes and labels on the stored image frames, which are then transmitted back to the ground station via image transmission.

[0088] like Figure 5 A specific implementation of a reasoning computation method is shown, including preprocessing of the reasoning image, postprocessing, and model reasoning.

[0089] Preprocessing methods, implemented using programmable logic, are crucial steps in ensuring the model can efficiently and accurately identify and locate target objects. These include image size consistency adjustment, pixel value normalization, RGB channel conversion, and data type conversion, transforming the original image into standard input acceptable to the model. The specific steps are as follows.

[0090] Step 1: Image Adjustment: Adjust the size of the input image to match the model's 640×640 pixel input size, maintaining consistency in the input image so that the model can process images of different sizes;

[0091] Step 2, Normalization: By dividing the pixel value by 255, the pixel value is normalized to the range between 0 and 1. This helps to improve the convergence speed of the model and makes it easier for it to learn effective features.

[0092] Step 3, padding: Add pixels around the image to ensure that the image dimensions match the model's expected input dimensions;

[0093] Step 4: Channel Conversion: Adjust the channel order of the image to the RGB order required by the intelligent recognition algorithm so that the model can correctly interpret and process color information;

[0094] Step 5: Data type conversion: Convert the image representation into a fixed-point data structure suitable for accelerator acceleration, so that the image data can fully take advantage of hardware acceleration.

[0095] The post-processing method relies on programmable logic to optimize and correct the model's output, ultimately obtaining accurate and reliable target detection results. This includes bounding box adjustment, class probability filtering, and multi-scale fusion, improving the performance and adaptability of intelligent recognition. The specific steps are as follows.

[0096] Step 1: Bounding box adjustment: The non-maximum suppression method is used to fine-tune the detection results, retaining the rectangle describing the target location with the highest confidence, so that the bounding box more accurately surrounds the target object;

[0097] Step 2, Category Probability Filtering: By setting a probability threshold, detection results with low probabilities corresponding to the target category are filtered out to ensure that the final detection result is based on a high-confidence target classification, thereby improving the overall detection reliability;

[0098] Step 3, Multi-scale fusion: The coordinates of detection boxes from different scales are scaled and adjusted, and then fused to ensure accurate detection of targets in different scenes and target sizes, thus improving the robustness of the algorithm.

[0099] Model inference is achieved using an intelligent computing accelerator and the lightest YOLOv5s network model. The intelligent network model is deployed on an edge platform through steps such as model transformation, model quantization, model compilation, and model deployment. Specific steps are as follows: Figure 6 As shown.

[0100] Step 1: Develop a good model using the mainstream deep learning framework PyTorch, and then convert it into an ONNX floating-point model optimized for intelligent computing accelerators using a model converter;

[0101] Step 2: Use a model quantization tool to convert the floating-point ONNX model into a fixed-point quantized model using a dynamic fixed-point algorithm to maximize model accuracy. Then, perform inference and loss assessment on the quantized model and the original floating-point model, and output the error analysis results.

[0102] Step 3: The compiler converts the fixed-point quantized neural network model into binary files containing instructions, configurations, weights, and computational flows that can be recognized by the intelligent computing accelerator;

[0103] Step 4: The firmware parses the binary file generated by the compiler and efficiently deploys the computing tasks and corresponding data on the corresponding hardware nodes according to the computing flow information.

[0104] This invention proposes an airborne intelligent recognition system and method based on edge computing for intelligent applications on unmanned aerial vehicles (UAVs). Its core lies in addressing the problems inherent in current centralized computing methods, such as real-time performance, network constraints, resource consumption, and privacy protection, represented by cloud computing. It deploys computational tasks within the chip's computing power range closer to the data source. Based on the concept of edge computing, it deploys intelligent recognition algorithms on UAVs using domestically produced intelligent reconfigurable chips, achieving rapid integration of image processing and target recognition applications at the edge.

[0105] Example

[0106] The implementation process is described below using an intelligent target recognition system.

[0107] Based on the algorithm computation and storage requirements of the UAV intelligent computing platform, an edge computing platform based on an intelligent reconfigurable chip was designed and developed. This includes the intelligent reconfigurable chip's computing and storage resources, data transmission interface modules, and peripheral circuitry for the storage device. Image acquisition and output interface modules connect to the intelligent reconfigurable chip via MIPI and HDMI interfaces; telemetry and remote sensing data transmission modules and flight control data interaction modules connect to the intelligent reconfigurable chip via LVDS and VPX bus interfaces (UART, GPIO, CAN); the intelligent image processing transmission interface module transmits programmable logic-processed image data to the intelligent computing accelerator and receives the results of intelligent computation via an SRIO interface; the power supply module and debug download module connect to the embedded intelligent algorithm processing architecture via A / D, Ethernet MAC, and JTAG interfaces; the intelligent recognition system also supports USB, eMMC, and PCIe peripheral interfaces.

[0108] Testing and verification of intelligent recognition algorithms deployed on an edge computing platform:

[0109] Step 1: Download the algorithm program to the internal storage unit of the edge computing platform based on the intelligent reconfigurable chip via the JTAG interface and start the execution instructions;

[0110] Step 2: The multi-core processor fetches instructions from the specified address entry point and performs the calculations. The program is an application program written in the high-level language C.

[0111] Step 3: The programmable logic transmits the image to the internal system via the HDMI input interface and performs preprocessing operations on the input image data;

[0112] Step 4: The programmable logic outputs the pre-processed data to the intelligent computing accelerator, which then performs behavioral calculations based on the selected model and the input weight data.

[0113] Step 5: The intelligent computing accelerator transmits the processed data to the programmable logic for post-processing.

[0114] Step Six: Output and display the final intelligent recognition processing results via the HDMI output interface.

[0115] Reference Figure 7 As shown, taking one image as an example, the image verified by the program has obvious bounding boxes and classification marks. When multiple images are selected, the image recognition accuracy reaches 80%.

[0116] Throughout the process, the intelligent recognition system used external interfaces, internal interfaces, storage units, processing units, and acceleration units. The successful deployment and output of the intelligent recognition algorithm verified the success and stability of the intelligent recognition system.

[0117] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims of the present invention.

Claims

1. An airborne intelligent identification system based on edge computing, characterized in that, The system is an edge computing platform based on an intelligent reconfigurable chip, including an intelligent reconfigurable chip and an interface module; The intelligent reconfigurable chip includes a multi-core processor, programmable logic, and an intelligent computing accelerator. The multi-core main processor is responsible for the parallel execution of functional tasks and the deployment of intelligent algorithms. The programmable logic is used to realize image acquisition and output, storage of raw images, intermediate results and processed image data, as well as intelligent preprocessing of image data. The intelligent computing accelerator, based on hardware optimization and flexible resource configuration, realizes intelligent algorithm acceleration for convolutional neural networks. Programmable logic and intelligent computing accelerators work together under the control of a multi-core main processor to complete the entire process of intelligent target recognition; The interface module includes an image acquisition interface module, an intelligent image processing and transmission interface module, a flight control data interaction module, and a telemetry and remote sensing data transmission module; the image acquisition interface and image output interface modules are connected to programmable logic; the telemetry and remote sensing data transmission module and the flight control data interaction module are connected to a multi-core processor. The intelligent image processing transmission interface module transmits the image data processed by programmable logic to the intelligent computing accelerator and receives the results of intelligent computing.

2. The system according to claim 1, characterized in that, Multi-core processors are used for general computing, responsible for data transmission and task scheduling control tasks, as well as the temporary storage of intermediate calculated values ​​of network data and weight data in artificial intelligence computing; The programmable logic deploys image processing algorithms, which use the programmable properties of the programmable logic to obtain input image data from the interface module. After preprocessing, the image is adjusted to a data format that conforms to the operation of the intelligent computing accelerator. The processed image data is then postprocessed to obtain the final expected structure, which is then output through the data transmission interface. Intelligent computing accelerators are used to support a variety of neural network operators and algorithms, perform convolution and pooling operations, and accelerate intelligent algorithm computation through parallel processing and hardware optimization.

3. The system according to claim 1, characterized in that, The system satisfies at least one of the following: The multi-core main processor is an octa-core processor based on SPARC V8; The programmable logic is an FPGA based on BQVU3P; The neural network operators supported by the intelligent computing accelerator include: convolutions of size 1x1 to 7x7, depthwise convolutions of size 1x1 to 7x7, ReLU, Leaky ReLU, Pooling, and Concat; The intelligent computing accelerator supports neural network algorithms including image classification and object recognition network models based on VGG, ResNet, and YOLO V3 / V4 / V5; The intelligent computing accelerator supports PyTorch, Keras, Tensorflow, and ONNX frameworks; The image acquisition interface and image output interface modules are connected to the programmable logic via MIPI and HDMI type interfaces; The telemetry and remote sensing data transmission module and the flight control data interaction module are connected to the multi-core processor through LVDS type interface and VPX bus interface; The intelligent image processing transmission interface module connects to programmable logic and intelligent computing accelerator via an SRIO type interface.

4. The system according to claim 1, characterized in that, The intelligent recognition method executed by the system includes an image adaptive reconstruction processing method. The image adaptive reconstruction processing method utilizes the reconfigurable characteristics of the intelligent reconfigurable chip to adaptively reconstruct the real-time image processing algorithm in response to different scenarios encountered by the UAV during flight, and switches the image processing algorithm to adaptively reconstruct the programmable logic.

5. The system according to claim 4, characterized in that, Image adaptive reconstruction processing methods include: Programmable logic acquires video data, extracts image frames, and stores the data. The programmable logic scales the raw image data horizontally and vertically before transmitting it to a multi-core processor. The multi-core processor performs noise calculations on the scaled image data and adaptively outputs reconstruction instructions based on the calculation results. It controls the programmable logic in real time to implement image processing algorithms for reconstruction, including median filtering and haze removal. The programmable logic responds to reconfiguration instructions and uses the reconfigured graphics processing algorithm to process the image data so that the noise of the image data meets the requirements. The multi-core processor transmits image data with noise levels meeting the requirements to an intelligent computing accelerator equipped with a target recognition model via an AXI bus; The intelligent computing accelerator completes image inference calculations and transmits the feature information of target recognition to the multi-core processor; Multi-core processors output bounding box coordinates and classification information; The programmable logic parses the detection box coordinates and classification information, and draws the detection box and label on the stored image frame.

6. The system according to claim 1, characterized in that, The intelligent recognition method executed by the system includes a reasoning calculation method, which includes preprocessing, postprocessing, and model reasoning. Preprocessing is implemented using programmable logic, including image size consistency adjustment, pixel value normalization, RGB channel conversion, and data type conversion, transforming the original image into a standard input acceptable to the model. Post-processing is implemented using programmable logic, including bounding box adjustment, class probability filtering, and multi-scale fusion. Model inference relies on intelligent computing accelerators to deploy intelligent network models on edge platforms through model transformation, model quantization, model compilation, and model deployment.

7. The system according to claim 6, characterized in that, Preprocessing includes: Image adjustment: Adjust the size of the input image to match the pixel input size of the model; Normalization operation: Standardizes pixel values ​​to a range between 0 and 1; Padding: Add pixels around the image to ensure that the image dimensions match the model's expected input dimensions; Channel conversion: Adjusts the channel order of the image to the RGB order that meets the requirements of the intelligent recognition algorithm; Data type conversion: Convert the image representation into a fixed-point data structure suitable for accelerator acceleration.

8. The system according to claim 6, characterized in that, Post-processing includes: Bounding box adjustment: Non-maximum suppression is used to fine-tune the detection results, retaining the rectangles describing the target location with the highest confidence. Category probability filtering: By setting a probability threshold, detection results with low probabilities corresponding to the target category are filtered out to ensure that the final detection result is based on a target classification with high confidence. Multi-scale fusion: The coordinates of detection boxes from different scales are scaled and adjusted, and then fused to enable accurate detection of targets in different scenes and target sizes.

9. The system according to claim 6, characterized in that, Model inference includes: The model was developed using the PyTorch deep learning framework and converted into an ONNX floating-point model for intelligent computing accelerators using a model converter. The ONNX floating-point model is converted into a fixed-point quantized neural network model using a dynamic fixed-point algorithm using a model quantization tool. The quantized model and the original floating-point model are then compared for inference and loss evaluation, and the error analysis results are output. The compiler converts the fixed-point quantized neural network model into a binary file that can be recognized by intelligent computing accelerators; The firmware parses the binary file generated by the compiler and deploys the computing tasks and corresponding data on the corresponding hardware nodes according to the computing flow information.

10. An airborne intelligent identification method based on edge computing, characterized in that, The method is applied to the system as described in any one of claims 1 to 9, the method comprising: The image adaptive reconstruction processing method utilizes the reconfigurable characteristics of intelligent reconfigurable chips to adaptively reconstruct real-time image processing algorithms in response to different scenarios encountered by UAVs during flight, and switches image processing algorithms to adaptively reconstruct programmable logic. The inference computation method includes preprocessing, postprocessing, and model inference. Preprocessing is implemented using programmable logic and includes image size consistency adjustment, pixel value normalization, RGB channel conversion, and data type conversion to transform the original image into a standard input acceptable to the model. Postprocessing is also implemented using programmable logic and includes bounding box adjustment, class probability filtering, and multi-scale fusion. Model inference is implemented using an intelligent computing accelerator, which deploys the intelligent network model on the edge platform through model conversion, model quantization, model compilation, and model deployment.

Images