Unmanned aerial vehicle general image processing system based on multiple FPGAs (Field Programmable Gate Array) and Feiteng Mihiinvasive heterogeneous processor

Abstract

The utility model relates to an unmanned aerial vehicle general image processing system based on multiple FPGAs (Field Programmable Gate Array) and a Feiteng Matrix heterogeneous processor, the system is connected with hardware structures of an FPGA calculation array, a Feiteng Matrix processor, a CPLD (Complex Programmable Logic Device) chip and other peripheral chips through a first FPGA chip, and under the constraint of limited hardware resources and pin resources of the first FPGA chip, the Feiteng Matrix processor is connected with the FPGA calculation array, the Feiteng Matrix processor, the CPLD chip and other peripheral chips. Flexible expansion of calculation and storage is realized; the calculation capacity amplification is realized through a calculation subsystem formed by a Feiteng Maiku processor and two FPGA chips for calculation; the CPLD chip is connected with the storage subsystem composed of the SRAM and the DRAM, and separation of calculation and storage is achieved. According to the system, the overall structure that the first FPGA chip, the computing subsystem and the storage subsystem are separated is adopted, and the problem that limited hardware resources of a single FPGA are difficult to meet requirements is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a general image processing system for unmanned aerial vehicles based on multiple FPGAs and Phytium Maichuang heterogeneous processors. Background Art

[0002] UAV image processing provides target information for drone mission decision-making and plays a vital role in a variety of application scenarios, including environmental monitoring, emergency rescue, security monitoring, and power inspections. Deep learning technology has developed rapidly in recent years, and real-time image processing and analysis based on deep learning has become the most widely used UAV image processing method. However, due to the limited payload capacity of drones, they can only carry embedded computing devices of limited size and weight as computing units. In particular, to improve the drone's endurance, strict restrictions are placed on the low power consumption of the computing system. Therefore, for computationally intensive tasks, FPGAs are often used as algorithm acceleration units. However, the computational complexity and parameter count of commonly used deep learning target detection models are enormous, and the limited hardware resources of a single FPGA cannot meet the scale and computing resource requirements of deep learning target detection models. Utility Model Content

[0003] Based on this, it is necessary to provide a general image processing system for UAVs based on multiple FPGAs and Feiteng Maichuang heterogeneous processors to address the technical problem that the limited hardware resources of a single FPGA cannot meet the needs.

[0004] A general image processing system for unmanned aerial vehicles based on multiple FPGAs and Phytium Maichuang heterogeneous processors, the system comprising: a storage subsystem, a computing subsystem, a camera subsystem, a first FPGA chip, a bridge chip, a PCIE interface, a first Flash chip, a first JTAG interface, and a power module;

[0005] The storage subsystem, the computing subsystem, the camera subsystem, the first Flash chip, and the first JTAG interface are all connected to the first FPGA chip, and the first FPGA chip is connected to the PCIE interface via the bridge chip; the power module is connected to the PCIE interface;

[0006] The power supply module is used to provide power to the storage subsystem, the computing subsystem, the camera subsystem, the first Flash chip, the first FPGA chip and the bridge chip.

[0007] In one embodiment, the storage subsystem includes a CPLD chip, a DRAM module, an SRAM module, a second Flash module, and a second JTAG interface;

[0008] The DRAM module, the SRAM module and the second Flash module are connected to the CPLD chip, the second JTAG interface is connected to the CPLD chip; and the CPLD chip is connected to the first FPGA chip.

[0009] In one embodiment, the computing subsystem includes: a Feiteng Maichuang FT-Matrix heterogeneous processor, an MRAM chip, an FPGA computing array, a flash memory module, and a JTAG interface module; the Feiteng Maichuang FT-Matrix heterogeneous processor and the MRAM chip are connected; the flash memory module and the JTAG interface module are connected to the FPGA computing array; the FPGA computing array is connected to the first FPGA chip;

[0010] In one embodiment, the camera subsystem includes: a video decoding chip and a CCD camera lens;

[0011] The CCD camera lens is connected to the video decoding chip, and the video decoding chip is connected to the first FPGA chip.

[0012] In one embodiment, the FPGA computing array includes a second FPGA chip and a third FPGA chip for computing; the flash memory module includes a third Flash chip and a fourth Flash chip; the JTAG interface module includes a third JTAG interface and a fourth JTAG interface;

[0013] The third Flash chip and the third JTAG interface are both connected to the second FPGA chip, and the second FPGA chip is connected to the first FPGA chip;

[0014] The fourth Flash chip and the fourth JTAG interface are both connected to the third FPGA chip, and the third FPGA chip is connected to the first FPGA chip.

[0015] The aforementioned universal image processing system for drones based on multiple FPGAs and Feiteng Maichuang heterogeneous processors utilizes a hardware structure that connects an FPGA computing array, a Feiteng Maichuang FT-Matrix processor, a CPLD chip, and other peripheral chips via a first FPGA chip. This allows for flexible expansion of computing and storage within the limited hardware and pin resources of the first FPGA chip. The computing subsystem, comprised of the Feiteng Maichuang FT-Matrix processor and two FPGA chips used for computing, achieves increased computing capacity. The storage subsystem, comprised of SRAM and DRAM memory, is connected via a CPLD chip, reducing the hardware and pin resources used by the first FPGA chip to implement the memory access controller, thereby achieving separation of computing and storage. This system utilizes a separate overall structure for the first FPGA chip, computing subsystem, and storage subsystem, overcoming the problem of a single FPGA's limited hardware resources being unable to meet demand. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a block diagram of a general image processing system for drones based on multiple FPGAs and Phytium Maichuang heterogeneous processors in one embodiment;

[0017] Figure 2 Another embodiment is a block diagram of the principle of a general image processing system for drones based on multiple FPGAs and Phytium Maichuang heterogeneous processors. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0019] In one embodiment, Figure 1 As shown, a general image processing system for drones based on multiple FPGAs and Feiteng Maichuang heterogeneous processors is provided, which includes: a storage subsystem 10, a computing subsystem 20, a camera subsystem 30, a first FPGA chip 40, a bridge chip 50, a PCIE interface 60, a first Flash chip 70, a first JTAG interface 80 and a power module 90.

[0020] The storage subsystem 10, computing subsystem 20, camera subsystem 30, first Flash chip 70 and first JTAG interface 80 are all connected to the first FPGA chip 40, and the first FPGA chip 40 is connected to the PCIE interface 60 through the bridge chip 50; the power module 90 is connected to the PCIE interface 60.

[0021] The power module 90 is used to provide power to the storage subsystem 10 , the computing subsystem 20 , the camera subsystem 30 , the first Flash chip 70 , the first FPGA chip 40 and the bridge chip 50 .

[0022] Specifically, the system uses the first FPGA chip 40 as a communication hub, which is respectively connected to the storage subsystem 10 and the computing subsystem 20, connected to the CCD camera lens through the video decoding chip, and connected to the PCIE interface 60 through the bridge chip 50; the first FPGA chip 40 is connected to the first JTAG interface 80 and the first Flash chip 70 to store the configuration data of the first FPGA chip 40;

[0023] The storage subsystem 10 uses the CPLD chip 101 to implement independent storage control functions based on its own configuration, connects the DRAM 1 storage chip and the DRAM 2 storage chip to implement dual-channel memory access, connects the SRAM 1 storage chip as a high-speed off-chip data cache, and connects the second Flash chip to provide flash memory functions for the storage subsystem; the computing subsystem includes a second FPGA chip and a third FPGA chip for computing, and these two FPGA chips are connected to the first FPGA chip 40; the second FPGA chip is connected to the third JTAG interface and the third Flash chip to store the FPGA computing chip 1; the third FPGA chip is connected to the fourth JTAG interface and the fourth Flash chip.

[0024] The PCIE interface 60 provides power to the power module 90 , and the power module 90 provides 5V power and 1.5V power to the first FPGA chip 40 , the computing subsystem 20 , the camera subsystem 30 , the first Flash chip 70 , the first FPGA chip 40 and the bridge chip 50 .

[0025] The PCIE interface 60 sends the 100M clock signal provided by the mainboard to the first FPGA chip 40 , and the first FPGA chip 40 provides the clock signal to the storage subsystem 10 and the computing subsystem 20 respectively.

[0026] It is worth noting that the first FPGA chip 40, the second FPAG chip and the third FPAG chip in the computing subsystem 20, and the storage CPLD chip in the storage subsystem 10 all use their own inherent configurations and interfaces to implement their computing and control functions. The configurations are inherent to the corresponding components themselves. There is no need to modify any configuration when used in this application. You only need to make electrical data connections according to the inherent connection instructions of each component. After power-on, the inherent function initialization of each component can be automatically completed and then enter normal working state.

[0027] The aforementioned universal drone image processing system based on multiple FPGAs and Feiteng Maichuang heterogeneous processors utilizes a hardware structure that connects the FPGA computing array, Feiteng Maichuang FT-Matrix processor, CPLD chip, and other peripheral chips via a first FPGA chip. This allows for flexible expansion of computing and storage within the constraints of the first FPGA chip's limited hardware and pin resources. The computing subsystem, comprised of the Feiteng Maichuang FT-Matrix processor and two FPGA chips used for computing, achieves expanded computing capacity. The storage subsystem, comprised of SRAM and DRAM memory, is connected via a CPLD chip, reducing the hardware and pin resources used by the first FPGA chip to implement the memory access controller, achieving separation of computing and storage. This system utilizes a separate overall structure for the first FPGA chip, computing subsystem, and storage subsystem, overcoming the problem of a single FPGA's limited hardware resources being unable to meet demand.

[0028] In one embodiment, Figure 2 As shown, the storage subsystem 10 includes a CPLD chip 101, a DRAM module 102, an SRAM module 103, a second Flash module 104 and a second JTAG interface 105;

[0029] The DRAM module 102 , the SRAM module 103 and the second Flash module 104 are connected to the CPLD chip 101 , and the second JTAG interface 105 is connected to the CPLD chip 101 ; the CPLD chip 101 is connected to the first FPGA chip 40 .

[0030] Specifically, the storage subsystem 10 uses the CPLD chip 101 to connect the DRAM 1 memory chip and the DRAM 2 memory chip in the DRAM module 102 to achieve dual-channel memory access, connects the SRAM module 103 memory chip as a high-speed off-chip data cache, and connects the second Flash module 104 to store the configuration data of the CPLD chip.

[0031] The CPLD chip 101 provides clock signals to the DRAM 1 memory chip and the DRAM 2 memory chip in the DRAM module 102 , the SRAM module 103 , and the second Flash module 104 .

[0032] In one embodiment, Figure 2As shown, the computing subsystem includes: a Feiteng Maichuang FT-Matrix processor 2013, an MRAM chip 206, an FPGA computing array 201, a flash memory module and a JTAG interface module; the Feiteng Maichuang FT-Matrix processor 2013 is connected to the MRAM chip 206; the flash memory module and the JTAG interface module are connected to the FPGA computing array 201; and the FPGA computing array 201 is connected to the first FPGA chip 40.

[0033] The MRAM chip serves as the external memory of the FT-Matrix processor, realizing high-speed local cache of the digital processor.

[0034] In one embodiment, Figure 2 As shown, the camera subsystem 30 includes: a video decoding chip 302 and a CCD camera lens 301 ; the CCD camera lens 301 is connected to the video decoding chip 302 , and the video decoding chip 302 is connected to the first FPGA chip 40 .

[0035] In one embodiment, Figure 2 As shown, the FPGA computing array 201 includes a second FPGA chip 2011 and a third FPGA chip 2012 for computing; the flash memory module includes a third Flash chip 205 and a fourth Flash chip 204; the JTAG interface module includes a third JTAG interface 202 and a fourth JTAG interface 203;

[0036] The third Flash chip 205 and the third JTAG interface 202 are both connected to the second FPGA chip 2011, and the second FPGA chip 2011 is connected to the first FPGA chip 40;

[0037] The fourth Flash chip 204 and the fourth JTAG interface 203 are connected to the third FPGA chip 2012 , and the third FPGA chip 2012 is connected to the first FPGA chip 40 .

[0038] Specifically, the computing subsystem 20 includes a Feiteng Maichuang FT-Matrix heterogeneous processor 2013, two second FPGA chips 2011 and a third FPGA chip 2012 for auxiliary computing; the Feiteng Maichuang FT-Matrix processor 2013, the second FPGA chip 2011 and the third FPGA chip 2012 are connected to the first FPGA chip 40; the Feiteng Maichuang FT-Matrix processor 2013 is connected to the MRAM chip 206; the second FPGA chip 2011 is connected to the third JTAG interface 202 and the third Flash chip 205; the second FPGA chip 2012 is connected to the fourth JTAG interface 203 and the fourth Flash chip 204.

[0039] The computing array 201 is composed of a Feiteng Maichuang FT-Matrix processor 2013 and two second FPGA chips 2011 and a third FPGA chip 2012 for computing. The computing array 201 adopts the structure of Feiteng Maichuang FT-Matrix processor and dual FPGA chips to achieve expansion of computing capacity and improve the computing power of the system; by using the hardware structure of the CPLD chip 101 to connect the SRAM memory and DRAM memory, the separation of computing and storage is achieved; by connecting the FPGA computing array 201, Feiteng Maichuang FT-Matrix processor 2013, CPLD chip 101 and other peripheral chips through the first FPGA chip 40, flexible expansion of computing and storage can be achieved under the constraints of limited hardware resources and pin resources of the first FPGA chip.

[0040] The PCIE interface provides power to the power module 90, which then supplies 5V and 1.5V power to the first FPGA chip 40, the second FPGA chip 2011, the second FPGA chip 2012, the CPLD chip 101, the video decoder chip 302, the bridge chip 50, the DRAM 1 and DRAM 2 memory chips in the DRAM module 102, the SRAM module 103, the first Flash chip 70, the second Flash chip 104, the third Flash chip 205, and the fourth Flash chip 204. The PCIE interface 60 transmits the 100M clock signal provided by the motherboard to the first FPGA chip 40, which in turn supplies the clock signal to the CPLD chip 101, the second FPGA chip 2011, and the second FPGA chip 2012, respectively. The CPLD chip 101 supplies the clock signal to the DRAM 1 and DRAM 2 memory chips in the DRAM module 102, the SRAM module 103, and the second Flash chip 104, respectively. The second FPGA chip 2011 provides a clock signal to the third Flash chip 205 , and the third FPGA chip 2012 provides a clock signal to the fourth Flash chip 204 .

[0041] like Figure 2As shown, the operating process of the universal image processing system for drones based on multiple FPGAs and Feiteng Maichuang heterogeneous processors is as follows: After the drone system starts up, the PCIE interface 60 provides power to the power module 90. The power module 90 supplies 5V and 1.5V power to the first FPGA chip 40, the second FPGA chip 2011, the third FPGA chip 2012, the FT-Matrix processor 2013, the MRAM chip 206, the CPLD chip 101, the video decoding chip 302, the bridge chip 40, the DRAM 1 and DRAM 2 memory chips in the DRAM module 102, the SRAM module 103, the first Flash chip 70, the second Flash chip 104, the third Flash chip 205, and the fourth Flash chip 204. The PCIE interface 60 transmits the 100M clock signal provided by the computer motherboard to the first FPGA chip 40. The first FPGA chip 40 loads the logic and provides the clock signal to the FT-Matrix processor 2013, the CPLD chip 101, the second FPGA chip 2011, and the third FPGA chip 2012. The CPLD chip 101 provides clock signals to the DRAM 1 and DRAM 2 memory chips in the DRAM module 102, the SRAM module 103, and the second Flash chip 104. The second FPGA chip 2011 provides clock signals to the third Flash chip 205 and loads configuration information. The third FPGA chip 2012 provides clock signals to the third Flash chip 204 and loads configuration information. After the first FPGA chip 40 is officially started, it continuously receives video input image frames from the video decoding chip 302. It collaborates with the storage subsystem and the computing subsystem to complete real-time image processing. The processing results are transmitted to the host computer via the PCIE interface 60.

[0042] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0043] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the utility model patent. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A general image processing system for unmanned aerial vehicles based on multiple FPGAs and Feiteng Maichuang heterogeneous processors, characterized in that: The system includes: a storage subsystem, a computing subsystem, a camera subsystem, a first FPGA chip, a bridge chip, a PCIE interface, a first Flash chip, a first JTAG interface and a power module; The storage subsystem, the computing subsystem, the camera subsystem, the first Flash chip, and the first JTAG interface are all connected to the first FPGA chip, and the first FPGA chip is connected to the PCIE interface through the bridge chip; the power module is connected to the PCIE interface; The power module is used to provide power to the storage subsystem, the computing subsystem, the camera subsystem, the first Flash chip, the first FPGA chip and the bridge chip.

2. The system according to claim 1, characterized in that The storage subsystem includes a CPLD chip, a DRAM module, an SRAM module, a second Flash module and a second JTAG interface; The DRAM module, the SRAM module and the second Flash module are connected to the CPLD chip, the second JTAG interface is connected to the CPLD chip; and the CPLD chip is connected to the first FPGA chip.

3. The system according to claim 1, characterized in that The computing subsystem includes: Feiteng Maichuang FT-Matrix heterogeneous processor, MRAM chip, FPGA computing array, flash memory module and JTAG interface module; the Maichuang FT-Matrix processor is connected to the MRAM chip; the flash memory module and the JTAG interface module are connected to the FPGA computing array; the FPGA computing array is connected to the first FPGA chip.

4. The system according to claim 1, characterized in that The camera subsystem includes: a video decoding chip and a CCD camera lens; The CCD camera lens is connected to the video decoding chip, and the video decoding chip is connected to the first FPGA chip.

5. The system according to claim 3, characterized in that The FPGA computing array includes a second FPGA chip and a third FPGA chip for computing; the flash memory module includes a third Flash chip and a fourth Flash chip; the JTAG interface module includes a third JTAG interface and a fourth JTAG interface; The third Flash chip and the third JTAG interface are both connected to the second FPGA chip, and the second FPGA chip is connected to the first FPGA chip; The fourth Flash chip and the fourth JTAG interface are both connected to the third FPGA chip, and the third FPGA chip is connected to the first FPGA chip.

Images