Image recognition architecture based on JFMQL100TAI programmable logic device

Abstract

The invention relates to an image recognition architecture based on a JFMQL100TAI programmable logic device, which comprises a controller, a main processing chip and three memories DDR3, an EMIF interface and an SRIO interface of the controller are connected with an EMIF interface and an SRIO interface of the main processing chip, the main processing chip is respectively connected with the three memories DDR3 through buses, the model of the controller is FT6678, the model of the main processing chip is JFMQL100TAI, and the model of the memory DDR3 is JFMQL100TAI. The memory DDR3 comprises a first double-rate synchronous dynamic random access memory, a second double-rate synchronous dynamic random access memory and a third double-rate synchronous dynamic random access memory, so that airborne video acquisition, caching, target recognition processing, character superposition and display output are realized; the system has the advantages of high flexibility, low power consumption, small size, high integration level and light weight, and is very suitable for an airborne application environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of airborne photoelectric detection and countermeasure, and specifically relates to an image recognition architecture based on a JFMQL100TAI programmable logic device. Background Art

[0002] The field of airborne optoelectronic detection and countermeasure involves a large number of image recognition and video transmission tasks. With the increasing complexity of image processing tasks and the development trend of high-quality design and low-cost implementation of airborne optoelectronic products, there is an urgent need for a more adaptable, higher-performance, and lower-cost image recognition architecture to improve the core competitiveness of airborne optoelectronic products.

[0003] At the same time, the image recognition architecture is mainly used to realize image processing technologies such as video transmission and display, image recognition, etc. However, there are still many problems in the specific use of the image recognition architecture. For example, the commonly used FPGA+FT6678+Shengteng 310 image recognition architecture has the following problems in its application in the airborne field: first, the peripheral circuit and structural design are complicated, resulting in no cost advantage; second, the operating temperature range is narrow, the environmental adaptability is not strong, and the promotion and use are not significant; finally, the data flow interaction is complex, debugging is difficult, and the requirements for operators are relatively high. Therefore, there is an urgent need for an image recognition architecture based on the JFMQL100TAI programmable logic device. Summary of the invention

[0004] In view of this, the present invention proposes an image recognition architecture based on the JFMQL100TAI programmable logic device, which is applied to the field of airborne optoelectronic detection and countermeasure technology to solve the existing technical problems of complicated peripheral circuit and structural design, narrow operating temperature range and complex data flow interaction.

[0005] In order to achieve the above technical objectives, the specific technical solutions adopted by the present invention are:

[0006] An image recognition architecture based on a JFMQL100TAI programmable logic device comprises a controller, a main processing chip and three DDR3 memories. The EMIF interface and SRIO interface of the controller are connected with the EMIF interface and SRIO interface of the main processing chip. The main processor chip is respectively connected with the three DDR3 memories through buses. The model of the controller is FT6678, the model of the main processing chip is JFMQL100TAI, and the memory DDR3 comprises a first double rate synchronous dynamic random access memory, a second double rate synchronous dynamic random access memory and a third double rate synchronous dynamic random access memory.

[0007] Furthermore, it also includes an SDI video acquisition module, a PS processing system, an AI acceleration engine NPU, a cache display module, an ARINC818 video output module and an SRIO communication module. The main processing chip includes a PL unit and a PS unit. The PL unit is used for SDI video acquisition, AI acceleration engine NPU calculation, cache display and ARINC818 video output. The PS unit is used for data interaction control of the AI ​​acceleration engine NPU, data interaction control of the cache display and communication of the EMIF interface.

[0008] Furthermore, a receiving sensor is provided on the SDI video acquisition module for outputting multiple channels of SDI videos to be processed and converting them into external synchronization interface data output.

[0009] Furthermore, the first double data rate synchronous dynamic random access memory is connected to the PS processing system, and the second double data rate synchronous dynamic random access memory and the third double data rate synchronous dynamic random access memory are connected to the PL unit through a bus.

[0010] Furthermore, the PS side of the main processor chip processing system adopts a "three-core with single core" architecture based on the amp mode, in which cores core0 to core2 are used for AI computing, and core3 is used for image display processing.

[0011] Furthermore, the AI ​​acceleration engine NPU includes a PL pre-processing unit and an AI acceleration engine NPU, and the PL pre-processing unit and the AI ​​acceleration engine NPU communicate through an AXI4 bus.

[0012] Furthermore, the SDI video acquisition module outputs an external synchronization format video, and the external synchronization format video is converted into a VDMA data stream via the AXI4 bus and then cached in a third double rate synchronous dynamic random access memory.

[0013] Furthermore, the ARINC818 video output module superimposes the identification characters output by the controller FT6678 with the video to be output according to the video output timing, converts the information into information conforming to the ARINC818 interface protocol, and then outputs the information.

[0014] Furthermore, the SRIO communication module implements SRIO interface communication between the main processing chip and the controller through the SRIO interface.

[0015] Furthermore, the PL unit collects video and sends the image frame number to the PS processing system. The PS processing system interactively controls the AI ​​acceleration engine NPU to perform target recognition, and sends the target recognition result to the controller via the EMIF interface. The recognition result is drawn into character information and then sent to the PL unit through the SRIO interface.

[0016] The JFMQL100TAI programmable logic device integrates PL module and PS module. The PL module is used for SDI video acquisition, AI acceleration engine NPU calculation, cache display and ARINC818 video output. The PS module is used for AI acceleration engine NPU, cache display data interaction control and EMIF interface communication.

[0017] The PL module mainly includes the SDI video acquisition module, the AI ​​acceleration engine NPU, the cache display module and the ARINC818 video output module. The PS module contains 4 Cortex-A53 processors, including a single-core module and a triple-core module. The single-core module drives and controls the AXI VDMA IP core, AXI SCALER IP core and AXI MIXER IP core of the cache display module in the PL module, and also initializes the SRIO communication module in the PL module. The triple-core module includes the AI ​​control module and the EMIF communication module. The AI ​​control module interactively controls the AI ​​acceleration engine NPU in the PL module, reads the calculation results in the second double-rate synchronous dynamic random access memory DDR3 cache area, writes them to the first double-rate synchronous dynamic random access memory DDR3 cache area, and performs the final target detection and recognition operation.

[0018] The principle of this invention is to utilize the image recognition architecture of JFMQL100TAI, in which the target recognition operation is based on the RT-DETR algorithm, and combined with feature pyramid, spatial pyramid sampling, residual network and other structures, to design a visible light target recognition model based on scale perception and an infrared target recognition model based on visual attention mechanism. At the same time, the AI ​​acceleration engine NPU is used for hardware acceleration to realize real-time automatic recognition of airborne dynamic video.

[0019] By adopting the above technical solution, the present invention can also bring the following beneficial effects:

[0020] 1. The present invention proposes an image recognition architecture based on the JFMQL100TAI programmable logic device. By using the JFMQL100TAI main processor chip as a carrier and using the AXI4 bus for internal interconnection and real-time data interaction between the PS module and the PL module, a "three-core with a single-core" architecture is designed based on the amp mode. The three-cores are used to drive the AI ​​acceleration engine NPU and realize EMIF interface communication, and the single-core is used to drive the cache display module and initialize SRIO interface communication. The functions of video acquisition, caching, target recognition, display output, etc. are integrated into a single chip. The high integration, high-speed and high-performance image recognition and video transmission can meet the real-time and accuracy requirements of airborne optoelectronic target recognition.

[0021] 2. The present invention proposes an image recognition architecture based on the JFMQL100TAI programmable logic device, which realizes the recognition of different types of targets through optimized target recognition models and algorithms, improves the recognition accuracy, can meet the compatibility requirements of airborne optoelectronic target recognition, has strong scalability, is compatible with the existing ecology, and has strong flexibility.

[0022] 3. The present invention proposes an image recognition architecture based on the JFMQL100TAI programmable logic device. Based on the JFMQL100TAI main processor chip AI acceleration engine NPU, the target recognition operation of a single image can be completed within the frame, and the recognition result can be synchronized with the real-time video through video display verification, meeting the requirements of airborne optoelectronic target recognition functions.

[0023] 4. The present invention proposes an image recognition architecture based on the JFMQL100TAI programmable logic device, using the cost-effective JFMQL100TAI main processor chip, whose single-chip cost is lower than that of the commonly used recognition architecture and has a wide operating temperature range (-55°C to 125°C). BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 The present invention is a schematic diagram of a structural framework of an image recognition architecture based on a JFMQL100TAI programmable logic device;

[0026] Figure 2 The present invention mentions a schematic diagram of a data communication framework in an image recognition architecture based on a JFMQL100TAI programmable logic device;

[0027] Figure 3 The present invention mentions a schematic diagram of a data communication architecture in an image recognition architecture based on a JFMQL100TAI programmable logic device. DETAILED DESCRIPTION

[0028] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0029] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0030] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present invention, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0031] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0032] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0033] Example 1

[0034] In one embodiment of the present invention, see Figure 1 and Figure 2The present invention mentions an image recognition architecture based on a JFMQL100TAI programmable logic device. The structural framework uses the JFMQL100TAI programmable logic device of Fudan Microelectronics Corporation with an ARM+FPGA+AI functional architecture as the main processor chip. The designed image board has SDI, ARINC818, SRIO and EMIF interfaces. The JFMQL100TAI main processing chip is connected to the sensor through the SDI interface, connected to the controller FT6678 through the SRIO interface and the EMIF interface, and connected to the terminal through the ARINC818 interface to realize video acquisition input, communication and display output.

[0035] The JFMQL100TAI programmable logic device integrates PL module and PS module. The PL module is used for SDI video acquisition, AI acceleration engine NPU calculation, cache display and ARINC818 video output. The PS module is used for AI acceleration engine NPU, cache display data interaction control and EMIF interface communication.

[0036] The PL module mainly includes an SDI video acquisition module, an AI acceleration engine NPU, a cache display module and an ARINC818 video output module. The SDI video acquisition module includes a SMPTE SD / HD / 3G-SDI IP core and a synchronization code analysis module. The SMPTE SD / HD / 3G-SDI IP core generates a 20-bit data stream with an embedded synchronization signal from the received video data. The synchronization code analysis module converts the 20-bit data into a video stream that meets the external synchronization format, thereby realizing SDI video acquisition.

[0037] The AI ​​acceleration engine NPU reads the preprocessed image in the second double-rate synchronous dynamic random access memory DDR3 cache area to perform target detection and recognition operations, and returns the operation results to the second double-rate synchronous dynamic random access memory DDR3 cache area.

[0038] The cache display module includes AXI VDMA IP core, AXI SCALER IP core and AXI MIXER IP core. The AXIVDMA IP core realizes the reading and writing of AXI Stream video stream in the third double-rate synchronous dynamic random access memory DDR3 cache area, the AXI SCALER IP core realizes the reduction and enlargement of AXI Stream video stream, and the AXI MIXER IP core realizes the layer display of AXI Stream video stream.

[0039] The ARINC818 video output module includes the AXI4-Stream To Video out IP core, the Video Timing Controller IP core and the ARINC818 timing generation module. The AXI4-Stream To Video out IP core realizes the conversion between the AXIStream video stream and the external synchronization format video stream. The Video Timing Controller IP core generates the timing required for the subsequent video output. The ARINC818 timing generation module outputs the processed video data and recognition results according to the corresponding ARINC818 output format.

[0040] The PL module also includes an SRIO communication module, which realizes the SRIO interface communication function with the controller FT6678 and is used to receive the recognition character information and other calculation information calculated by the controller FT6678.

[0041] The PL module also includes an EMIF bridge module to implement data conversion between the EMIF bus and the AXI BRAM.

[0042] The PS module contains four Cortex-A53 processors, including a single-core module and a triple-core module. The single-core module drives and controls the AXI VDMA IP core, AXI SCALER IP core, and AXI MIXER IP core of the cache display module in the PL module, and also implements the initialization of the SRIO communication module in the PL module.

[0043] The three-core module includes an AI control module and an EMIF communication module. The AI ​​control module interactively controls the AI ​​acceleration engine NPU in the PL module, reads the calculation results in the second double-rate synchronous dynamic random access memory DDR3 cache area, writes them to the first double-rate synchronous dynamic random access memory DDR3 cache area, and performs the final target detection and recognition operation.

[0044] The EMIF communication module implements EMIF bus reading and writing by reading and writing accesses to the AXI BRAM connected to the EMIF bridge module of the PL module.

[0045] Meanwhile, in the specific communication architecture of the image recognition architecture of the present invention, the PS module and the PL module are connected via an AXI4 bus to achieve data communication between the modules.

[0046] The AXI VDMA IP core, AXI SCALER IP core, and AXI MIXER IP core in the cache display module of the PL module are connected to the AXI4 GP0 interface on the PS side through the AXI4 Interconnect IP core for driving the single-core module in the PS module. The AI ​​acceleration engine NPU is connected to the AXI4 HP1 and AXI4 HP3 interfaces on the PS side for interactive control of the three-core modules in the PS module, thereby realizing data interactive transmission between the PS module and the PL module.

[0047] The three-core module of the PS module reads the first line of sensor annotation information of the external synchronization format video output by the SDI video acquisition module of the PL module through AXI BRAM, and outputs the final recognition result to the PL module through AXI BRAM. The three-core module of the PS module reads and writes the EMIF bus data on the EMIF bridge module of the PL module through AXI BRAM. The single-core module of the PS module interactively controls the SRIO communication module of the PL module through AXI Lite.

[0048] See also Figure 2 The present invention mentions an image recognition architecture based on JFMQL100TAI programmable logic device, and the specific structural design and implementation principle and process are as follows:

[0049] 1. The SDI video acquisition module sends the external high-speed serial data to the SMPTE SD / HD / 3G-SDI IP core. The SMPTE SD / HD / 3G-SDI IP core decodes the high-speed serial data to obtain 20-bit data. The synchronization code analysis module converts the 20-bit data into video field synchronization, line synchronization, data validity and data signals according to the embedded SAV and EAV synchronization codes in accordance with the SMPTE274M-2003 SDI protocol.

[0050] 2. After the video data is converted into a data format that meets the AXI Stream bus interface protocol, it is sent to the first double-rate synchronous dynamic random access memory DDR3 cache area through PS AXI4 HP2 after pre-processing such as size cropping; after the video data is converted into a data format that meets the AXI Stream bus interface protocol, it is sent to the second double-rate synchronous dynamic random access memory DDR3 cache area after the three-core module of the PS module drives the AI ​​acceleration engine NPU after pre-processing such as size cropping; after the video data is converted into a data format that meets the AXI Stream bus interface protocol, it is sent to the third double-rate synchronous dynamic random access memory DDR3 cache area after the single-core module of the PS module drives the AXI VDMA IP core;

[0051] 3. The single-core module of the PS module drives the AXI VDMA IP core to realize the read and write cache of the video stream in the third double-rate synchronous dynamic random access memory DDR3 buffer area, drives the AXI SCALER IP core to realize the zooming in and out of the video stream to achieve the electronic zoom effect, and drives the AXI MIXER IP core to realize the layer display and black edge overlay display of the video stream;

[0052] 4. The three-core module of the PS module drives the AI ​​acceleration engine NPU to read and write the video stream in the second double-rate synchronous dynamic random access memory DDR3 buffer area, perform corresponding neural network pre-processing operations, and write the operation results to the first double-rate synchronous dynamic random access memory DDR3 buffer area through PS AXI4HP3 for the remaining post-processing calculations;

[0053] 5. The three-core module of the PS module reads the calculation results of the AI ​​acceleration engine NPU, first decodes the features of the neural network model inference results output by the NPU, then uses the target confidence threshold to filter out targets greater than the set threshold as the output recognition results, and selects a recognition result with the highest confidence as the target recognition area, performs non-maximum suppression processing on the area by category, and calculates the target position, size, category and other information of the final recognition output;

[0054] 6. The three-core module of the PS module sends the recognition result to the controller FT6678 through the EMIF interface, and sends the recognition result to the PL module through AXIBRAM;

[0055] 7. The single-core module of the PS module interactively controls the SRIO communication module of the PL module through AXI-LITE, completes the SRIO interface initialization between JFMQL100TAI and the controller FT6678, and receives the recognition character information calculated by the controller FT6678;

[0056] 8. The ARINC818 video output module of the PL module generates the timing format required for ARINC818 video output through the Video Timing Controller IP core. The AXI4-Stream To Video out IP core converts the video stream into an external synchronization format video stream, generates ARINC818 timing through the ARINC818 timing generation module, fills the recognition result into the ARINC818 FC0 frame, and fills the video data after the recognition character information is superimposed into the ARINC818 FCn frame;

[0057] The data communication architecture of the present invention is as follows Figure 3 As shown, the PS module, the PL module and the double rate synchronous dynamic random access memory DDR3 are connected through the AXI4 bus to achieve high-speed data communication between the modules.

[0058] The specific data communication principle and process are as follows:

[0059] 1. The external synchronization format video output by the SDI video acquisition module of the PL module is converted into a data format that meets the AXI4Stream bus interface protocol and then saved to the first-rate synchronous dynamic random access memory DDR3 through the PS AXI4 HP2 interface, saved to the second-rate synchronous dynamic random access memory DDR3 through the AXI4 bus, and saved to the third-rate synchronous dynamic random access memory DDR3 through the AXI4 bus of the AXI VDMA IP core;

[0060] 2. The three-core module of the PS module is connected to the AI ​​acceleration engine NPU through PS AXI4 HP1 and PS AXI4 HP3. The PS AXI4 HP1 three-core module interactively controls the CDMA reading and writing of the AI ​​acceleration engine NPU. The PS AXI4 HP3 three-core module writes the last layer feature map of the AI ​​acceleration engine NPU into the first rate synchronous dynamic random access memory DDR3 buffer;

[0061] 3. The PS three-core module writes the recognition result into the dual-port BRAM buffer through AXI BRAM. The PL module reads the recognition result in the dual-port BRAM buffer and sends the recognition result and video to the terminal through the ARINC818 timing generation module. The SDI video acquisition module of the PL module outputs the first line of sensor annotation information of the external synchronous video and writes it into the dual-port BRAM buffer. The PS three-core module reads it through AXI BRAM.

[0062] 4. The PS three-core module writes the recognition results into the dual-port BRAM buffer through AXI BRAM. The PL module reads the recognition results in the dual-port BRAM buffer and maps them to the EMIF bus for FT6678 to read. At the same time, FT6678 writes control commands and other information to the EMIF bus. The PL module reads and writes them into the dual-port BRAM buffer. The PS three-core module reads them through AXI BRAM.

[0063] 5. The PS single-core module interactively controls the initialization of the SRIO communication module in the PL module through the AXI Lite interface, and drives IP cores such as AXI VDMA, AXI SCALER, and AXI MIXER through the AXI Lite interface.

[0064] In summary, the present invention is an image recognition architecture based on the JFMQL100TAI programmable logic device, which realizes airborne video acquisition, caching, target recognition processing, character overlay and display output, and has the characteristics of high flexibility, low power consumption, small size, high integration, light weight, etc., and is suitable for airborne application environments.

[0065] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An image recognition architecture based on JFMQL100TAI programmable logic device, characterized by: The invention comprises a controller, a main processing chip and three DDR3 memories. The EMIF interface and the SRIO interface of the controller are connected with the EMIF interface and the SRIO interface of the main processing chip. The main processor chip is respectively connected with the three DDR3 memories through buses. The model of the controller is FT6678. The model of the main processing chip is JFMQL100TAI. The DDR3 memories comprise a first double rate synchronous dynamic random access memory, a second double rate synchronous dynamic random access memory and a third double rate synchronous dynamic random access memory.

2. According to claim 1, an image recognition architecture based on JFMQL100TAI programmable logic device is characterized in that: It also includes an SDI video acquisition module, a PS processing system, an AI acceleration engine NPU, a cache display module, an ARINC818 video output module and an SRIO communication module. The main processing chip includes a PL unit and a PS unit. The PL unit is used for SDI video acquisition, AI acceleration engine NPU calculation, cache display and ARINC818 video output. The PS unit is used for data interaction control of the AI ​​acceleration engine NPU and cache display, and EMIF interface communication.

3. According to claim 2, an image recognition architecture based on JFMQL100TAI programmable logic device is characterized in that: The SDI video acquisition module is provided with a receiving sensor for outputting multiple channels of SDI videos to be processed and converting them into external synchronization interface data output.

4. According to claim 3, an image recognition architecture based on JFMQL100TAI programmable logic device is characterized in that: The first double rate synchronous dynamic random access memory is connected to the PS processing system, and the second double rate synchronous dynamic random access memory and the third double rate synchronous dynamic random access memory are connected to the PL unit through a bus.

5. According to claim 4, an image recognition architecture based on JFMQL100TAI programmable logic device is characterized in that: The PS side of the main processor chip processing system adopts a "three-core with single core" architecture based on the amp mode, in which cores core0 to core2 are used for AI computing, and core3 is used for image display processing.

6. The image recognition architecture based on JFMQL100TAI programmable logic device according to claim 5, characterized in that: The AI ​​acceleration engine NPU includes a PL pre-processing unit and an AI acceleration engine NPU, and the PL pre-processing unit and the AI ​​acceleration engine NPU communicate through an AXI4 bus.

7. The image recognition architecture based on JFMQL100TAI programmable logic device according to claim 6, characterized in that: The SDI video acquisition module outputs external synchronization format video, and the external synchronization format video is converted into a VDMA data stream via the AXI4 bus and then cached in a third double rate synchronous dynamic random access memory.

8. The image recognition architecture based on JFMQL100TAI programmable logic device according to claim 7, characterized in that: The ARINC818 video output module superimposes the identification characters output by the controller FT6678 with the video to be output according to the video output timing, converts them into information that complies with the ARINC818 interface protocol, and then outputs them.

9. The image recognition architecture based on JFMQL100TAI programmable logic device according to claim 8, characterized in that: The SRIO communication module implements SRIO interface communication between the main processing chip and the controller through the SRIO interface.

10. The image recognition architecture based on JFMQL100TAI programmable logic device according to claim 9, characterized in that: The PL unit collects video and sends the image frame number to the PS processing system. The PS processing system interactively controls the AI ​​acceleration engine NPU to perform target recognition, and sends the target recognition result to the controller via the EMIF interface. The recognition result is drawn into character information and then sent to the PL unit through the SRIO interface.

Images