Heterogeneous computing architecture test method of ISP module

Abstract

The invention discloses a heterogeneous computing architecture testing method of an ISP module, and belongs to the technical field of heterogeneous computing. According to the technical scheme, a high-level synthesis tool is used for generating and verifying an IP to be tested; a target algorithm IP application system is constructed by using a microprocessor IP hardcore and an AXI bus connector of a heterogeneous computing platform, a target algorithm IP core is operated by using software to carry out an application test, and actually measured data is stored in a nonvolatile storage medium; transplanting an operation system and an OpenCV library to the heterogeneous computing platform for the development board, and storing a target algorithm software operation result to a data file after a test program is operated; and evaluating the function correctness and the performance index of the target algorithm IP by using the actually measured data and the software operation result. The method has the advantages that effective testing and optimization of the heterogeneous computing architecture of the ISP module are achieved through the software and hardware co-simulation and IP core verification technology, middle-level optimization, a comprehensive testing framework and other methods, and powerful support is provided for technology development, application and popularization in related fields.

Description

Technical Field

[0001] The present invention belongs to the technical field of heterogeneous computing, and in particular to a heterogeneous computing architecture testing method for an ISP module. Background Art 1. Background of the invention

[0003] With the rapid development of today's microelectronics industry, the continuous strengthening of the design capabilities of the entire integrated circuit industry, and the gradual improvement of process levels have substantially increased the complexity of large-scale digital circuits, while the circuit size has remained unchanged or is still shrinking. The development of deep submicron semiconductor technology has made it possible to integrate the entire embedded system on a single chip. In other words, the capacity of integrated circuits (ICs) has grown to the point where soft processors and hard processors can coexist on a single chip to form a programmable system on chip (SoPC). Over the years, the continuous advancement of the semiconductor manufacturing industry and the continuous increase in the complexity of SoPC design have led to more and more IP (Intellectual Property) modules being integrated into the same chip, such as Figure 1 Figure 1 shows a typical SoPC design architecture. Large SoPCs may even integrate hundreds of IP cores to provide different functions such as network communication, data storage, fast computing, multimedia applications, etc.

[0004] The design of contemporary large-scale SoPC is mainly based on the idea of ​​bus interconnection and IP multiplexing technology. The bus protocols come from different manufacturers. Each type of bus protocol specifies a unique data transmission method. Communication between different buses requires data conversion modules such as bus bridges. The IP cores used in SoPC can be developed by ourselves or purchased from other manufacturers.

[0005] The first is to purchase IP cores from other manufacturers: IP cores from different manufacturers not only have different design concepts, but also have different code styles and functional models. In addition, the bus protocol specifications may cause IP cores to affect each other in SoPC, making it impossible to realize their own functions in the system. Therefore, before the purchased IP core is finally integrated into the SoPC design, it is very necessary for the user to re-verify its functions and performance in the overall system. On the one hand, it can test the compatibility between the IP core and the existing system, and on the other hand, it greatly reduces the design and use risks and reduces the error rate of the overall function in the later stage of SoPC design.

[0006] The second is to develop IP core by ourselves: Since the development board used in this project is from Xilinx, the integrated development tools provided by Xilinx will be analyzed and introduced below.

[0007] The integrated development environment consists of three software: Vivado HLS, Vivado and Vivado SDK. The HLS tool is mainly used for simulation verification of modules and IP levels before system integration. When designing IP, test functions are generated by adding TestBench to perform C or C and RTL joint simulation. However, on the premise that the software simulation passes, the generated RTL-level hardware circuit may still fail, but it is difficult to locate problems in high-level languages ​​due to the above reasons, resulting in debugging difficulties. The Vivado SDK tool is used for module and IP verification after system integration, mainly to implement operations such as IP and related peripheral drivers, but the IP core is a highly encapsulated whole for the SDK. Even in debug mode, the data in the processing process of the IP core cannot be obtained. Therefore, it is impossible to assist developers in analysis and testing.

[0008] To sum up: In the overall process of chip design, verification occupies a large workload, and how to verify the IP core in real time and effectively becomes the key to hardware design. Using EDA simulation software to dynamically simulate the design is the most widely used functional verification method. After continuous improvement by designers, a mature verification methodology (UVM) has been developed. Although it has good internal signal visibility, real-time debugging and reusability, when using EDA software for simulation, a large number of verification stimuli will make the verification cycle as long as several days. In order to overcome the problem of long software simulation verification cycle, FPGA prototype verification has become one of the most effective solutions. However, since FPGA prototype verification requires the construction of a test framework around the design to be verified, it brings great inconvenience to designers. In order to solve this problem, researchers have proposed a method of using a combination of software and hardware to increase the reusability of the verification framework while ensuring the efficiency advantage of FPGA simulation. Currently common software and hardware combined design architectures are such as Figure 2 shown.

[0009] 2. Research status

[0010] 2.1. Research status of ISPIP module development and optimization under HLS tools

[0011] like Figure 3 As shown in (a) and (b), there are two main ways to develop IP cores: (1) Using Vivado tools to design based on register transfer level (RTL). If the design phase takes up 20% of the R&D time, then 80% of the time will be spent on debugging to make it work. (2) Design based on C language and IP cores. The design efficiency of the first phase will increase by 10-15 times, and the efficiency of subsequent design phases will increase by 40 times.

[0012] The high-level synthesis (HLS) tool provided by FPGA giant Xilinx can realize system-level design input based on C language and IP core. And HLS can accelerate the development process and realize system integration based on C, C++ language library and IP core integrator. The IP core design process based on HLS tool is as follows: Figure 4 As shown in the figure, the design process of Vivado HLS is divided into C code design of source files, verification of source files, high-level synthesis and continuous iterative optimization, RTL output and packaging of IP cores.

[0013] The performance improvement tools for systems developed using HLS can be divided into three parts according to the development process:

[0014] 1) Lower-level compilation optimization. HLS is essentially a compiler that can convert high-level languages ​​into RTL-level designs, so the design of lower-level compilation is undoubtedly crucial. However, the lower-level compilation problem is automatically solved by Vivado-related tools after adding optimization instructions.

[0015] 2) Optimize memory access and parallelism in the middle layer. The first is memory access optimization. How to use the storage layer on the FPGA chip to improve the throughput of the application is one of the important issues in the middle layer design. The second is parallelism optimization. A single PE of the FPGA cannot be scaled up indefinitely by occupying more on-chip resources to obtain higher performance. Therefore, in order to fully utilize the computing resources on the FPGA, it is necessary to design a good parallel architecture and improve the performance of the system by having multiple PEs working in parallel.

[0016] 3) Ease of use optimization of higher-level domain-specific languages ​​(DSL) or RTL verification. Currently, HLS still has problems such as incomplete instructions and difficult debugging. These problems do not directly affect the final system performance.

[0017] The HLS tool provides a large number of optional compilation instructions (pragma), and users can insert compilation instructions into the program in the form of comments. During the compilation process, HLS will automatically reconstruct the code according to the pragma without the user having to change the code manually. Therefore, the modules designed on HLS can be directly programmed using high-level languages ​​such as C, and the overall performance can be optimized by setting optimization variables and parameters in the Directive control bar, or using the #pragma command in the code to define the variable implementation type and structure.

[0018] 2.2. Research Status of Heterogeneous Computing Architecture

[0019] Heterogeneous computing refers to the use of computing units with multiple different computing architectures to form a computing system. Common computing units include central data processing units, graphics processing units (GPUs), application specific integrated circuits (ASICs), digital signal processors (DSPs), and field programmable gate arrays (FPGAs).

[0020] In recent years, with the rapid development of artificial intelligence, big data mining and the Internet of Everything, CPU, a processor suitable for serial computing, is unable to cope with large amounts of complex data processing and cannot meet the needs of users with high complexity and high real-time performance. However, heterogeneous computing systems have attracted widespread attention due to their excellent processing capabilities and parallel characteristics. General heterogeneous computing platforms usually include heterogeneous computing units, reconfigurable devices and controllers, network interfaces, and memory.

[0021] At present, the most widely used heterogeneous computing platforms mainly refer to CPU+GPU heterogeneous computing platforms, CPU+FPGA heterogeneous computing platforms, and the combination of the two. From a hardware perspective, the CPU is mainly used for logic control, while heterogeneous hardware is used as a high-performance accelerator to perform computing tasks. GPU is a processor that processes graphics and images. It consists of a large number of computing units and an extremely long pipeline, and is good at large-scale parallel processing of a single type of data. FPGA is a functionally reconfigurable hardware resource. The underlying layer is based on a lookup table (LUT). The function of its internal logic unit is determined when it is reprogrammed, so FPGA is instruction-independent.

[0022] The computing performance of FPGA is related to the clock frequency and the number of resources such as RAM and DSP. The algorithm implemented by FPGA is expressed in sequential logic, which is based on clock processing. Resources such as RAM and DSP will affect the degree of parallel execution of data, thereby affecting the computing efficiency of FPGA. GPU is limited by the pipeline depth and can only achieve data parallelism, while FPGA can build multi-level pipeline parallelism and achieve data parallelism at the same time. When processing tasks arrive one by one, the pipeline parallelism of FPGA will bring greater flexibility, thereby obtaining lower latency. For computationally intensive tasks, the throughput, latency and power consumption of FPGA are better than those of CPU and GPU, and because of the programmable nature of FPGA, the logic can be updated in a short time. This flexibility will also bring better economic benefits. For communication-intensive tasks, FPGA can achieve stable and extremely low latency, which is suitable for processing streaming tasks. Table 1 shows the comparison of several indicators of CPU, GPU and FPGA under computationally intensive tasks.

[0023] Table 1 Comparison of parameters of several processors under computationally intensive tasks

[0024] processor Throughput Delay Power consumption Flexibility CPU ～1T N / A ～100w Very high GPU ～10T ~1ms ～300w high FPGA ～1T ~1μs ～30w high

[0025] 2.3. Research status of IP simulation and verification methods based on heterogeneous computing architecture

[0026] With the development of heterogeneous computing architecture technology, many IC design simulation and verification solutions have also combined the idea of ​​heterogeneous computing software and hardware collaboration, thereby speeding up the design process and improving the accuracy of verification.

[0027] Huang C et al. proposed four FPGA-based verification modes after analyzing the advantages and disadvantages of various types of verification methods, namely RTL-FPGA co-simulation, SystemC-FPGA co-simulation, vector prototype mode, and circuit prototype mode. With the corresponding FPGA operation and configuration in different modes, users can verify the correctness of design modules at different abstract levels at different stages of SoC design. The method is very efficient and provides a solid guarantee for hardware-software co-design and overall verification under SoC heterogeneous computing architecture.

[0028] The University of Electronic Science and Technology of China proposed a software-hardware collaborative simulation platform based on network transmission. The platform mainly consists of two subsystems: a local control master system and a remote embedded slave system. The local control master system is generally a PC or server, while the remote embedded slave system is mainly a development board containing a microprocessor (such as Microblaze, NiosII, etc.). Compared with traditional design methods, this method is more efficient in VLSI design and speeds up verification. It provides a simple and clear data interaction method for remote collaborative development.

[0029] In view of the high cost, inflexible verification and difficult portability of the current image processing algorithm verification platform, Yin Feihao and others built a video image processing algorithm IP verification system on the ZYNQ heterogeneous computing platform, developed the system with the help of software and hardware collaboration, and fully utilized the advantages of the ZYNQ platform. On the processing system side, the embedded real-time operating system Free RTOS was deployed and the lightweight network protocol stack was used to develop TCP / IP network communication, realizing the reception and transmission of image data; on the programmable logic side, the image algorithm verification test and the control of video image output display were carried out. By customizing multiple data transmission IP cores based on the AXI bus protocol and controlling the ARM-side cache, three direct memory access operations were completed, realizing the efficient interaction of big data between the ARM and FPGA sides. Finally, two different white balance algorithm IPs were verified. After testing, the average transmission rate of the system can reach more than 400Mbps, which can meet the requirements of real-time image transmission and processing. In addition, the system hardware logic resource occupancy rate is less than 20% and rich BRAM and DSP resources are retained, which can provide sufficient expansion space for the subsequent verification of complex image algorithm IP.

[0030] 3. Current shortcomings

[0031] As Yin Feihao et al. described, the advantages of ZYNQ heterogeneous architecture can be used to develop a system responsible for image algorithm IP verification. The board used in the present invention integrates a dual-core ARM Cortex-A9 processor and an FPGA architecture. The ARM processor in the software part of the board has an MMU, so that the present invention can install and run an operating system with advanced software algorithms. Thanks to this, the present invention can achieve single-board verification, which not only simplifies the development process, but also avoids additional measures such as exporting data to a PC, thereby improving data security and system reliability.

[0032] At present, the existing hardware algorithm verification platforms on the market generally use FPGA / ZYNQ as the control means, which is mainly implemented by developing cameras. However, these implementation schemes have certain defects. Some image processing algorithms will produce obvious differences in different image data due to different implementation methods. For development based on the camera, it is necessary to select the actual scene so that the camera can collect data and provide it to the algorithm end for use. Secondly, the number of frames and pixels of the image captured by the camera are often limited when the communication protocol is configured, so it is impossible to meet the verification scenarios that require flexible configuration of the number of image frames and pixels. Based on this deficiency, the present invention proposes to store the test image and related configuration in the SD card to enhance the scalability of the verification work of the image processing algorithm IP.

[0033] Zhao Lu et al. pointed out that the verification efficiency of the IP real-time verification framework is determined by the bandwidth of the image processing algorithm IP core, the read and write rate between the memory and the FPGA, and the communication rate between the software and hardware. Taking an image with a resolution of 720P (1280 bits wide and 720 bits high) and a bit width of 8 bits as an example, the processing timeliness of the test loop is analyzed. Figure 5 To test the time consumption of image processing, Figure 5 The processing delay of the verification framework is 14.85ms, and the communication delay between the software part and the verification framework is 20.89ms. Based on the above analysis, the bottleneck of the verification efficiency lies in the Ethernet communication rate between the software part and the verification framework. Therefore, the present invention proposes an interactive mode of heterogeneous computing architecture based on SD card, which uses SD card to perform data interaction between the software part and the hardware part to be tested. Avoid affecting the overall test efficiency due to Ethernet transmission delay.

[0034] In the test systems proposed by Shen Junxing and others, most of them use the system performance optimization, that is, the time required for the same algorithm to run purely in software and to run in coordination with software and hardware, as a measure of the functional accuracy of the test system. Summary of the invention

[0035] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a method of software-hardware co-simulation and IP core verification technology, and proposes a new type of general IP core verification method. On the one hand, the verification speed is improved, and on the other hand, the authenticity of IP core verification is enhanced. The present invention mainly studies and tests memory access optimization and parallelism optimization in middle-level optimization. In the middle-level optimization, memory access optimization can be mainly achieved by inserting instructions such as data splitting and copying. In parallelism optimization, it is mainly achieved by inserting statement-type parallel optimization instructions, and optimizing by executing different instructions at the same time to reduce the start time interval. The present invention measures whether the inserted optimization instructions are reasonable by comparing and analyzing the hardware acceleration situation and the number of resources used more after optimization. In addition, the present invention takes the hardware IP of the FAST corner detection algorithm, a key algorithm for image splicing, as the test target, and further improves the test framework by introducing direct evaluation indicators and derived evaluation indicators.

[0036] To achieve the above object, the present invention provides the following technical solutions:

[0037] A heterogeneous computing architecture testing method for an ISP module, the steps are as follows:

[0038] S1. Use high-level synthesis tools to generate and verify the IP to be tested, and determine whether the optimization instructions added to the IP core can achieve a balance between execution acceleration and resource usage;

[0039] S2. Use the microprocessor IP hard core and AXI bus connector of the heterogeneous computing platform to build the target algorithm IP application system, use the software to operate the target algorithm IP core to carry out application testing, and store the measured data in the non-volatile storage medium according to the set data storage format;

[0040] S3, porting the operating system and OpenCV library for the development board to the heterogeneous computing platform, designing a software test program for the target algorithm using a programming language, and saving the target algorithm software running results to a non-volatile memory after running the test program;

[0041] S4. Design a comprehensive test program for the operating system running environment, and use measured data and software running results to evaluate the target algorithm IP in terms of functional correctness and execution efficiency.

[0042] Furthermore, in step S1, the steps of generating IP are as follows:

[0043] S1.1. Define three interfaces, which are used for data input and output and AXI bus communication control, and set them as AXI-lite interfaces;

[0044] S1.2. Declare three two-dimensional arrays to store the input matrix and the output matrix respectively. Through the nested loop structure, the function traverses the rows and columns of the matrix and performs multiplication and accumulation operations. In each iteration, the local variables are used to calculate the product of and and update the corresponding elements of the output matrix.

[0045] S1.3, read data from the AXI bus and convert it into a C language data type function;

[0046] S1.4. Convert the C language data type to the ap_axiu data type function.

[0047] Furthermore, in step S1, the optimization of IP is performed as follows:

[0048] T1, Array partitioning optimization: improve the parallelism of data access by partitioning multidimensional arrays;

[0049] T2, loop pipeline optimization mainly involves pipeline processing of the iterative relationship between loops;

[0050] T3. The C source code after design optimization is exported as an RTL-level description file; the source code is converted into an RTL-level description and the IP core is exported; the export operation is performed to export the general matrix multiplication IP as a file.

[0051] Furthermore, in step S2, building a hardware circuit includes:

[0052] Zynq 7000 processor connected to AXI_Interconnected module for implementing multiple parallel data transfers;

[0053] An AXI_DMA module for converting register type data into Stream type data or converting Stream data into register type data, connected to the AXI_Interconnected module;

[0054] The IP module to be tested is connected to the AXI_DMA module;

[0055] AXI_timer module for counting, connected with the AXI_Interconnected module.

[0056] Further, the AXI_DMA module performs an initialization operation, the steps are as follows:

[0057] U1, specify the lock of DMA controller;

[0058] U2, DMA controller is initialized;

[0059] U3, test whether the DMA processor is used to handle complex data transfer tasks;

[0060] U4, DMA interrupt disable setting.

[0061] Furthermore, the initialization and configuration of the custom matrix multiplication IP module performs the following steps:

[0062] V1. Call the HLS tool to perform IP synthesis and use the built-in XMmult_accel_core_Initialize function to instantiate a matrix multiplication IP object. By passing in an instantiated xmmult_dev object, the initialized custom IP core is obtained.

[0063] V2. Customize the interrupt control of the matrix multiplication IP module. By configuring the interrupt system, the required interrupt service routine can be correctly called for interrupt response and processing;

[0064] V3. Start the custom matrix multiplication IP module: The custom matrix multiplication IP can be enabled by passing the pointer object of type XMmult_accel_core initialized in step V1 to the XMmult_accel_core_Start function.

[0065] Furthermore, the step V2 performs the following specific steps:

[0066] V2.1, get the relevant configuration information of the interrupt controller through the XScuGic_LookupConfig function;

[0067] V2.2, use XScuGic_CfgInitialize function to initialize the interrupt controller.

[0068] V2.3, execute the self-test function of the interrupt controller to ensure the normal operation of the interrupt controller;

[0069] V2.4, initialize, register, and enable exception handlers;

[0070] V2.5, set the interrupt service function, by specifying the interrupt number

[0071] XPAR_FABRIC_MMULT_ACCEL_CORE_0_INTERRUPT_INTR is connected to the interrupt service routine XMmultIsr to respond to interrupt events;

[0072] V2.6, enable interrupts.

[0073] Furthermore, in step S4, the accuracy of the custom matrix multiplication IP is determined by traversing and comparing the software and hardware execution results one by one.

[0074] Furthermore, in step S3, the Linux operating system based on the ARM architecture is transplanted to the development board, and the steps are as follows:

[0075] W1. Prepare a virtual machine with Linux operating system installed and install a cross compiler for it;

[0076] W2, create the environment variables required by the system, including the source file path, development board information, hardware project information, cross-compilation tool information, and CPU core number;

[0077] W3, modify the hardware description file, bit stream, and update the zynq_fsbl file;

[0078] W4. Configure and compile uboot, and select the makefile compilation method for processing; after compilation, store the u-boot.elf, u-boot.img, and boot.bin files generated by the compilation;

[0079] W5, configure and compile the kernel; modify the device tree file according to the board resources and the hardware circuit diagram built by vivado;

[0080] W6. Compile the kernel using the makefile file. During compilation, you need to specify the starting address of UIMAGE loading. After executing the make statement, store the generated zImage and uImage files.

[0081] W7. Compile the created device tree file; compile the DeviceTree Source file into a Device Tree Blob file by using Device Tree Compiler.

[0082] W8. Install the created kernel file into the root file system directory, decompress the root file system to the mount point, use the make command, specify the ARM architecture and the modules_install instruction to perform the module installation operation.

[0083] Furthermore, in step S3, the OpenCV library is ported to the development board, and the steps are as follows:

[0084] X1. Configuration of the compilation tool chain: specify the system as Linux system, the processor as ARM architecture, the compiler as cross compiler, and the installation directory of OpenCV library functions, etc. and create the corresponding .cmake file;

[0085] X2. OpenCV library configuration and Makefile generation: Execute the .cmake file generated in step X1, configure the installation path of OpenCV, select the projects to be installed, and generate the MakeFile file;

[0086] X3. Compile and install OpenCV source code: Compile the OpenCV source code and install it after the compilation is completed. After the installation is completed, the OpenCV library already exists in the specified folder;

[0087] X4. Deployment of OpenCV runtime library: After storing the OpenCV runtime library compiled in step X3 into the default library function directory of the development board, you can call the OpenCV library function on the development board.

[0088] Beneficial effects of the present invention:

[0089] Compared with the prior art, the heterogeneous computing architecture testing method of the ISP module of the present invention has the following technical features and

[0090] Beneficial effects:

[0091] (1) Improved efficiency and authenticity of IP core verification: Through hardware-software co-simulation and IP core verification technology, the present invention not only accelerates the verification process, but also enhances the authenticity of the verification results through testing in the actual hardware environment, providing a more reliable guarantee for the performance of the IP core in actual applications.

[0092] (2) Optimized middle-level access and parallelism: In the middle-level optimization, the present invention optimizes memory access by inserting instructions such as data splitting and copying, and reduces the start time interval through statement-type parallel optimization instructions, thereby improving the performance of the target algorithm IP. This optimization method not only improves the parallelism of data access, but also reduces the execution time and improves the overall performance.

[0093] (3) Providing a comprehensive testing framework: This paper takes the hardware IP of the FAST corner detection algorithm, a key algorithm for image splicing, as the testing target, and constructs a complete testing framework by introducing direct evaluation indicators and derived evaluation indicators. This not only helps to comprehensively evaluate the functional correctness and performance indicators of the target algorithm IP, but also provides strong support for subsequent optimization and improvement.

[0094] (4) Achieved a balance between hardware acceleration and resource usage: During the IP generation and optimization process, the present invention measures whether the inserted optimization instructions are reasonable by comparing and analyzing the hardware acceleration situation and the number of resources used after optimization. This method ensures hardware acceleration while controlling the growth of resource usage, achieving a balance between performance and resource usage.

[0095] (5) Enhanced system scalability and flexibility: The present invention enhances the scalability and flexibility of the system by using high-level synthesis tools to achieve the generation and verification of the IP to be tested, and by using the microprocessor IP hard core and AXI bus connector of the heterogeneous computing platform to build the target algorithm IP application system. This enables the system to adapt to the needs of different algorithms and applications, and facilitates subsequent algorithm upgrades and system expansion.

[0096] (6) Detailed transplantation and configuration steps are provided: In the process of transplanting the operating system and OpenCV library on the development board, the present invention provides detailed steps and instructions, so that relevant personnel can easily complete the transplantation and configuration work. This not only lowers the technical threshold, but also improves work efficiency and system stability.

[0097] In summary, the present invention realizes effective testing and optimization of the heterogeneous computing architecture of the ISP module through methods such as software-hardware co-simulation and IP core verification technology, middle-layer optimization, and a comprehensive testing framework, providing strong support for the technology development and application promotion in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0099] Figure 1 This is a typical SOPC architecture diagram;

[0100] Figure 2 This is the architecture diagram of software and hardware co-simulation;

[0101] Figure 3 There are two main ways to develop IP cores:

[0102] Figure 4 Design flow chart for vivado HLS IP;

[0103] Figure 5 Processing time-efficiency graphs for validation framework;

[0104] Figure 6 Design flow chart for Xilinx SoPC;

[0105] Figure 7 This is the principle diagram of matrix multiplication;

[0106] Figure 8 This is a diagram showing the relationship between the modules of the matrix_multiply test framework;

[0107] Fig. 9 This is the system hardware circuit diagram;

[0108] Fig.10 This is the test data set diagram;

[0109] Fig.11 Configuration diagram for Image2LCD module software;

[0110] Fig.12 Design a flow chart for FAST corner detection IP;

[0111] FIG13( a ) is a diagram showing data flow of the hardware circuit of Embodiment 2;

[0112] FIG13( b) is a hardware circuit diagram of Example 2;

[0113] Fig.14 Data flow diagram for the test framework;

[0114] Fig.15 Schematic diagram for writing image data into DDR;

[0115] Fig.16 This is the FAST corner point detection IP accuracy test data diagram designed by the present invention. DETAILED DESCRIPTION

[0116] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Figure 1-16 The heterogeneous computing architecture testing method of the ISP module is further explained.

[0117] The core board model of the development board used in the present invention is Xilinx xc7z100ffg900-1, and the baseboard is developed and produced by Milinker. The present invention uses an SD card to export and store the test data of the IP core to be tested, and combines the OpenCV environment configured on the Linux system deployed on the development board and FPGA to perform collaborative simulation and comparative testing, thereby performing system-level verification work on the IP core to be tested.

[0118] 1. First: Use high-level synthesis tools to generate and verify the IP to be tested, and determine whether the optimization instructions added to the IP core can achieve a balance between execution acceleration and resource usage.

[0119] 2. Secondly: Use the microprocessor IP hard core, AXI bus connector and other resources of the heterogeneous computing platform to build the target algorithm IP application system, use software to operate the target algorithm IP core to carry out application testing, and store the measured data in non-volatile storage media (such as SD card, etc.) according to the set data storage format.

[0120] 3. Again: port the operating system and OpenCV library for the development board to the heterogeneous computing platform, design the software test program of the target algorithm using C++ language, and save the target algorithm software running results to a data file after running the test program.

[0121] Finally: Design a comprehensive test program for the operating system running environment, and use measured data and software running results to evaluate the functional correctness and performance indicators of the target algorithm IP.

[0122] The invention will provide two embodiments, the first of which is to complete the basic function and performance verification to verify that the test framework proposed by the invention is feasible, and the second is to perform the overall function verification based on the system-level application of the FAST corner detection algorithm.

[0123] The first is verification based on standard data sets, and the second is test applications for specific IPs.

[0124] Xilinx SoPC design generally requires the use of three tools: ISE (Integrated Software Environment), XPS (Xilinx Platform Studio), and SDK (Software Development Kit). XPS is used to create hardware platforms, SDK is used to debug software applications, and ISE is an optional tool that can generally be used when designing custom IP cores and calling Xilinx's own FIFO, RAM and other IP cores. The specific design process of Xilinx SoPC is as follows: Figure 6 shown.

[0125] In the present invention, the design process is as follows: first, Xilinx's Viviado HLS tool is used to design and implement custom IP. Secondly, the XPS tool is used to create hardware modules, import IP cores, and build hardware circuit diagrams that meet functional requirements. Finally, the SDK tool is used to configure and schedule the modules in the hardware circuit diagram, so that the system can work normally and meet actual needs. Figure 6 The custom IP core described in the present invention adopts Figure 4 Develop and implement according to the development process shown in the figure.

[0126] Example 1

[0127] The basic function verification uses a custom general matrix multiplication IP core. In this embodiment, the accuracy of the test framework of the present invention is tested using the matrix multiplication hardware IP.

[0128] The test objectives are as follows: First, test whether the hardware IP can correctly complete the matrix multiplication operation.

[0129] Secondly, by comparing with the simulation report provided by the Xilinx Vivado HLS tool, the hardware IP execution time calculated by the test framework proposed in the present invention is tested and analyzed to see whether it is correct, thereby verifying whether the test framework proposed in the present invention can accurately complete the test task.

[0130] Then, the running time of the software using the ARM architecture, the execution time of the software and hardware collaborative processing without adding optimization instructions, and the running time after the software and hardware collaborative adding optimization instructions are compared to analyze the impact of adding optimization instructions on the system execution time.

[0131] Finally, with the help of the simulation report of the HLS tool, a comparative analysis is conducted on the number of hardware resources generated by adding more optimization instructions.

[0132] According to the above test objectives, the causal relationship between the system execution speed acceleration brought by the optimization instructions and the hardware resource consumption can be compared and analyzed, and finally it can be concluded whether the selected optimization instructions are reasonable.

[0133] The following describes the implementation method of this embodiment:

[0134] General Matrix Multiplication (GEEM) consumes a lot of computing resources. In addition to the updates and developments in computer architecture and digital circuits, software-level optimization also plays a vital role in accelerating matrix multiplication.

[0135] The IP core includes an input port, INPUT_STREAM, an output port, OUTPUT_STREAM, and an interface for AXI bus control. This embodiment includes the following modules:

[0136] (I) Customized IP design to be tested:

[0137] (1) C source code design, the general form of general matrix multiplication is: C = AB, where C is an M × N matrix, A is an M × K matrix, and B is a K × N matrix. The dark dots in the figure represent the input data required to calculate an output point in C. Figure 7 shown.

[0138] The following is the C language source code implementation method of the general matrix multiplication IP:

[0139] a) Data interface definition: Define three interfaces, which are used for data input and output and AXI bus communication control, and set them as AXI-lite interfaces.

[0140] b) Matrix multiplication function implementation.

[0141] Declare three DIM×DIM two-dimensional arrays to store the input matrices a and b and the output matrix out. Through the nested loop structure, the function traverses the rows and columns of the matrix and performs multiplication and accumulation operations. In each iteration, the local variable sum is used to calculate the product of a[ia][id] and b[id][ib] and update the corresponding elements of the output matrix out[ia][ib].

[0142] c) Read data types from the AXI bus and convert them into C language data types:

[0143] Since the data type read from the AXI-lite bus is ap_axiu, and when programming in C language is used during IP design, the data type required is in C language. Therefore, this data type conversion function is designed. The HLS tool does not fully support all features in C / C++, so the conversion between different pointer types cannot be implemented in the HLS tool. The union allows variables of different types to be stored in the same memory location, because the union can be used to implement the conversion between different types of data without modifying the representation of the data in memory. The union is defined to convert the data field of the ap_axiu object to a user-defined template type.

[0144] (2) General Matrix Multiplication IP Optimization

[0145] The first thing to solve in matrix multiplication is the delay caused by too many memory accesses. The following will optimize the matrix multiplication IP from two aspects. First, add array optimization - array partition. Second, add loop optimization - loop pipeline.

[0146] a) Array partitioning optimization is mainly to improve the parallelism of data access by partitioning multidimensional arrays.

[0147] #pragma HLS array_partition variable=a block factor=FACTOR dim=2

[0148] #pragma HLS array_partition variable=b blockfactor=FACTOR dim=1 The above two commands partition the two input matrices a and b respectively, where dim=2 and dim=1 respectively specify that array a is partitioned on the second dimension, while array b is partitioned on the first dimension. This partitioning method takes into account the dual-port characteristics of block random access memory (BRAM), allowing more efficient data access.

[0149] b) Loop pipeline optimization mainly involves pipeline processing of the iterative relationship between loops.

[0150] #Dragma HLS PIPE LINE II = 1

[0151] Where II = 1 means that one loop iteration is executed per clock cycle. This pipelining technique reduces the execution time of the loop and allows the read, calculate, and write operations within the loop body to be executed in parallel in different iterations, thereby improving the overall computational efficiency. Through these two steps, the hardware implementation of the matrix multiplication function is optimized, allowing the matrix multiplication algorithm to be executed on hardware at a higher speed and lower latency.

[0152] (II) Export of RTL hardware files

[0153] according to Figure 4 In the HLS development process shown in the figure, the C source code after design optimization will be exported as an RTL level description file. Synthesis implementation. By clicking the synthesis button in the tool, the source code is converted into an RTL level description to prepare for hardware implementation.

[0154] IP core export. Execute the export operation to export the general matrix multiplication IP as a file.

[0155] (III) Construction of hardware circuit (XPS development)

[0156] like Figure 6 As shown, after completing the implementation of the custom IP core, it is necessary to create a hardware project to coordinate other IP modules to call the self-designed IP core. The hardware circuit in this embodiment mainly consists of the following parts.

[0157] (1) The core component is the Zynq 7000 processor, which is the development board Zynq 7100 series selected by the present invention.

[0158] (2) AXIDMA module: for input data: convert register type data into Stream type data; for output data: convert Stream type data into register type data.

[0159] (3) Customized IP module to be tested.

[0160] (4) AXI Interconnected module: It is used to connect multiple master devices and slave devices on the AXI bus architecture to realize the transmission of multiple parallel data.

[0161] (5) AXI timer module. A timer used in FPGA design. It implements the timer function based on the AXI bus. It can be used as a timer, a counter, and generates timer interrupts. It is used as a counter in the present invention.

[0162] The connection diagram between the modules is as follows Figure 8 shown.

[0163] (IV) SDK development process

[0164] During the SDK development process, the main responsibility is to schedule the above custom designed optimized exported IP cores, build a complete data path, and realize the required functions.

[0165] In this embodiment, the role of SDK development is: first, to initialize and configure each IP module. Secondly, to create test data and test functions to test the functional normality of the custom matrix multiplication IP core. Then: use the added timing counter IP to calculate the execution time to obtain the final insertion optimization instruction hardware acceleration effect and whether the working condition of the actual hardware circuit is consistent with the working condition of the simulation tool provided by Xilinx. Finally: by using the resource usage report provided by the vivado HLS tool, to compare and analyze from multiple angles whether the added optimization statements are reasonable.

[0166] The present invention is introduced in modules below.

[0167] (1) First, the DMA module is initialized. The main function of this module is to read or write data in a certain address range from the AXI bus, and then convert this data into a data stream and transmit it to the processing unit.

[0168] a) Specify the lock of the DMA controller. First, obtain the configuration information of the DMA controller through the XAxiDma_LookupConfig statement, and find the configuration information of the specified DMA by passing in the unique ID number of the DMA. Define a structure object CfgPtr of the XAxiDma_Config type. And use the returned pointer CfgPtr to initialize the DMA-related configuration. And judge the returned result to ensure that the specified DMA can be correctly captured and initialized and configured later.

[0169] b) Initialization of DMA controller. First, initialize it through XAxiDma_CfgInitialize() statement. Instantiate DMA by passing in an object AxiDma of XAxiDma type. And judge the result after processing. If the return value is not 0, it means that initialization failed.

[0170] c) Test whether the DMA processor is used to handle complex data transfer tasks. The AXIDMA controller is configured in Scatter Gather (SG) mode, which means that the DMA controller is set to handle complex data transfer tasks, which may involve transferring data from multiple non-contiguous memory addresses to the target device, or from the device to multiple non-contiguous memory addresses. This setting can improve the system's data processing capabilities and efficiency and reduce the burden on the CPU. In this function, the AxiDma instantiation object is passed in for judgment. If the returned value is non-zero, it means that it is not in SG mode.

[0171] d) DMA interrupt prohibition setting. This embodiment adopts the polling mode, but the interrupt driven mode. And the driving source can be triggered by the DMA controller to the peripheral device, or it can be triggered by the peripheral device to the DMA. Therefore, two statements need to be written to prohibit the interrupt from the device to the DMA and the interrupt from the DMA to the device respectively.

[0172] (2) The second step is the initialization of the counter module

[0173] a) Use a low-level timer to calculate the function execution time. First, define a function TmrCtrLowLevelExample(), and pass in the timer base address TmrCtrBaseAddress and the timer number TmrCtrNumber as the parameters. The timer number is set because two timers can work at the same time in the same AXI timer IP, so it is necessary to specify which timer is used to implement the counting function. In this embodiment, timer 0 is selected.

[0174] b) Clear the value of the counter status register and load the value of the status register into the Axi timer by calling the XTmrCtr_SetControlStatusReg function, and pass in the base address of the timer, the timer number, and the initial value to be set for the timer. In this embodiment, the initial value is set to 0x0.

[0175] c) Configure the value of the status register to auto-reload XTC_CSR_AUTO_RELOAD_MASK, clear the load bit XTC_CSR_LOAD_MASK, and write the above configuration back to the status register. Here, the role of setting the auto-reload value is to automatically reset the counter value to 0 when the counter reaches the maximum count value.

[0176] d) Start the timer.

[0177] e) Get the timer count value and verify it to eliminate the impact of the time consumption in the read and write timer counter register on the overall test. Call the XTmrCtr_GetTimerCounterReg() function to pass in the timer base address and timer number to get the current counter value. The difference between calling this function twice is the time consumption of reading and writing the timer counter register value.

[0178] (3) Next is the initialization and configuration of the custom matrix multiplication IP module. The main functions of this module are: initializing the custom matrix multiplication IP and configuring the interrupt controller to ensure the normal operation of the IP.

[0179] a) Call the HLS tool to perform IP synthesis and use the built-in XMmult_accel_core_Initialize function to instantiate a matrix multiplication IP object. The specific implementation method is to pass in an instantiated xmmult_dev object to obtain the initialized custom IP core.

[0180] b) Customize the interrupt control of the matrix multiplication IP module and configure the interrupt system so that the required interrupt service routine can be correctly called for interrupt response and processing.

[0181] c) Start the custom matrix multiplication IP module. The custom matrix multiplication IP can be enabled by passing the pointer object of type XMmult_accel_core initialized in 1) to the XMmult_accel_core_Start function.

[0182] (4) Standard data generation module. This module is used to generate standard data. The ARM processor mounted on the board runs the same matrix multiplication algorithm to obtain a set of accurate data. The specific implementation method is: define a function matrix_multiply_ref. The implementation method of this function is consistent with the matrix multiplication function in the hardware and will not be repeated here.

[0183] (5) SD card data reading module. This module is used to read the "resource usage exported from the HLS tool" stored in the SD card and parse it. By combining the resource usage before and after the optimization statement is added, the impact of the optimization statement is dynamically analyzed. In this part, the improvement of the present invention compared to other inventions is that the present invention not only focuses on the effect of hardware acceleration. The present invention performs multi-level analysis and testing of optimization statements from the perspective of resource usage and accuracy. The format of resource usage stored in the SD card is shown in Table 3.

[0184] Table 3 is stored in the format of hardware resource usage

[0185] category BRAM_18K DSP48E FF LUT URAM DSP 0 0 0 0 0 Expression 0 0 0 0 0 FIFO 0 0 0 0 0 Instance 33 80 8381 13734 0 Memory 0 0 0 0 0 Multiplexer 0 0 0 141 0 Register 0 0 3 0 0 Total 33 80 8384 13875 0

[0186] Specific implementation method: The xilffs library introduced in the Board Support Package Setting system configuration provides support for the SD card file system in the bare metal environment.

[0187] a) Introduce file reading and writing related APIs by importing ff files.

[0188] b) A custom data storage structure is used to receive data from the SD card. In this embodiment, a structure is selected to store data. In the structure, the column headers of the table are used as members of the structure.

[0189] c) SD card data reading function. First, use the f_mount() function to mount the SD card. The function returns the result of the mount. Only when the function returns 0, the mount is successful. Use the f_open() function to open the specified file and specify the permission when opening the file. This embodiment only needs to be opened with read-only permission.

[0190] d) Data parsing function. Define a structure array of TableEntry type. Used to store the parsed resource usage data. Use the strtok() function to split the data read from the SD card according to the line break, and store it line by line in a char* pointer named line. Use the sscanf() function to read the specified element from each line and store it in the structure array table. The sscanf() function is used to read the elements of the specified format from the string. So far, the resource usage can be obtained by accessing the values ​​in the structure array.

[0191] (6) Comparative test analysis module. Accuracy test: The accuracy of the custom matrix multiplication IP is determined by traversing and comparing the software and hardware execution results one by one. Performance optimization analysis: First, by comparing the resource usage in the SD card, the causal relationship between the optimization instructions and the hardware resource usage can be obtained. Secondly, the execution time can be calculated by the inserted timer to obtain the causal relationship between the optimization instructions and the resulting system execution time. First, the number of system execution clock cycles is analyzed: Repeating the experiment many times can yield the following:

[0192]

[0193] The heterogeneous architecture verification platform proposed in the present invention is compared with the simulation report generated by the Vivado HLS tool of Xilinx Company, showing a high degree of consistency.

[0194] In the simulation tool statistical results, the number of system execution clock cycles without adding optimization instructions is 47238, while the clock cycles calculated by the test framework of the present invention are 47658, 47640, 47656 and 47642 respectively. The number of system execution clock cycles with adding optimization instructions is 1119, while the clock cycles calculated by the test framework of the present invention are 1726, 1734, 1730 and 1723 respectively. The verification platform statistical results are similar to the simulation tool results.

[0195] These data indicate that the verification system of the present invention can not only accurately calculate the required number of clock cycles, but also its results are basically consistent with the simulation report of the Xilinx official integrated development environment, thereby indirectly verifying the accuracy and reliability of the verification platform proposed in the present invention.

[0196]

[0197] The table above shows the number of clock cycles of pure software execution and the number of clock cycles of whether or not optimized instructions are inserted in the environment of software and hardware collaboration in this test framework. From the table, we can conclude that:

[0198] (1) Compared with pure software execution, the execution speed of the unoptimized hardware circuit is about 0.42 times that of the software. This shows that the performance of the hardware circuit is lower than that of the software without optimization. However, the optimized hardware circuit is 11.6 times that of the pure software. It is a significant improvement.

[0199] (2) In the execution environment of software and hardware collaboration, the execution rate after adding the optimization instruction is about 27.5 times that before adding the optimization instruction. It can be seen that the optimization instruction has a significant performance improvement for this matrix multiplication.

[0200] The following is a comparative analysis of resource usage. It can be concluded from the table below that after adding the optimization instructions, the performance is improved by 27.5 times, and the proportion of the extra hardware resources is negligible compared to the total number of resources. Therefore, it can be tested that the optimization instructions selected in this embodiment are reasonable.

[0201]

[0202] Example 2

[0203] This embodiment still uses Figure 6 Developed according to the development process shown in .

[0204] Since the test data of most existing corner detection algorithm implementation platforms are all from cameras, they need to be placed in real-time scenes for collection, which is inconvenient. Therefore, the present invention proposes a method for storing image data in a non-volatile memory (SD) card. The image data selected in this embodiment comes from an open source dataset on corner detection and matching proposed by Dr. Mohammad Awrangjeb of Gippsland School of Information Technology, Monash University. The dataset contains a total of 23 pictures. Select multiple images that are easy for the human eye to recognize and facilitate the fuzzy set annotation of real corners, such as Fig.10 shown.

[0205] Since the present invention will eventually display the processed result comparison chart on the LCD screen. Therefore, the Image2LCD software is selected to perform a modulus operation on the image and convert the image into a C language array with four RGBA channels. The specific operation method is: set the output type to a C language array, the output grayscale to 32-bit true color, set the maximum image width and height to 512×512, set the output image to 32-bit color, and set the color data arrangement to the ABGR format, that is, arrange the color channel data in the order of transparency (Alpha), blue (Blue), green (Green), and red (Red); its configuration interface is as follows: Fig.11 Shown

[0206] It should be noted that the exported C language array will contain an 8-bit header file, which is used to describe image information and mainly assists the present invention in determining the output of the C language array. It needs to be deleted manually.

[0207] The present embodiment is described in detail below.

[0208] according to Figure 6 The development process shown in the figure first designs a FAST corner detection IP core according to the test needs, and then uses the SDK software to program the ARM part of the development board after building the hardware circuit diagram to realize the data path, calls each hardware module of the system, realizes the corner detection function, and exports the data to be tested to the SD card. Then, the onboard Linux operating system is used to compare and evaluate the corner quality. In addition, the present invention not only uses performance optimization as a measurement indicator, but also tests the corner detection IP quality of the detected corners through direct evaluation indicators and derived evaluation indicators.

[0209] (I) Design of custom IP core to be tested.

[0210] In this module, the present invention designs and implements an IP core for verifying the FAST corner detection algorithm to verify the test framework proposed by the present invention. The source code implementation process of the FAST corner detection algorithm is as follows: a) Read image data from DDR (Double Data Rate memory) and configure the image data in the memory into the Mat type format to prepare for subsequent processing. b) Convert the image data of the Mat type from the RGB color space to the Gray (grayscale) format to reduce the processing complexity and provide a necessary single-channel input for the corner detection algorithm. c) Apply the FAST corner detection algorithm to the image data that has been converted to the Gray type to identify the corner features in the image. d) Export the corner information obtained through the sub-function processing to provide data support for subsequent algorithm performance evaluation.

[0211] The flowchart is as Fig.12 shown. The specific code is as follows:

[0212] a) Define 7 interfaces, which are respectively used for the input of pre-processing image data, the output of post-processing image data, the setting of the width and height of the image to be detected, the setting of the corner detection threshold, the export of the detected corner data, and the transmission of the AXI bus control signal.

[0213] b) Define two image matrices to store the input matrix data and the output matrix data respectively. Among them, <MAX_HEIGHT, MAX_WIDTH> refers to the maximum height and width of the image, and HLS_8UC3 refers to the data type of the image, and each pixel consists of 3 8-bit unsigned integers.

[0214] c) Call the data type conversion function AXIvideo2Mat() provided by HLS to convert the data of the ap_axiu type to the data of the Mat type.

[0215] d) Convert the Mat type data to a grayscale image, where src0 refers to the input original RGB image, and gray refers to the output grayscale image. BGR2GRAY means converting the image from color to GRAY.

[0216] e) Call the FASTX function provided in the official HLS library to implement the corner detection function. The main improvements in this part are: in the FASTX function provided by HLS, there are two functions. One is to draw the keypoints on the original video image in the form of a mask, but the corner information cannot be obtained at the application layer and subsequent judgments cannot be made, so the second function is selected to obtain the data of the keypoints. In the present invention, Point_ <t>Type, directly use int array for storage, the first element of the array records the total number of keypoints, and each subsequent 32-bit data is used to store corner point information, the data format is: the upper 16 bits store the X coordinate, and the lower 16 bits store the Y coordinate. And modify the lower-level function FAST_t_opr() it calls. When a corner point is still a corner point after non-maximum suppression, its y coordinate is shifted 16 bits to the left and stored in a temporary variable, and an OR operation is performed on the X-axis coordinate.

[0217] f) Call Mat2AXIvideo() function to convert MAT type data into ap_axiu type data. So far, the design of custom FAST corner detection IP core is completed.

[0218] (II) Next, the hardware circuit is constructed: The hardware circuit constructed in this embodiment mainly consists of the following modules:

[0219] a) The core component is the Zynq 7000 processor, that is, the development board selected by the present invention is the Zynq 7100 development board.

[0220] b) FAST corner detection algorithm IP core module: This module is the core of the hardware circuit and is responsible for executing the FAST corner detection algorithm. It is responsible for the feature extraction process of image data and maps the algorithm into a hardware executable form to accelerate the calculation process. c) AXI_VDMA (Video DirectMemoryAccess) data conversion module: This module is responsible for high-speed data transmission, allowing image data to be directly transmitted between the memory and the hardware processing unit, that is, converting AXi_stream type data with Memory Map type data without CPU intervention, thereby improving data processing efficiency. And by introducing a frame buffer mechanism, image data display is achieved.

[0221] d) AXI_interconnect module: This module acts as a bridge between PL and PS to achieve efficient data interaction. It exchanges and communicates data between the hardware logic (PL) and the processor subsystem (PS) to achieve complex image processing tasks.

[0222] e) AXI-Stream to video out module. This module is responsible for converting images of AXI-Stream data type into data that can be displayed by HDMI.

[0223] During the data transmission process, as shown in FIG13(a), each module works together through the communication protocol and data channel to ensure high-speed and orderly data transmission. The architecture of the overall system hardware circuit is shown in FIG13(b), which shows the connection relationship between modules and the data flow. Through this modular design, the hardware circuit construction of the present invention not only improves the efficiency of image processing, but also enhances the scalability and flexibility of the entire system through the efficient collaboration of each module, providing a stable and reliable hardware platform for image processing applications.

[0224] (III) Next, proceed to the SDK development process.

[0225] In this part, the main improvements made by the present invention are that the image data comes from the SD card and the final processed corner point data is sent to the SD card. Compared with real-time camera data collection, it has more options and is more stable than using network protocol to collect and store image data.

[0226] In this part: The SDK tool is used to program the ARM (PS) part, which includes the following parts, as shown in the schematic diagram Fig.14 As shown: ① Read the image data from the SD card, ② Write the read image data into the DDR of the PS, ③ Read the image data via the AXI bus, ④ Initialize and configure the VDMA module, and use the VDMAIP to convert the data type, ⑤ Initialize the FAST corner detection IP core, and send the converted image data to the custom FAST ⑥ Corner detection IP to be tested for processing, ⑦ Convert the processed image data to a format via the VDMA, send it back to the PS (ARM) part via the AXI bus, and then write it into the DDR and SD card. ⑧ Initialize and configure another VDMA module, and send it to the video display module via the AXI bus to display the processed image.

[0227] (1) First, read the image data from the SD card and write it into PS_DDR. This part is consistent with the principle of reading the "resource usage exported from the HLS tool" from the SD card in Example 1, and will not be repeated here. The basic logic of writing data into DDR is: obtain the data read from the SD card according to rows and columns. Set up a double-layer loop, traverse the column and row numbers respectively to obtain the values ​​of the three channels b, g, and r, and shift the b, g, and r values ​​left by 8, 16, and 24 bits respectively to form 32-bit unsigned pixel data. The specific implementation method is as follows Fig.14 shown.

[0228] (2) Initialization and configuration of the VDMA module. The configuration of VDMA is mainly achieved by modifying the data in the register offset address. a) Configure the S2MM VDMA control register to the initial values ​​of 0x4 and 0x0, reset the MM2S VDMA control register, and set the synchronization enable. Use the Xil_Out32() function to write the value to the memory address space with the starting address of FAST_VDMA plus the offset address of 0x30.

[0229] b) Configure the initial address of the S2MM frame buffer. Since VDMAIP is set to 3 frame buffers in the block design, the starting address of the frame buffer needs to be set for the memory space with 0xAC as the starting address and every 32 bytes as the offset address. Use the Xil_Out32() function to write the value to the memory address space with the starting address of FAST_VDMA plus the offset address of 0xAC, 0xAC+4, and 0xAC+8.

[0230] c) Configure the S2MM horizontal length register to the initial value of image width*4, and use the Xil_Out32() function to write the value to the memory address space with FAST_VDMA as the starting address plus the offset address of 0xA4.

[0231] d) Configure the S2MM step register to the initial value of image width*4, and use the Xil_Out32() function to write the value to the memory address space with FAST_VDMA as the starting address plus the offset address of 0xA8.

[0232] e) Configure the S2MM VDMA control register to the initial value of 0x3, set the S2MM VDMA channel to start, and execute it in a loop. Use the Xil_Out32() function to write the value to the memory address space with FAST_VDMA as the starting address plus the offset address of 0x30.

[0233] f) Configure the S2MM vertical length register to the image height. Use the Xil_Out32() function to write the value to the memory address space with FAST_VDMA as the starting address plus the offset address of 0xA0.

[0234] The configuration of the MM2S channel is similar to the above steps and will not be described in detail. Set the MM2S VDMA channel to start and execute in a loop; configure the initial address of the MM2S three-frame buffer; configure the MM2S horizontal length register; configure the MM2S step register; and configure the MM2S vertical length register.

[0235] g) Execute the statement to refresh the memory buffer, Xil_DCacheFlush();

[0236] (3) Initialize the custom FAST corner detection module to be tested. In this part, the main tasks are to initialize and configure the custom FAST corner detection IP, start the IP, and export the corner data to be tested to the SD card.

[0237] Call the XHls_FAST_corner_Initialize() function synthesized by the HLS tool, and perform initialization operations by passing in an instantiated XHls_FAST_corner type object FAST and the address XPAR_HLS_FAST_CORNER_0_DEVICE_ID of the custom FAST corner detection IP.

[0238] Parameter setting: By calling XHls_FAST_corner_Set_rows(), XHls_FAST_corner_Set_cols(), XHls_FAST_corner_Set_threhold(), the object is passed in and the FAST_ROW, FAST_COL, and threhold parameters are used to set the image width, height, and threshold for corner detection respectively.

[0239] By calling XHls_FAST_corner_DisableAutoRestart() and passing in the FAST object, the IP core is disabled from restarting automatically. By calling XHls_FAST_corner_InterruptGlobalDisable() and passing in the FAST object, all interrupts are disabled. By calling XHls_FAST_corner_Start() and passing in the FAST object, the IP core is started. By passing in the FAST object, the array of stored corner point information, and the number of corner points to be read, the XHls_FAST_corner_Read_keypoints_Words() function is used to read the detected corner point information.

[0240] d) The detected corner point data is exported to the SD card. This part is the main improvement content of the present invention: the implementation method is to export the keypoint data (that is, the data to be tested) exported by the FAST corner point detection algorithm IP to the SD card. This part needs to parse the keypoint array, obtain the x and y coordinates of the corner points in turn, and write them into the file of the SD card. It belongs to the improvement of the present invention. The parsing process is given below: when creating the IP, the corner point format is set to store the X coordinate in the upper 16 bits and the Y coordinate in the lower 16 bits. Therefore, the coordinates can be split by shifting the keypoint array by 16 bits to the right to obtain the x coordinate, and the keypoint array is ANDed with 0xffff to obtain the lower 16 bits of the y coordinate. The corner point data is exported to the SD card, which is similar to the data reading module. First, mount the SD card and use the f_open function to open the file to be read and written. The difference from data reading is that it needs to be opened with writable permission. Use the f_write function to write data, where &file refers to the file to be written, data refers to the data to be written, strlen(data) refers to the length of the data to be written, bytes_written refers to the actual number of bytes to be written, and res refers to the result of writing. Only when it is 0, it means that the writing is successful. After the processing is completed, use f_close() to close the file.

[0241] At this point, the corner point data to be tested can be exported to the SD card.

[0242] e) The following is to calculate the execution time of the custom IP and export it to the SD card. When building the hardware circuit diagram, the CPU clock frequency was set to 666.666666, and the actual frequency is 666.666687. When configuring the hardware platform, the frequency division coefficient was 6:2:1, so it was divided by 2 6 After frequency division, it is used as the number of executions per second. By introducing the time.h function, use XTime_GetTime(&time_begin); to obtain the current system time as the start time. Use XTime_GetTime(&time_end); to run the FAST corner detection algorithm IP and other related modules to obtain the current system time as the end time. Calculate the time difference to get the execution time of the custom FAST corner detection IP system. After calculating the execution time of the custom FAST corner detection IP system, export it to the SD card according to the process of exporting corner data. At this point, all the data to be tested are exported to the SD card.

[0243] (4) Then transplant the Linux operating system based on ARM architecture to the board. The present invention will transplant the operating system for the selected board, and the Debian system is selected, which is relatively stable and has very small kernel and memory usage. And the operating system is transplanted in the form of device tree. Transplanting the operating system mainly includes steps such as configuration, compiling uboot, kernel, and installing the file system.

[0244] When porting the operating system, you need to tailor the kernel according to the onboard resources, modify the device tree, and so on. The following steps port the operating system:

[0245] a) First, you need to prepare a virtual machine with Linux operating system installed and install a cross compiler for it. The reason is: the board is ARM architecture, while the virtual machine is X86_64 architecture. So cross compilation is required.

[0246] b) Create the environment variables required by the system, including some source file paths, development board information, hardware engineering information, cross-compilation tool information, CPU core count, etc. This part is created based on the board resources and the actual needs of this embodiment. Modify the hardware description file, bit stream, and update the zynq_fsbl file.

[0247] c) Configure and compile uboot. This part is used to boot the operating system on the development board. Download the official uboot file and compile it. During compilation, because there are too many files involved, the present invention selects the makefile compilation method. After compilation, the u-boot.elf, u-boot.img, and boot.bin files generated by the compilation are stored.

[0248] d) Configure and compile the kernel. In this part, the device tree file needs to be modified according to the board resources and the hardware circuit diagram built by vivado. The modification of the device tree mainly includes the following steps: First: the dual-core CPU needs to be set as a child node. Next, according to the address of the vdma set in the hardware circuit diagram, the device node is created. The settings in this part are mainly used for operations such as image display. This part is for self-modification of the hardware circuit built by the present invention.

[0249] Add an AXI bus node, which can contain multiple sub-nodes, such as adc, can, gpio, i2c, etc.

[0250] Add the node of intc interrupt controller, add the node of L2 cache controller, add the node of mc memory controller,

[0251] Add uart asynchronous receiver and transmitter node, add spi serial peripheral interface and other general configuration nodes.

[0252] This completes the modification of the device tree file.

[0253] The kernel is compiled using the makefile file. During compilation, the starting address of UIMAGE loading needs to be specified. In this embodiment, the present invention sets it to 0x8000. After executing the make statement compilation, the generated zImage and uImage files need to be stored.

[0254] Compile the created device tree file. Use Device Tree Compiler (DTC) to compile the DeviceTree Source (DTS) file into a Device Tree Blob (DTB) file.

[0255] e) Install the created kernel file to the root file system directory. Unzip the root file system to the mount point. Use the make command, specify the ARM architecture and the modules_install instruction to install the module. The code is as follows, MAKE_JOBS refers to the number of threads used, and ZN_ROOTFS_MOUNT_POINT refers to the mount point. This completes the compilation and installation of the kernel.

[0256] f) To install the root file system, you need to configure the root file system.

[0257] Specify the system architecture to be ARM, use the debian9 version, and download the system image file. Check whether the root file system already exists. If it does, use the existing system. If it does not exist, create the root file system. Execute the make parted command to format the SD card partition, and execute the deploy_image.sh command to burn the root system to the SD card. This part is no different from the existing transplantation method and will not be described in detail. The installation of the root file system is now complete. Insert the SD card into the development board and set the development board startup mode to SD card startup. You can view the startup information through the serial port tool.

[0258] g) After booting, run the commands to format the partition and load the system, and then set the operating system to boot from qspi and load the file system from emmc. This completes the operating system transplantation.

[0259] (5) Then transplant the OpenCV library

[0260] a) Configuration of the compilation tool chain. Specify the system as Linux, the processor as ARM architecture, the compiler as a cross compiler, and the installation directory of the OpenCV library function, and create the corresponding .cmake file.

[0261] b) OpenCV library configuration and Makefile generation: Execute the .cmake file generated in step S4 to configure the installation path of OpenCV, select the projects to be installed, and generate the Makefile file.

[0262] c) Compile and install OpenCV source code. Compile the OpenCV source code and install it after the compilation is complete. After the installation is complete, the OpenCV library will exist in the specified folder.

[0263] d) Deployment of OpenCV runtime library: After the OpenCV runtime library compiled in step S4 is stored in the development board directory, the OpenCV library functions can be called on the development board.

[0264] Through the above steps, the present invention realizes the transplantation of the OpenCV library on the Zynq development platform, providing powerful library support for the development of image processing and computer vision applications.

[0265] (6) Finally, the FAST corner detection algorithm based on pure software is implemented, and the test results generated by the custom test IP core in the SD card are read to perform comparative analysis of the corner data.

[0266] This part is the main improvement of the present invention. The main work is: based on the Linux operating system loaded on the board and the Opencv library environment, the FAST corner point detection algorithm of the pure software and standard openCV library is implemented. The data generated by it is used as the standard data. It is compared with the results produced by the self-designed IP, the hardware circuit construction, and the SDK programming for system scheduling. The corner point data generated by the self-designed IP has been exported to the SD card before. Therefore, the present invention can obtain it by directly reading the file in the SD card.

[0267] Furthermore, the present invention not only focuses on the execution time of the system during testing, but also analyzes from multiple angles such as corner point quality. The following are the evaluation indicators selected by the present invention.

[0268] The evaluation will be conducted from two aspects: direct evaluation indicators and derived evaluation indicators.

[0269] First, the direct evaluation indicators:

[0270] The number of standard corner points Nd generated by pure software

[0271] Customize the number of corner points Nc generated by the IP to be tested

[0272] The number of correct corner points Nr

[0273] Number of wrongly detected corner points Nw

[0274] Number of missed corner points Nm

[0275] At the same time: the actual detected corner point set extracted by the self-designed IP is recorded as A, and the real corner point set generated by pure software is recorded as B. Set A and B are matched and compared. If the Euclidean distance between the actual detected corner point and the real corner point in the image space is <3 pixels, it means that the match is correct. If there is no real corner point within the 3-pixel area near the actual detected corner point, it is regarded as a false detection corner point. In addition, if there is no actual detected corner point within the 3-pixel area near the real corner point, it is a missed detection corner point.

[0276] The second is the derived evaluation indicators:

[0277] Corner point repetition rate: Corner consistency: Average repetition rate:

[0278] Precision and recall: The recall rate mainly measures the ratio of the number of correctly detected corner points to the total number of true corner points. Under the premise that the total number of true corner points is determined, this indicator increases as the number of correctly detected corner points increases, and vice versa. However, the defect of this indicator is that as long as the decision threshold of the corner point response value is lowered and the number of detected corner points is large enough, the recall rate of the algorithm can be infinitely close to 1.

[0279]

[0280] The accuracy mainly measures the ratio of the number of correctly detected corner points to the total number of detected corner points. This index increases with the increase of the number of correct corner points and the decrease of the number of falsely detected corner points, and decreases on the contrary.

[0281]

[0282] Let's start designing this module.

[0283] a) First, mount the SD card. In this section, the logic of programming the SD card mounting in the SDK tool is consistent with that in the previous section, so I will not repeat it here.

[0284] b) The second step is to implement the pure software FAST corner detection algorithm to generate standard data and calculate the time consumption of the pure software-based corner detection algorithm.

[0285] The implementation of the algorithm involves defining a data container to store the detected corner point information. The specific implementation steps are as follows: First, by calling the detect method of the FAST algorithm object, the input image image is detected and the results are stored in the keypoints container. Then, the drawKeypoints() function in the OpenCV library is used to draw the detected corner point information back to the original image for easy visualization. In order to save the processed image, the imwrite() function is called to write the image to the local file system. For the calculation of execution time, the sys / time.h file is introduced and the gettimeofday() function is used to record the time points before and after the algorithm is executed, and the difference between the two time points is calculated to obtain the actual execution time of the FAST corner detection algorithm.

[0286] Next is: read the corner point data and execution time generated by the self-designed hardware IP in the SD card. And evaluate and analyze the data generated by the current self-defined IP core according to the direct evaluation index and derived evaluation index. And save the information of correctly matched corner points, false detection and missed detection corner points.

[0287] It mainly involves: data reading, calculating the Euclidean distance between two corner points in sequence, determining the type of corner points, and evaluating the corner point detection system according to indicators.

[0288] The specific implementation method is:

[0289] Read the test data generated by the self-created IP and the standard data generated by the ARM software from the SD card. This section will not be repeated here.

[0290] Define three integer variables to store the number of correct corner points, the number of incorrectly detected corner points, and the number of missed corner points respectively.

[0291] Define a vector container of point type to store the three processed corner point data.

[0292] Define a Boolean container to store whether the corner point is matched, set its size to the number of real corner points read, and set its initial value to false.

[0293] Next, we determine the corner point data. We define a function calculateDetectionMetrics(), and the parameters passed in are the corner point data to be tested, the standard real corner point data, the number of correct corner points, the number of falsely detected corner points, the number of missed corner points, the container for storing correct corner points, the container for storing falsely detected corner points, and the container for storing missed corner points.

[0294] Traverse the corner points to be tested one by one, and determine whether the Euclidean distance between them and the corner points generated by the software is less than 3. If it is less than 3, mark it as a correct corner point, and set the value of truePointsMatched of the real corner point to true. And store the corner point information in the corresponding container.

[0295] If no match is found, the number of incorrectly detected corner points is increased by 1, and the corner point information is stored in the corresponding container.

[0296] Calculation of missed corner points: One by one, the number of unmatched corner points in the real corner points is passed through, that is, the truePointsMatched value of the corner point is false, and the missed corner points can be obtained. And the number of missed corner points is added by 1, and the corner point information is stored in the corresponding container. Next, calculate other derived indicators. For this part of the calculation, just substitute the values ​​of the formula mentioned above. I will not repeat it here.

[0297] After analyzing the corner point data according to the above indicators:

[0298]

[0299]

[0300] The data in the following table can be obtained:

[0301] After calculating and analyzing the above corner point data according to the derived evaluation indicators, the corner point analysis data shown in the figure below can be obtained: From the above data, it can be analyzed that the self-designed FAST corner point detection IP is more sensitive when the color difference between the background color of the image and the area to be detected is large and the lines of the corner point area to be detected are easier to distinguish.

[0302] In summary: the test framework proposed in the present invention not only tests and evaluates the corner point detection system at the execution time level, but also evaluates the quality of the detected corner points from multiple angles to determine the detection scenarios to which the algorithm is applicable.

[0303] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.< / t>

Claims

1. A heterogeneous computing architecture testing method for an ISP module, characterized in that: Here are the steps: S1. Use high-level synthesis tools to generate and verify the IP to be tested, and determine whether the optimization instructions added to the IP core can achieve a balance between execution acceleration and resource usage; S2. Use the microprocessor IP hard core and AXI bus connector of the heterogeneous computing platform to build the target algorithm IP application system, use the software to operate the target algorithm IP core to carry out application testing, and store the measured data in the non-volatile storage medium according to the set data storage format; S3, porting the operating system and OpenCV library for the development board to the heterogeneous computing platform, designing a software test program for the target algorithm using a programming language, and saving the target algorithm software running results to a data file after running the test program; S4. Design a comprehensive test program for the operating system environment, and use measured data and software running results to evaluate the functional correctness and performance indicators of the target algorithm IP.

2. The heterogeneous computing architecture testing method of the ISP module according to claim 1, characterized in that: In step S1, the steps of generating IP are as follows: S1.

1. Define three interfaces, which are used for data input and output and AXI bus communication control, and set them as AXI-lite interfaces; S1.

2. Declare three two-dimensional arrays to store the input matrix and the output matrix respectively. Through the nested loop structure, the function traverses the rows and columns of the matrix to perform multiplication and accumulation operations. In each iteration, the local variables are used to compute the product of and and update the corresponding elements of the output matrix; S1.3, read data from the AXI bus and convert it into a C language data type function; S1.

4. Convert the C language data type to the ap_axiu data type function.

3. The heterogeneous computing architecture testing method of the ISP module according to claim 1, characterized in that: In step S1, the optimization of IP is performed as follows: T1, Array partitioning optimization: improve the parallelism of data access by partitioning multidimensional arrays; T2, loop pipeline optimization mainly involves pipeline processing of the iterative relationship between loops; T3. The C source code after design optimization is exported as RTL level description file; Convert source code into RTL level description and export IP core; Perform the export operation to export the general matrix multiplication IP to a file.

4. The heterogeneous computing architecture testing method of the ISP module according to claim 1, characterized in that: In step S2, building a hardware circuit includes: Zynq 7000 processor connected to AXI_Interconnected module for implementing multiple parallel data transfers; An AXI_DMA module for converting register type data into Stream type data or converting Stream data into register type data, connected to the AXI_Interconnected module; The IP module to be tested is connected to the AXI_DMA module; AXI_timer module for counting, connected with the AXI_Interconnected module.

5. The heterogeneous computing architecture testing method of the ISP module according to claim 4, characterized in that: The AXI_DMA module performs initialization operations as follows: U1, specify the lock of DMA controller; U2, DMA controller is initialized; U3, test whether the DMA processor is used to handle complex data transfer tasks; U4, DMA interrupt disable setting.

6. The heterogeneous computing architecture testing method of the ISP module according to claim 1, characterized in that: The initialization and configuration of the custom matrix multiplication IP module performs the following steps: V1. Call the HLS tool to perform IP synthesis and use the built-in XMmult_accel_core_Initialize function to instantiate a matrix multiplication IP object. By passing in an instantiated xmmult_dev object, the initialized custom IP core is obtained. V2. Customize the interrupt control of the matrix multiplication IP module. By configuring the interrupt system, the required interrupt service routine can be correctly called for interrupt response and processing; V3. Start the custom matrix multiplication IP module: The custom matrix multiplication IP can be enabled by passing the pointer object of type XMmult_accel_core initialized in step V1 to the XMmult_accel_core_Start function.

7. The heterogeneous computing architecture testing method of the ISP module according to claim 6, characterized in that: The step V2 performs the following specific steps: V2.1, get the relevant configuration information of the interrupt controller through the XScuGic_LookupConfig function; V2.2, use XScuGic_CfgInitialize function to initialize the interrupt controller. V2.3, execute the self-test function of the interrupt controller to ensure the normal operation of the interrupt controller; V2.4, initialize, register, and enable exception handlers; V2.5, set the interrupt service function, by specifying the interrupt number XPAR_FABRIC_MMULT_ACCEL_CORE_0_INTERRUPT_INTR is connected to the interrupt service routine XMmultIsr to respond to interrupt events; V2.6, enable interrupts.

8. The heterogeneous computing architecture testing method of the ISP module according to claim 6, characterized in that: In step S4, the accuracy of the custom matrix multiplication IP is determined by traversing and comparing the software and hardware execution results one by one.

9. The heterogeneous computing architecture testing method of the ISP module according to claim 1, characterized in that: In step S3, the Linux operating system based on the ARM architecture is transplanted to the development board. The steps are as follows: W1. Prepare a virtual machine with Linux operating system installed and install a cross compiler for it; W2, create the environment variables required by the system, including the source file path, development board information, hardware project information, cross-compilation tool information, and CPU core number; W3, modify the hardware description file, bit stream, and update the zynq_fsbl file; W4. Configure and compile uboot, and select the makefile compilation method for processing; After compiling, store the generated u-boot.elf, u-boot.img, and boot.bin files; W5, configure and compile the kernel; modify the device tree file according to the board resources and the hardware circuit diagram built by vivado; W6. Compile the kernel using the makefile file. During compilation, you need to specify the starting address of UIMAGE loading. After executing the make statement, store the generated zImage and uImage files. W7. Compile the created device tree file; compile the Device TreeSource file into a Device Tree Blob file by using Device Tree Compiler. W8. Install the created kernel file into the root file system directory, decompress the root file system to the mount point, use the make command, specify the ARM architecture and the modules_install instruction to perform the module installation operation.

10. The heterogeneous computing architecture testing method of the ISP module according to claim 1, characterized in that: In step S3, the OpenCV library is ported to the development board. The steps are as follows: X1. Configuration of the compilation tool chain: specify the system as Linux, the processor as ARM architecture, the compiler as a cross compiler, and the installation directory of the OpenCV library function, etc. and create the corresponding .cmake file; X2. OpenCV library configuration and Makefile generation: Execute the .cmake file generated in step X1, configure the installation path of OpenCV, select the projects to be installed, and generate the MakeFile file; X3. Compile and install OpenCV source code: Compile the OpenCV source code and install it after the compilation is completed. After the installation is completed, the OpenCV library already exists in the specified folder; X4. Deployment of OpenCV runtime library: After storing the OpenCV runtime library compiled in step X3 into the development board directory, you can call the OpenCV library functions on the development board.

Images