Interface adaptive image processing system and method based on FPGA

Abstract

The invention relates to an interface adaptive image processing system and method based on an FPGA (Field Programmable Gate Array). The system mainly comprises a host and an image processing module, host software is arranged on the host, the image processing module comprises two identical image processing cards, one image processing card is a main image processing card, and the other image processing card is a standby image processing card. And the state detection module regularly notifies the host software to detect the state of the image processing card, and when the main image processing card fails, the host software switches the main image processing card to the standby image processing card. The main image processing card and the standby image processing card are provided with the state detection module which is used for detecting the state of hardware and allowing host software to recognize and switch, once the main image processing card breaks down, the standby image processing card can be switched to work, and the reliability of image processing is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an interface adaptive image processing system and method based on FPGA. Background Art

[0002] With the development of computer technology, image processing technology is increasingly widely used in many fields. Image processing technology can be applied to artificial intelligence, Internet of Things, medicine, security, industry and other fields. Since image processing has high requirements on computer performance, traditional computer architecture often cannot meet the processing requirements. Field-Programmed Gate Array (FPGA) is a programmable logic device that can conveniently realize the functions of logic circuits by opening the high programmability of logic gates and timing resources and combining hardware programming languages ​​(such as Verilog). With the continuous development of FPGA technology, especially Xilinx's FPGA, in addition to conventional logic arrays, it also integrates high-performance processing technologies such as digital signal processing (DSP) and random access memory (RAM), and provides a relatively powerful IP core. With the characteristics of high performance, fast running speed, strong portability and high cost performance, it provides strong support for the realization of efficient image processing.

[0003] The application of existing image processing technology mainly uses dedicated processors or FPGA as the core to develop hardware boards for transmission and processing. Once the board hardware fails, the system will be unusable and the reliability is not high. In addition, in terms of interface form, a single interface is generally used, such as High Definition Multimedia Interface (HDMI), Display Port (DP), Digital Visual Interface (DVI) to complete image input, which is not very versatile.

[0004] In view of this, how to overcome the defects of the existing technology and how to solve at least one of the problems of low reliability and low versatility of the existing image processing technology have become important technical problems that the industry needs to solve urgently. Summary of the invention

[0005] The present invention provides an interface adaptive image processing system and method based on FPGA, by setting a main and standby image processing card and host software, wherein the main and standby image processing card has a state detection module for detecting the hardware state for the host software to identify and switch, once the main image processing card fails, it can switch to the standby image processing card to work, thereby improving the reliability of image processing. In addition, the image processing card can complete the image signal acquisition of different interfaces, realize the adaptation of different interfaces, perform video data normalization processing, provide a normalized video signal interface, and improve the versatility of image processing.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides an interface adaptive image processing system based on FPGA, comprising a host and an image processing module, wherein the host is provided with host software, and the image processing module comprises two identical image processing cards, one of which is a main image processing card and the other is a standby image processing card;

[0008] The image processing card includes an FPGA chip, a memory and several image interface circuits. The FPGA chip includes an image acquisition controller, an image processor, an image controller, a PCIE controller and a state detection module. The image acquisition controller acquires data through the image interface circuit and transmits it to the memory. The image processor reads and processes data from the memory through the image controller. The processed data is transmitted to the memory through the image controller and transmitted to the host for display through the PCIE controller.

[0009] The status detection module periodically notifies the host software to detect the status of the image processing card. When the main image processing card fails, the host software switches the main image processing card to the standby image processing card.

[0010] In some embodiments, the state detection module generates an interrupt signal at a fixed time. After the host software detects the interrupt signal, it performs a read and write test on the memory to determine whether it is normal. If it is normal, the interrupt state flag is kept at 0 and the interrupt service program is exited; if it is abnormal, the interrupt state flag is set to 1;

[0011] The host software periodically detects the interrupt status mark of the main image processing card through the status detection and identification task. Each timed interrupt releases a binary semaphore. The status detection and identification task waits for the received semaphore in a loop, and immediately determines the status of the main image processing card after obtaining the semaphore. When the interrupt status mark is 0, the main image processing card is judged to be normal, and the loop continues to wait for the next semaphore. When the interrupt status mark is 1, the main image processing card is judged to be faulty, and the standby image processing card takes over the main image processing card to start working.

[0012] In some embodiments, the image processor includes a user-defined image processing function module, a timing transformation module, a pre-processing data synchronization and conversion module, and a post-processing data synchronization and conversion module; wherein the user-defined image processing function module is used to configure and design user-defined image processing functions in real time according to different application requirements to enhance image quality; the timing transformation module is used to convert image data into video timing; the pre-processing data synchronization and conversion module is used to synchronize the video timing signal with the AXI bus, output a normalized video signal, and provide a unified interface for processing by the user-defined image processing function module; the post-processing data synchronization and conversion module is used to convert the data signal after processing by the user-defined image processing function module into AXI bus data, and transmit it to the memory via the image controller.

[0013] In some embodiments, the image controller includes a register and a data transmission module, and the image controller receives a host's initialization request for the register through an AXI4-Lite bus; when performing a write operation, the data transmission module writes the video stream data into the memory in a DMA manner according to the register information; when performing a read operation, the data transmission module reads the data in the memory in a DMA manner according to the register information;

[0014] In some embodiments, the image controller includes a first image controller and a second image controller, the first image controller is responsible for reading data, and the second image controller is responsible for writing data.

[0015] In some embodiments, the image acquisition controller includes an AXI bus DMA controller, an adaptive logic circuit and a selection logic circuit; the adaptive logic circuit is used to detect the signal of the image interface circuit and determine which interface the image data is input from; the selection logic circuit selects one channel from multiple channels of image data and sends it to the AXI bus DMA controller to write it to the memory in DMA mode.

[0016] In some embodiments, the image interface circuit corresponds to a DP interface, an HDMI interface and a VGA interface. When multiple image interface circuits are connected at the same time, the interface selection priority is DP interface>HDMI interface>VGA interface.

[0017] In some embodiments, the PCIE controller uses AXI-Lite configuration to complete the mapping of the PCIE controller interface address to the AXI-Lite address, so that the host side can access the registers on the user logic side or other AXI4-Lite bus devices through the PCIE controller.

[0018] In some embodiments, the FPGA chip also includes a bus controller, which uses an AXI4 bus IP core interconnection bus. The image controller, the PCIE controller and the image acquisition controller are interconnected through the bus controller to achieve access to the memory.

[0019] In a second aspect, the present invention further provides an interface adaptive image processing method based on FPGA, which is applied to the interface adaptive image processing system based on FPGA as described in the first aspect, comprising:

[0020] The image acquisition controller acquires data through the image interface circuit and transmits it to the memory;

[0021] The image processor reads and processes data from the memory through the image controller, and the processed data is transmitted to the memory through the image controller and transmitted to the host display through the PCIE controller;

[0022] The status detection module periodically notifies the host software to detect the status of the image processing card. When the main image processing card fails, the host software switches the main image processing card to the standby image processing card.

[0023] Compared with the prior art, the present invention provides an interface adaptive image processing system and method based on FPGA, and its beneficial effect is that: by setting the main and standby image processing cards and the host software, wherein the main and standby image processing cards have their own status detection modules for detecting the hardware status for the host software to identify and switch, once the main image processing card fails, it can switch to the standby image processing card to work, thereby improving the reliability of image processing. In addition, the image processing card can complete the image signal acquisition of different interfaces, realize the adaptation of different interfaces, perform video data normalization processing, provide a normalized video signal interface, and improve the versatility of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required to be used in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A principle block diagram of an interface adaptive image processing system based on FPGA provided by an embodiment of the present invention;

[0026] Figure 2 A principle block diagram of an image processing card provided by an embodiment of the present invention;

[0027] Figure 3A functional block diagram of an image controller provided by an embodiment of the present invention;

[0028] Figure 4 A functional block diagram of an image processor provided by an embodiment of the present invention;

[0029] Figure 5 A block diagram of a PCIE controller interface provided by an embodiment of the present invention;

[0030] Figure 6 A block diagram of a memory controller interface provided by an embodiment of the present invention;

[0031] Figure 7 A block diagram of a bus controller interface provided by an embodiment of the present invention;

[0032] Figure 8 A block diagram of a command controller interface provided by an embodiment of the present invention;

[0033] Fig. 9 A principle block diagram of a state detection module provided by an embodiment of the present invention;

[0034] Fig.10 An interface block diagram of an image acquisition controller provided by an embodiment of the present invention;

[0035] Fig.11 The present invention provides a flowchart of an interface adaptive image processing method based on FPGA. DETAILED DESCRIPTION

[0036] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present invention, several variations and improvements can also be made. These all belong to the protection scope of the present invention. It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the protection scope of this application. In addition, although the functional module division may be performed in the device schematic diagram, and the logical order may be shown in the flow chart, in some cases, it may be different from the module division in the device, or the steps shown or described in the order in the flow chart may be performed.

[0037] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification and in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0039] In order to make the purpose, technical scheme and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0040] Embodiment 1:

[0041] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides an interface adaptive image processing system based on FPGA, including a host and an image processing module, wherein the host is provided with host software, and the image processing module includes two identical image processing cards, one of which is a main image processing card and the other is a standby image processing card; the image processing card includes an FPGA chip, a memory and several image interface circuits, and the FPGA chip includes an image acquisition controller, an image processor, an image controller, a PCIE controller and a status detection module; the image acquisition controller acquires data through the image interface circuit and transmits it to the memory; the image processor reads and processes data from the memory through the image controller, and the processed data is transmitted to the memory through the image controller, and is transmitted to the host for display through the PCIE controller; the status detection module regularly notifies the host software to detect the status of the image processing card, and when the main image processing card fails, the host software switches the main image processing card to the standby image processing card. Through the above method, high-performance processing and high-reliability transmission of image acquisition can be achieved.

[0042] In some embodiments, the status detection module generates an interrupt signal at regular intervals. After the host software detects the interrupt signal, it performs a read and write test on the memory to determine whether it is normal. If it is normal, the interrupt status flag is kept at 0 and the interrupt service program is exited; if it is abnormal, the interrupt status flag is set to 1; the host software regularly detects the interrupt status flag of the main image processing card through the status detection and identification task, and each timed interrupt releases a binary semaphore. The status detection and identification task waits for the received semaphore in a loop, and immediately determines the status of the main image processing card after obtaining the semaphore; when the interrupt status flag is 0, it is determined that the main image processing card is normal, and the loop continues to wait for the next semaphore; when the interrupt status flag is 1, it is determined that the main image processing card is faulty, and the standby image processing card takes over the main image processing card to start working.

[0043] Based on the above settings, the main and standby image processing cards are responsible for image acquisition, transmission, and processing; specifically, the image processing card completes image signal acquisition of different interfaces and completes the adaptation of different interfaces; the acquired data is transmitted and processed by the hardware of the image processing card, and the processed data is transmitted to the host for display by the host. At the same time, the image processing card has its own status detection module, which generates status detection information for host software to identify and switch. Once the main image processing card fails, it can switch to the standby image processing card to improve the reliability of the application.

[0044] Embodiment 2:

[0045] Based on Example 1, this Example 2 provides an expanded description of the provided FPGA-based interface adaptive image processing system.

[0046] refer to Figure 1 As shown, the two image processing cards of the image processing module have the same hardware and both communicate with the host through PCIE (peripheral component interconnect express, a high-speed serial computer expansion bus standard); the host software includes application software and underlying driver software, and its main functions are: 1. Controlling the initialization of registers such as the image processor, image controller, image acquisition controller, and user-defined image processing function module; 2. Sending the local image data of the host to the memory and the host reading the data processed by the image processor from the memory; 3. Image processing card underlying driver; 4. Status monitoring and confirmation. Once a failure of the main image processing card is detected, it will quickly switch to the backup image processing card to ensure the high reliability of image processing.

[0047] refer to Figure 2As shown, the image processing card is mainly composed of a programmable logic unit FPGA, a memory and an image interface circuit. The memory mainly completes the storage of raw data and processed data, and the image interface circuit mainly completes the conversion and buffering of images. The main functions of the image processing card are completed by the FPGA. Various functional modules are integrated inside the FPGA, including: PCIE controller, memory controller, bus controller, command controller, image processor, state detection module, image acquisition controller, and the image processor can also be divided into a first image processor and a second image processor. The main working principle is as follows: the host first operates by commanding the registers of the image processing card and other modules to initialize the relevant parameters to meet the different application requirements of image parameters and image processing modes. According to the initialization method, the image processing card collects data through the image acquisition controller and transfers it to the memory in a direct memory access (DMA) mode. The image processor reads the image data from the memory in a DMA mode through the first image controller. After being processed by the image processor, the second image controller transfers it to the memory in a DMA mode. The processed data is transmitted to the host through the PCIE controller for display. The state detection module notifies the host to obtain the state of the image processing card through the host timing query or timing interrupt mode.

[0048] When the image processing card is executed, it mainly includes the following functions: 1. The image processing card receives the initialization of the host, including the initialization of the image timing register, image parameter register, image controller register, user-defined parameters, etc. 2. The image processing card collects the original image data through the image acquisition controller, writes the original data into the memory through the bus controller in DMA transmission mode, and the memory controller provides the memory read and write timing. 3. The image controller reads the original data in the memory through the bus controller in DMA transmission mode, normalizes it through video timing conversion, and converts it into a standard video data format, and sends it to the image processor for image processing. 4. After the image processor processes the image saturation, the image controller converts the processed data into a video stream format, and writes the processed data into the memory in DMA transmission mode. 5. The PCIE controller transmits the processed image data in the memory to the host through DMA transmission mode for the host to process and display. 6. The status detection module completes the status detection of the image processing card, notifies the host of the status of the image processing card through query or interrupt mode, and allows the host to complete the switching of the image processing card, ensuring the high reliability of image transmission and processing.

[0049] refer to Figure 3As shown, in some embodiments, the image controller includes registers and a data transmission module. The image controller receives a host initialization request for registers through the AXI4-Lite bus, mainly including the initialization of control registers, status registers, index registers, pointer registers, vertical display, horizontal display size, cache start address registers, etc. The specific working mode of the image controller is controlled by the relevant registers, and the working status is obtained by reading the status register. When performing a DMA transfer write operation, according to the control register information, the data transmission module writes the video stream data into the memory in DMA mode; when performing a DMA transfer read operation, according to the control register information, the data transmission module reads the data in the memory in DMA mode.

[0050] In order to improve data transmission efficiency, two image controllers can also be used, that is, in some embodiments, the image controller includes a first image controller and a second image controller, the first image controller is responsible for reading data, and the second image controller is responsible for writing data; at this time, the image controller uses independent read and write channels to convert image data into data streams, so that the image stream can be directly connected to the memory with high bandwidth; the data stream is transmitted using DMA transmission.

[0051] refer to Figure 4 As shown, in some embodiments, the image processor includes a user-defined image processing function module, a timing conversion module, a pre-processing data synchronization and conversion module, and a post-processing data synchronization and conversion module to realize video timing normalization processing and user-defined image processing functions. Among them, the timing conversion module is used to convert image data into video timing for display; it mainly converts information such as image resolution and pixels into standard video data, lines, fields, blanking and enable signals. The pre-processing data synchronization and conversion module mainly uses a first-in first-out queue (First Input First Output, abbreviated as: FIFO) to complete the synchronization timing function, which is used to synchronize the video timing signals, including clock, line, field, line blanking, field blanking and AXI bus, output the normalized video signal, and provide a unified interface to facilitate the processing of the user-defined image processing function module. The post-processing data synchronization and conversion module is used to synchronize the video data, line, field, blanking and enable signals after the user-defined image processing function module has finished processing, using FIFO to complete the synchronization, convert them into AXI bus data, and transmit them to the memory through the image controller.

[0052] The user-defined image processing function module is used to configure and design user-defined image processing functions in real time according to different application requirements to enhance image quality. For example, taking the adjustment of image saturation as an example, a function for adjusting the saturation of a color image is designed, and the saturation of the input image is adjusted according to the register configuration, and then output. Using an eight-bit register, the host sets the image saturation value, with the register median as the base. When the register value becomes smaller, the saturation decreases; when the register value becomes larger, the saturation increases. Under the premise of ensuring that the color is not distorted, the color saturation of the image is increased in a targeted manner, making it look more vivid and vivid, and with a stronger sense of layering.

[0053] refer to Figure 5 As shown, in some embodiments, the PCIE controller provides a PCIE2.0 and above PCIE bus interface for the image processing card, supports DMA transmission mode, and is used for large data volume transmission. AXI4 and AXI4-Lite bus interfaces selectable by users can be provided, wherein the AXI4 bus interface is used to connect to the bus controller to implement data read and write access. The AXI4-Lite bus interface is used to read and write user registers, complete the initialization of the controller, and write and read control information. Reference Figure 2 The PCIE controller is also connected to an auxiliary circuit, which is used for functions such as conversion from differential clock to single-ended clock, reset synchronization, etc.

[0054] Specifically, the PCIE controller mainly uses the PCI Express IP core provided by Xilinx, which supports PCIE*4 bus and above interfaces, two independent write channels and two independent read channels to complete DMA reading and writing, support AXI, AXI-Lite bus interface and other main functions. The PCIE controller uses AXI-Lite configuration to complete the mapping of the PCIE controller interface address to the AXI-Lite address, which is used by the host side to access the registers on the user logic side or other AXI4-Lite bus devices through the PCIE controller. AXI-Lite and AXI4-Lite are simplified versions of the Advanced Extensible Interface (AXI), suitable for low-throughput communication.

[0055] refer to Figure 6 As shown, in some embodiments, the memory controller is used to implement the control flow of external double data rate (DDR) memory read and write operations. Specifically, the memory controller uses the memory IP core provided by Xilinx, which integrates the memory control module to implement the control flow of memory read and write operations, mainly including setting parameters such as device model, family, speed grade, interface protocol, clock cycle, voltage, data width, address width, physical layer and controller clock ratio.

[0056] refer to Figure 7 As shown, in some embodiments, the FPGA chip also includes a bus controller, which uses an AXI4 bus IP core interconnect bus and supports 5 channels, namely, read and write address channels, read and write data channels, and write response channels. Image data can be transmitted from the host to the slave and from the slave to the host at the same time, supporting 1 to many, many to 1, and many to many interconnections, supporting bus priority access, and internally realizing cross-clock domain conversion, FIFO, registers, bit width conversion, bus arbitration and other functions, reducing the complexity of mutual communication. There are 4 main devices in the interconnection system, namely, two image controllers, PCIE controller, and image acquisition controller, all of which need to access the memory. The two image controllers, PCIE controller, and image acquisition controller are interconnected through the bus controller to realize access to the memory, and the AXI4 interconnection bus is set for the application to improve the interconnection timing, which can improve the system performance. The bus controller also includes AXI4-Lite bus interconnection, which is mainly used to interconnect the two image controllers and PCIE controllers, etc., to complete the host's image timing configuration, image processor configuration and user-defined configuration; in addition, since the AXI4-Lite bus has cross-clock, the bus controller is used to solve the cross-clock domain problem to prevent the occurrence of metastable states.

[0057] refer to Figure 8 As shown, in some embodiments, a command controller is also included. The command controller completes the configuration of the image timing register, image parameter register, image controller register, and user-defined parameters by the host through the configuration interface, and accesses the user logic side register or other AXI4-Lite bus device mapping space through PCIE on the host side. The PCIE controller has an AXI-Lite bus Master interface, which maps the base address register (Base address register, abbreviated as: BAR) address to the AXI address. The host can access the BAR address of the PCIE and then convert the host's access operation to the AXI-Lite bus Master bus operation through the conversion of the PCIE controller, and then can implement the address access to other AXI-Lite bus Slaves inside the FPGA. As an AXI Slave device, the command controller maps the address to the AXI-Lite Master space of the PCIE controller, and converts the AXI-Lite to the memory mapping block random access memory (Block Random Access Memory, abbreviated as: BRAM) read and write timing, which can greatly facilitate the operation.

[0058] refer to Fig. 9As shown, in some embodiments, the state detection module mainly includes three parts: AXI interface, programmable timer, and control register. The AXI interface connects the state detection module to the bus controller through the AXI bus. As a slave device of the bus controller, the host can read and write the registers of the state detection module and perform interrupt response operations. The programmable timer is responsible for generating timed interrupts, and the interval between two interrupts is determined by the timer. The control register includes an interrupt mask register IMR, an interrupt request register IRR, an interrupt service register ISR, etc. The interrupt mask register IMR sets mask information for the interrupt request. When IMR=01, the interrupt source is prohibited from generating an interrupt. The interrupt request register IRR is used to latch the interrupt request signal sent by the timer. When the interrupt request signal is high, IRR=01, which is used to save the interrupt request status of the interrupt source. The interrupt service register ISR is used to store the interrupt request signal responded by the host. When an interrupt request arrives and is recorded in the interrupt request register, the state detection module notifies the host to generate an interrupt and waits for the host to process. At this time, the interrupt service register ISR=01, and IRR is cleared by the program, that is, IRR=0.

[0059] refer to Fig.10 As shown, in some embodiments, the image acquisition controller includes an AXI bus DMA controller, an adaptive logic circuit and a selection logic circuit; the adaptive logic circuit is used to detect the hotplug, +5V and other signals of the image interface circuit to determine which interface the image data is input from; the selection logic circuit selects one channel from multiple channels of image data and sends it to the AXI bus DMA controller to write it to the memory in DMA mode. Figure 2 As shown, in some embodiments, the image interface circuit includes a first image interface circuit, a second image interface circuit and a third image interface circuit, which respectively correspond to a DP interface, an HDMI interface and a video graphics array (Video Graphics Array, abbreviated as: VGA) interface. When multiple image interface circuits are connected at the same time, the interface selection priority is DP interface>HDMI interface>VGA interface.

[0060] The external interface signals of the image acquisition controller include: 1. AXI bus interface signal, which is the signal that the image acquisition controller, as the master device, accesses the bus controller; 2. Video interface signal, which is the signal after the image interface circuit transforms and buffers the image signal; 3. Memory interface signal, which is the signal that the image acquisition controller receives the data sent by the image interface circuit, synchronizes it, caches it in FIFO, and sends it to the AXI bus DMA controller, and writes it to the memory in DMA mode.

[0061] This embodiment also provides an interface adaptive image processing method based on FPGA, which is applied to the above-mentioned interface adaptive image processing system based on FPGA. The method includes the following steps:

[0062] Step 100: The image acquisition controller acquires data through the image interface circuit and transmits the data to the memory.

[0063] Step 200: The image processor reads and processes data from the memory through the image controller. The processed data is transmitted to the memory through the image controller and transmitted to the host for display through the PCIE controller.

[0064] Step 300: The status detection module periodically notifies the host software to detect the status of the image processing card. When the main image processing card fails, the host software switches the main image processing card to the standby image processing card.

[0065] In summary, the embodiment of the present invention provides an interface adaptive image processing system and method based on FPGA, by setting a main and standby image processing card and host software, wherein the main and standby image processing card has a status detection module for detecting the hardware status for the host software to identify and switch, once the main image processing card fails, it can switch to the standby image processing card to work, thereby improving the reliability of image processing. In addition, the image processing card can complete the image signal acquisition of different interfaces, realize the adaptation of different interfaces, perform video data normalization processing, provide a normalized video signal interface, and improve the versatility of image processing.

[0066] Embodiment 3:

[0067] Based on the above embodiments, this embodiment describes in detail the functions of the host software and the combination of software and hardware.

[0068] For the host software, it first completes the initialization of various hardware, including the initialization of the control image processor, timing, image processing module, user-defined registers, and image acquisition controller; and completes the establishment of threads, including the establishment of interrupt monitoring programs, sending and receiving programs. Then call the Pthread_join() function to wait for the specified thread to end in a blocking manner. Then read and write video data. Data reading and writing adopts interrupt mode. After the image controller generates an interrupt, the application obtains user interrupt information through the event event. If it is an interrupt for reading image data, the host reads data from the memory through the PCIE controller and sends it to the host memory; if it is an interrupt for writing data, the host reads image data from the memory and writes it to the memory through the PCIE controller; after the interrupt ends, the host clears the interrupt through software. Among them, the interrupt and data receiving and sending processes are synchronized in the form of semaphores; semaphores mainly complete the transmission function and are used for multiple processes to access shared resources. It is mainly reflected in that a thread completes a specific function under certain conditions, and other threads provide conditions. Other threads can call the sem_post() function, and this thread calls the sem_wait() function to wait. When the semaphore comes, it exits the blocking. In the interrupt process, the sem_post() function is called. The sem_post() function is used to release the semaphore after a process completes access to the shared resource. Other processes waiting for the semaphore can be executed. Calling this function will make one of the threads no longer blocked. In the data receiving and sending process, the sem_wait() function is used to wait for the semaphore. It starts to execute after the interrupt process releases the semaphore. After execution, the thread resources are recovered.

[0069] The combination of the above hardware and software has high reliability for image processing. Specifically, the image processing device is composed of a main image processing card and a standby image processing card. The main image processing card and the standby image processing card have the same hardware. When the hardware of the main image processing card fails, the main image processing card is switched to the standby image processing card through software. The hardware and software are combined to achieve high reliability of the device. For hardware failure, the hardware of the image processing card is mainly composed of FPGA and dynamic random access memory (Dynamic Random Access Memory, abbreviated as: DRAM) chips. Since most of the hardware functions of the image processing card are implemented by FPGA, the hardware failure is mainly caused by the failure of the FPGA and DRAM chips in the image processing card or the welding process. The software can include initialization and interrupt service programs. After the initialization is completed, the FPGA chip status detection is completed by the internal status detection module. The host software initializes the detection module, and the status detection module generates a timing detection interrupt and sends it to the host. After the host receives the interrupt, the interrupt service program performs a read and write detection on the specific storage space of DRAM to determine whether the DRAM read and write is normal. If no detection interrupt is generated, it indicates that the FPGA is faulty. If an interrupt is generated, but the read and write data is abnormal, it is determined that the DRAM is faulty.

[0070] For the switch from the main image processing card to the standby image processing card, the main process includes: 1. The main software program initializes the main image processing card and the standby image processing card. First, the device table in the configuration file is traversed. The device table contains the addresses and interrupt configuration parameters of the two image processing cards. The configuration parameters of the two cards are passed to the loading function. The loading function creates a control structure for each card, parses the configuration parameters, and allocates memory space for the main image processing card and the standby image processing card. The system configuration requires the initialization of the two image processing cards, and the control structure of the image card is filled at the same time. The interrupt status flag of the main image processing card and the standby image processing card detection signal is set to 0, and the interrupt mask register is set to 1, and then the fault diagnosis and identification task is started. 2. The interrupt mask register of the main image processing card and the standby image processing card is set to 0. The main image processing card and the standby image processing card generate an interrupt, and the corresponding interrupt request register IRR is set to 1. After the host detects the interrupt signal, the interrupt service program performs read and write detection on the specific storage space of DRAM to determine whether it is normal, and the interrupt service register ISR is set to 1. If it is normal, the interrupt status flag remains 0, the interrupt service program exits, and the interrupt service register ISR and the interrupt request register IRR are set to 0 synchronously. If abnormal, the interrupt status flag is set to 1. The main image processing card completes the image data acquisition, processing and other functions. 3. The status detection and identification task uses the auxiliary clock interrupt to periodically detect the image card semaphore interrupt status flag. Each timed interrupt releases a binary semaphore. The status detection task waits for the received semaphore in a loop. After receiving the semaphore, it immediately determines the status of the main image processing card device. When the semaphore interrupt status flag is in the state of 0, the task waits for the next semaphore in a loop. When the state is 1, that is, a fault, the status detection task exchanges the control structure content of the main image processing card and the standby image processing card, and the standby image processing card starts working. The standby image processing card completes the image data acquisition, processing and other functions. Among them, the control structure includes interrupt vector, address and other information, which are in the image card device table.

[0071] In summary, the embodiments of the present invention provide an interface adaptive image processing system and method based on FPGA, which adopts a combination of hardware and software to achieve high reliability of image processing; realize adaptive access and acquisition functions of multiple video interfaces; realize video data normalization processing, and provide a normalized video signal interface for the image processor. Combined with the programmable characteristics of FPGA, high-performance processing of images with different functional requirements is completed through reconstruction, thereby realizing the versatility of hardware.

[0072] The system and method realize image acquisition, transmission and processing based on FPGA, and have the characteristics of interface adaptability, high reliability, high performance, versatility, strong portability, high cost performance, etc., which can be applied to general image processing, especially high reliability transmission and processing application scenarios. It can also meet other application requirements such as high-performance heterogeneous computing, data processing, AI and other application scenarios, and has a very broad application prospect.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interface adaptive image processing system based on FPGA, characterized in that: It includes a host and an image processing module, wherein the host is provided with host software, and the image processing module includes two identical image processing cards, one of which is a main image processing card and the other is a standby image processing card; The image processing card includes an FPGA chip, a memory and several image interface circuits. The FPGA chip includes an image acquisition controller, an image processor, an image controller, a PCIE controller and a state detection module. The image acquisition controller acquires data through the image interface circuit and transmits it to the memory. The image processor reads and processes data from the memory through the image controller. The processed data is transmitted to the memory through the image controller and transmitted to the host for display through the PCIE controller. The status detection module periodically notifies the host software to detect the status of the image processing card. When the main image processing card fails, the host software switches the main image processing card to the standby image processing card.

2. The interface adaptive image processing system based on FPGA according to claim 1, characterized in that: The state detection module generates an interrupt signal at a fixed time. After the host software detects the interrupt signal, it performs a read and write test on the memory to determine whether it is normal. If it is normal, the interrupt state flag is kept at 0 and the interrupt service program is exited; if it is abnormal, the interrupt state flag is set to 1; The host software periodically detects the interrupt status mark of the main image processing card through the status detection and identification task. Each timed interrupt releases a binary semaphore. The status detection and identification task waits for the received semaphore in a loop, and immediately determines the status of the main image processing card after obtaining the semaphore. When the interrupt status mark is 0, the main image processing card is judged to be normal, and the loop continues to wait for the next semaphore. When the interrupt status mark is 1, the main image processing card is judged to be faulty, and the standby image processing card takes over the main image processing card to start working.

3. The interface adaptive image processing system based on FPGA according to claim 1, characterized in that: The image processor includes a user-defined image processing function module, a timing conversion module, a pre-processing data synchronization and conversion module, and a post-processing data synchronization and conversion module; wherein the user-defined image processing function module is used to configure and design the user-defined image processing function in real time according to different application requirements to enhance the image quality; the timing conversion module is used to convert the image data into video timing; the pre-processing data synchronization and conversion module is used to synchronize the video timing signal with the AXI bus, output a normalized video signal, and provide a unified interface for processing by the user-defined image processing function module; the post-processing data synchronization and conversion module is used to convert the data signal after processing by the user-defined image processing function module into AXI bus data, and transmit it to the memory via the image controller.

4. The interface adaptive image processing system based on FPGA according to claim 1, characterized in that: The image controller includes a register and a data transmission module. The image controller receives a host's initialization request for the register through an AXI4-Lite bus. When performing a write operation, the data transmission module writes the video stream data into the memory in a DMA manner according to the register information. When performing a read operation, the data transmission module reads the data in the memory in a DMA manner according to the register information.

5. The interface adaptive image processing system based on FPGA according to claim 4, characterized in that: The image controller includes a first image controller and a second image controller. The first image controller is responsible for reading data, and the second image controller is responsible for writing data.

6. The interface adaptive image processing system based on FPGA according to claim 1, characterized in that: The image acquisition controller includes an AXI bus DMA controller, an adaptive logic circuit and a selection logic circuit; the adaptive logic circuit is used to detect the signal of the image interface circuit and determine which interface the image data is input from; The selection logic circuit selects one channel of image data from multiple channels and sends it to the AXI bus DMA controller, and writes it into the memory in DMA mode.

7. The interface adaptive image processing system based on FPGA according to claim 6, characterized in that: The image interface circuit corresponds to a DP interface, an HDMI interface and a VGA interface. When multiple image interface circuits are connected at the same time, the interface selection priority is DP interface>HDMI interface>VGA interface.

8. The interface adaptive image processing system based on FPGA according to claim 1, characterized in that: The PCIE controller uses AXI-Lite configuration to complete the mapping of the PCIE controller interface address to the AXI-Lite address, which is used by the host side to access the registers on the user logic side or other AXI4-Lite bus devices through the PCIE controller.

9. The interface adaptive image processing system based on FPGA according to any one of claims 1 to 8, characterized in that: The FPGA chip also includes a bus controller, which uses an AXI4 bus IP core interconnection bus. The image controller, the PCIE controller and the image acquisition controller are interconnected through the bus controller to achieve access to the memory.

10. An interface adaptive image processing method based on FPGA, applied to the interface adaptive image processing system based on FPGA as claimed in any one of claims 1 to 9, characterized in that: include: The image acquisition controller acquires data through the image interface circuit and transmits it to the memory; The image processor reads and processes data from the memory through the image controller, and the processed data is transmitted to the memory through the image controller and transmitted to the host display through the PCIE controller; The status detection module periodically notifies the host software to detect the status of the image processing card. When the main image processing card fails, the host software switches the main image processing card to the standby image processing card.

Images