Three-party cooperative processing video real-time transmission and processing heterogeneous system

Abstract

The invention relates to the technical field of digital videos, in particular to a three-party cooperative processing video real-time transmission and processing heterogeneous system which comprises a server CPU end, an ARM end, an FPGA end and a plurality of modules, after the system is powered on, the server CPU end serves as a client side to obtain video stream data and sends the video stream data to the FPGA end and the ARM end, the ARM end uses an LWIP interface as an IP protocol stack, and the LWIP interface serves as an IP protocol stack; the FPGA receives video stream data sent by the CPU through an udp protocol, the video stream data is temporarily stored in the DDR storage chip externally connected with the FPGA, the image processing module obtains the video stream data and then processes the video stream data, and the processed video stream data displays and compares an original video and a processed video through the HDMI display IP module. The high parallel computing capability of the FPGA, the low power consumption control capability of the ARM and the flexibility of the CPU are cooperatively utilized, real-time transmission, processing and display of video streams are achieved, the replaceable image processing module is further designed, various image processing functions are achieved through the user-defined image processing module, and various application requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of digital video technology, and in particular to a video real-time transmission and processing heterogeneous system for three-party collaborative processing. Background Art

[0002] The wide application of video images in the fields of intelligent monitoring, autonomous driving, and intelligent manufacturing has led to increasingly higher requirements for real-time and flexibility in video image processing. In addition to general-purpose CPU chips, other types of special-purpose computing chips such as GPU, DSP, ARM, FPGA, and ASIC are widely used in real-time video transmission and processing acceleration computing tasks. Among them: CPU, as a general-purpose computing architecture, has a mature software ecosystem and flexible programming interface, but when facing 4K and above high-resolution video processing, its single-thread computing power is low, its computing efficiency is insufficient, and its power consumption is high during high-frequency operations, which is not suitable for energy-sensitive embedded devices; ARM is widely used in embedded scenarios and is known for its low power consumption and high efficiency, but ARM's memory bandwidth is limited and it is difficult to meet the real-time requirements of high-resolution video processing. There is a significant gap compared to CPU and GPU, and because ARM uses a special instruction set that is incompatible with the x86 architecture, there are development efficiency issues in software transplantation; FPGA has shown great potential in the field of video processing with its highly parallel computing power and low latency characteristics, and is especially suitable for computing-intensive tasks, but FPGA development has a high threshold, debugging is difficult, and lacks a flexible software ecosystem. When processing complex logic or dynamic task switching, its flexibility is far less than that of CPU or ARM.

[0003] Since a single computing architecture cannot meet the comprehensive requirements of real-time, flexibility and low power consumption of video image processing at the same time, designing heterogeneous systems with different chip combinations based on video transmission and processing requirements and application purposes, and reasonably allocating computing tasks and achieving complementary advantages have become important directions for solving efficient video transmission and processing. However, existing heterogeneous system solutions usually focus on single task acceleration and lack comprehensive optimization design for video streaming and multi-module collaboration, which limits their performance potential in complex application scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a three-party collaborative video real-time transmission and processing heterogeneous system, aiming to solve the problems of insufficient real-time and high efficiency and poor functional scalability of the existing video real-time transmission and processing heterogeneous system.

[0005] To achieve the above-mentioned purpose, the present invention provides a three-party collaborative video real-time transmission and processing heterogeneous system, including a server CPU side, an ARM side, an FPGA side, a VDMA data stream transport IP, a VTC timing control IP, a VID OUT IP, an HDMI display IP and a custom image processing module, and each part is interconnected by an AXI bus;

[0006] The server CPU is responsible for video stream acquisition and preliminary transmission as well as preprocessing of input data. The ARM is responsible for receiving the video and transmitting it to the FPGA. The FPGA is responsible for data calculation acceleration, separating the calculation load to the FPGA, and transmitting the processed data back to the server CPU.

[0007] The ARM module includes a cortex-A9 processor, an AXI communication interface, a DDR storage and an SD card. The cortex-A9 processor is connected to the AXI bus transmission protocol through the AXI communication interface, and the cortex-A9 processor is also bidirectionally connected to the DDR storage and the SD card respectively;

[0008] The VDMA data stream handling IP can read data from the ARM-side DDR storage and then output it in the form of AXI-Stream;

[0009] The VTC timing control IP generates RGB video timing, is responsible for generating timing signals synchronized with display output, and works in conjunction with the VDMA data stream handling IP module;

[0010] VID OUT IP converts Stream data into RGB timing and gives it to HDMI display IP;

[0011] The HDMI display IP is responsible for converting the RGB image data of the video output into an output signal that complies with the HDMI protocol, and transmitting the original video and the processed video data to an external display device for comparison;

[0012] The custom image processing module is a free expansion module, which is encapsulated into an AXI4 standard interface and has six main input and output interfaces, including three AXI4 interfaces and three control interfaces.

[0013] The custom image processing module is designed with three replaceable modules, including a SOBEL edge detection module, a Harris corner detection module, and an image classification module;

[0014] The SOBEL edge detection module is used to calculate the edge features of the image, the Harris corner detection module is used to extract image feature points, and the image classification module is used for image classification based on CNN LeNet. The three modules all realize data interaction and control through six main interfaces.

[0015] Among them, the three AXI4 interfaces of the SOBEL edge detection module are s_axi_AXILiteS, INPUT_STREAM, and OUTPUT_STREAM, and the three control interfaces are ap_clk, ap_rst_n, and interrupt;

[0016] The s_axi_AXILiteS interface is responsible for register configuration communication with the host system for IP core initialization, parameter setting, and status reading;

[0017] The INPUT_STREAM interface accepts video stream data and supports the AXI4-Stream protocol. It is used to receive raw image data or data output from the previous processing stage.

[0018] The OUTPUT_STREAM interface outputs the processed edge detection image data and supports the AXI4-Stream protocol, which is convenient for connecting to subsequent processing modules or display interfaces;

[0019] The ap_clk interface is responsible for the system clock signal and drives the logic operation inside the IP core;

[0020] The ap_rst_n interface is responsible for the global reset signal, which is valid at low level and is used to initialize all internal states of the IP core;

[0021] The interrupt interface is responsible for the interrupt signal, indicating the completion status of the current processing task.

[0022] Among them, the three AXI4 interfaces of the Harris corner point detection module are s_axi_control, img_inp, and img_out, and the three control interfaces are ap_clk, ap_rst_n, and interrupt;

[0023] The s_axi_control interface is based on the AXI4-Lite protocol and is used to communicate with the host processor. It is responsible for the parameter configuration and operation status monitoring of the IP core.

[0024] The img_inp interface is based on the AXI4-Stream protocol and is used to receive input image data streams;

[0025] The img_out interface uses the AXI4-Stream protocol to output the image data stream containing the corner detection results;

[0026] The ap_clk interface represents the system global clock signal, which drives all logical operations inside the IP core;

[0027] The ap_rst_n interface represents the global reset signal, which is valid at low level and is used to initialize all registers and internal states of the IP core during system startup or abnormal conditions.

[0028] The interrupt interface represents an interrupt signal. A high level output indicates that the current processing task is completed or an exception occurs.

[0029] Among them, the three AXI4 interfaces of the picture classification module are s_axi_CTRL_BUS, input_r_PORTA, and output_r_PORTA, and the three control interfaces are ap_clk, ap_rst_n, and interrupt respectively;

[0030] The s_axi_CTRL_BUS interface is based on the AXI4-Lite protocol and is used to interact with the host system to implement the configuration and operation status management of the IP core;

[0031] The input_r_PORTA interface is based on the AXI4 protocol and is used to receive input image data;

[0032] The output_r_PORTA interface is used to output classification results, including the classification labels and confidence score data of the image, and supports direct integration with downstream processing modules.

[0033] The ARM end uses the LWIP interface as an IP protocol stack and receives the video stream data sent by the server CPU end through the UDP protocol.

[0034] The FPGA ends are all externally connected to a DDR memory chip for temporarily storing video stream data.

[0035] The present invention provides a three-party collaborative video real-time transmission and processing heterogeneous system, including a server CPU end, an ARM end, an FPGA end and a plurality of modules. After the system is powered on, the server CPU end obtains video stream data as a client, and sends the video stream data to the FPGA end and the ARM end. The ARM end uses the LWIP interface as an IP protocol stack, receives the video stream data sent by the CPU through the UDP protocol, and temporarily stores the video stream data in a DDR storage chip external to the FPGA. The image processing module obtains the video stream data and processes it. The processed video stream data is displayed by the HDMI display IP module to compare the original video and the processed video. The present invention collaboratively utilizes the high parallel computing capability of FPGA, the low power consumption control capability of ARM and the flexibility of CPU to realize real-time transmission, processing and display of video streams. In addition, a replaceable image processing module is designed, and a variety of image processing functions are realized through a custom image processing module to adapt to a variety of application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a structural schematic diagram of a three-party collaborative video real-time transmission and processing heterogeneous system of the present invention.

[0038] Figure 2 It is a schematic diagram of the structural principle of an FPGA module of a three-party collaborative video real-time transmission and processing heterogeneous system of the present invention.

[0039] Figure 3 It is a schematic diagram of a Sobel edge detection IP module according to a specific embodiment of the present invention.

[0040] Figure 4 It is a schematic diagram of the system operation results detected by the Sobel edge detection IP module in a specific embodiment of the present invention.

[0041] Figure 5 It is a structural diagram of a Harris corner detection IP module according to a specific embodiment of the present invention.

[0042] Figure 6 It is a schematic diagram of system operation results detected by using Harris corner detection IP module in a specific embodiment of the present invention.

[0043] Figure 7It is a structural diagram of a CNN LeNet image classification module according to a specific embodiment of the present invention.

[0044] Figure 8 This is a schematic diagram of the system operation results detected by the CNN LeNet image classification module in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0045] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0046] See also Figure 1 , the present invention provides a three-party collaborative video real-time transmission and processing heterogeneous system, including a server CPU end, an ARM end, an FPGA end, a VDMA data stream transport IP, a VTC timing control IP, a VID OUT IP, an HDMI display IP and a custom image processing module, and each part is interconnected by an AXI bus;

[0047] The server CPU is responsible for video stream acquisition and preliminary transmission as well as preprocessing of input data. The ARM is responsible for receiving the video and transmitting it to the FPGA. The FPGA is responsible for data calculation acceleration, separating the calculation load to the FPGA, and transmitting the processed data back to the server CPU.

[0048] The ARM module includes a cortex-A9 processor, an AXI communication interface, a DDR storage and an SD card. The cortex-A9 processor is connected to the AXI bus transmission protocol through the AXI communication interface, and the cortex-A9 processor is also bidirectionally connected to the DDR storage and the SD card respectively;

[0049] like Figure 2As shown, in this embodiment, after the system is powered on, the server CPU side acts as a client, and through the UDP protocol, with the help of the python host computer, calls the camera to obtain video stream data, and sends the video stream data to the FPGA-ARM side, that is, the server side. The ARM side uses the LWIP interface as an IP protocol stack, receives the video stream data sent by the CPU through the UDP protocol, and temporarily stores the video stream data in the DDR storage chip connected to the FPGA side. After the custom image processing module (ISP) obtains the video stream data, it takes SOBEL edge detection as an example to perform edge detection processing. The processed video stream data is displayed through the HDMI display IP to display the original video and the edge detection video as a comparison. The ARM processor is connected to the DDR3 / DDR4 SDRAM memory chip for caching the video stream. The FPGA is used to carry the video stream data, then perform the detection processing of the video stream, and display the detected data. The system is designed with a replaceable image processing module. Through the flexibility of the modular design of the custom image processing module (ISP), users can configure different IP cores to implement a variety of image processing functions, such as corner detection, target recognition, etc., to adapt to a variety of application requirements.

[0050] The following is a further description in combination with specific embodiments and system processes:

[0051] See also Figures 3 to 8 , In this embodiment, the Zynq processor is used as the core processing unit.

[0052] The workflow is as follows:

[0053] ①Image data input

[0054] The server CPU runs the host computer, collects the video stream, and then transmits it to the system card through the Gigabit Ethernet port after preprocessing. The system card receives the server data source through the ARM-side LWIP protocol stack and inputs the image data into the Zynq processor as the raw data for subsequent image processing. Zynq is responsible for coordinating the operation of each module and the distribution of data streams.

[0055] ②Custom programmable image processing module (ISP)

[0056] The input image data is first passed to the custom programmable image processing module (ISP) IP module. This module is encapsulated into a standard AXI4 interface. Taking the SOBEL edge detection module as an example, a specific algorithm is used to extract feature points from the image and generate an edge detection result data stream containing feature information.

[0057] ③AXI bus interconnection

[0058] The data generated by the SOBEL edge detection module is transmitted through the AXI bus interconnection IP module. This module serves as the core channel for data exchange within the system, ensuring that data streams can be transmitted between different IP modules in an efficient and reliable manner.

[0059] ④VDMA data stream transport IP module processing

[0060] The data stream enters the VDMA data stream handling IP module and is output in the form of an AXI-Stream stream. The module converts the data from the AXI bus format to a storage format suitable for subsequent processing, and buffers and transmits the data to ensure the real-time and stability of system processing.

[0061] ⑤VTC timing control IP module

[0062] The VTC timing control IP module is responsible for generating RGB video timing and generating timing signals synchronized with display output, ensuring that the image can be presented in the correct time sequence on the display device. This module works in conjunction with the VDMA data stream handling IP module to provide stable timing support for subsequent video output modules.

[0063] ⑥Video output and display

[0064] After the VTC timing control IP module generates the display timing, the data stream is passed to the video output (VID OUT IP) module, which completes the format conversion and final processing of the image data and converts it into a signal that conforms to the RGB format.

[0065] ⑦HDMI interface output

[0066] The image data processed by the video output module is further transmitted to the HDMIIP module, which converts the RGB image data into an output signal that complies with the HDMI protocol, and transmits the original video and processed video data to an external display device for comparison.

[0067] ⑧Display terminal presentation

[0068] Finally, the processed image is presented in real time on the display screen. Through closed-loop design, the system ensures that the output image has high-precision timing synchronization and feature accuracy to meet the requirements of high-definition display.

[0069] Among them, the custom programmable image processing module (ISP) can be freely expanded and encapsulated into an AXI4 standard interface. It has six main input and output interfaces, including three AXI4 interfaces (s_axi_AXILite, INPUT_STREAM, OUTPUT_STREAM) and three control interfaces (ap_clk, ap_rst_n, interrupt). Three modules are designed for experiments, including the SOBEL edge detection module, the Harris corner detection module, and the image classification module.

[0070] 1. Custom Sobel edge detection IP core

[0071] The custom Sobel edge detection IP core is a hardware module based on FPGA implementation, which is used to efficiently calculate the edge features of images. The IP core has six main input and output interfaces, including three AXI4 interfaces (s_axi_AXILiteS, INPUT_STREAM, OUTPUT_STREAM) and three control interfaces (ap_clk, ap_rst_n, interrupt).

[0072] AXI4 Interface

[0073] s_axi_AXILiteS: Responsible for register configuration communication with the host system (such as an ARM processor) for initialization, parameter setting, and status reading of the IP core.

[0074] INPUT_STREAM: Accepts video stream data and supports the AXI4-Stream protocol. It is used to receive raw image data or data output from the previous processing stage.

[0075] OUTPUT_STREAM: Outputs processed edge detection image data, supports AXI4-Stream protocol, and is easy to connect to subsequent processing modules or display interfaces.

[0076] Control interface

[0077] ap_clk: system clock signal, driving the logic operations inside the IP core.

[0078] ap_rst_n: global reset signal, valid at low level, used to initialize all internal states of the IP core.

[0079] interrupt: An interrupt signal that indicates the completion status of the current processing task, making it easier to collaborate with the host system to achieve multitasking.

[0080] Extended description: interface functions and their implementation details

[0081] ① Functional extension and role of s_axi_AXILiteS interface

[0082] This interface is based on the AXI4-Lite protocol and is mainly used for low-speed configuration and control. Users can flexibly adjust the threshold parameters, enable switches or mode selections (such as edge direction selection, grayscale image format, etc.) of the Sobel algorithm through register settings. In addition, this interface allows reading the operating status of the IP core, including processing frame rate, error flags or interrupt status register values, which is convenient for debugging and operation monitoring.

[0083] Register layout:

[0084] 0x00-Control register: start / stop control, reset signal enable.

[0085] 0x04-Status register: interrupt status flag, current task completion flag.

[0086] 0x08 - Parameter register: User configurable threshold or filter strength parameters.

[0087] ② Functions and advantages of INPUT_STREAM and OUTPUT_STREAM interfaces

[0088] Both interfaces are based on the AXI4-Stream protocol and are used to process input and output video stream data.

[0089] INPUT_STREAM: supports standardized video stream formats, such as RGB or grayscale data streams, and can be directly connected to video acquisition modules (such as VDMA or external cameras).

[0090] OUTPUT_STREAM: Outputs the edge map data stream processed by the Sobel algorithm, which can be seamlessly connected to the display module (such as the HDMI interface) or subsequent algorithm modules (such as the Harris corner detection IP core).

[0091] Key Features:

[0092] Supports high-throughput data stream processing, suitable for high-resolution video applications (such as 4K or 8K).

[0093] Flow control with upstream and downstream modules is achieved through handshake signals (VALID / READY) to ensure system data consistency.

[0094] ③Implementation and timing constraints of ap_clk and ap_rst_n interfaces

[0095] ap_clk: The global clock signal of the FPGA system, which ensures the synchronous operation of the logic circuit inside the IP core. The clock frequency needs to match the frame rate of the external data stream. For example, a 1080p video stream usually requires a clock frequency of more than 75MHz.

[0096] ap_rst_n: system reset signal, which ensures that the IP core can be initialized correctly during power-on or abnormal conditions to avoid data loss or abnormal processing.

[0097] ④Interrupt signal multi-task collaboration support

[0098] Interrupt signal function: used to indicate the processing completion status of the current frame data. When the edge detection task of a video frame is completed, the interface outputs a high-level signal to trigger the host system to respond.

[0099] Typical application: In a multi-frame processing scenario, the host uses an interrupt signal to determine whether the next frame of data can be transmitted to avoid resource conflicts.

[0100] 2. Customize Harris Corner Detection IP Core

[0101] The custom Harris corner detection IP core is an efficient image feature point extraction module based on FPGA hardware implementation, designed for embedded real-time image processing systems. The IP core implements data interaction and control through six main interfaces, including three AXI4 interfaces (s_axi_control, img_inp, img_out) and three system control interfaces (ap_clk, ap_rst_n, interrupt).

[0102] Interface functions and features

[0103] AXI4 Interface

[0104] s_axi_control

[0105] This interface is based on the AXI4-Lite protocol and is used to communicate with the host processor (such as ARM Cortex-A9). It is responsible for the parameter configuration and operation status monitoring of the IP core. Through this interface, users can dynamically set the parameters of corner detection (such as response function threshold, window size) and read the metadata of corner detection results (such as the number of corners, processing status flags).

[0106] img_inp

[0107] This interface is based on the AXI4-Stream protocol and is used to receive input image data streams. It supports standard image formats (such as grayscale images) or pre-processed image data to ensure seamless connection with video acquisition modules or other front-end processing modules (such as Sobel edge detection IP core).

[0108] img_out

[0109] The AXI4-Stream protocol is also used to output the image data stream containing the corner detection results. This interface supports superimposing the corner features on the original image or outputting the feature point information separately, and passing it to the display module or further processing module through the downstream interface.

[0110] System control interface

[0111] ap_clk

[0112] The system global clock signal drives all logical operations inside the IP core. The clock frequency must match the frame rate of the input image data stream to ensure the real-time performance of image processing.

[0113] ap_rst_n

[0114] Global reset signal, low level is effective. It is used to initialize all registers and internal states of the IP core during system startup or abnormal conditions to ensure stable operation of the system.

[0115] interrupt

[0116] Interrupt signal, outputting high level indicates that the current processing task is completed or an exception occurs. This signal facilitates collaboration with the host processor and enables efficient management of multiple tasks through the interrupt mechanism.

[0117] Extended description: interface implementation and system integration

[0118] ① Extended functions and implementation of s_axi_control interface

[0119] This interface is not only used for conventional parameter configuration and status reading, but also supports dynamic runtime adjustment to adapt to different image processing scenarios.

[0120] Register function distribution:

[0121] Control register (0x00): used to start / stop the corner point detection task and reset the module.

[0122] Parameter register (0x04-0x08): Set key parameters of corner detection, such as response function threshold and Gaussian filter window size.

[0123] Status register (0x0C): records the processing status (idle, busy, completed) and error flags.

[0124] Real-time monitoring function: Through this interface, the host can dynamically query statistical information such as the number of corner points and response value distribution, which is convenient for subsequent feature analysis and utilization.

[0125] ②Functions and advantages of img_inp and img_out interfaces

[0126] img_inp interface: supports the standard AXI4-Stream protocol and can be directly connected to external camera acquisition modules (such as MIPI camera interface) or pre-processing modules (such as VDMA or Sobel edge detection module). Its flow control mechanism ensures the synchronization and integrity of data transmission through VALID / READY signals, avoiding data loss even in complex multi-tasking scenarios.

[0127] img_out interface: The output data stream is also based on the AXI4-Stream protocol. The output image data can contain corner feature annotations (such as marking corners with different colors on the original image) or only output corner coordinate data. The flexibility of the data stream format facilitates seamless connection with subsequent modules (such as HDMI display module or deep learning classification module).

[0128] ③ System integration requirements for ap_clk and ap_rst_n

[0129] Clock frequency adaptation: To ensure real-time performance, the clock frequency must match the video frame rate and data stream rate. For example, for a 1080p@30fps video stream, the system needs to be configured with a clock frequency of more than 75MHz.

[0130] Reset mechanism: When the system is powered on or restarted, a low-level reset signal triggers the initialization operation of all registers and caches of the IP core, ensuring that the task starts from a clean state.

[0131] ④Interrupt task collaboration capability

[0132] The interrupt signal provides an efficient collaboration mechanism between the host and the IP core. When the IP core completes the corner detection task of a frame of image, the interrupt signal is pulled high to notify the host to proceed to the next step, such as transmitting the next frame of image or starting other processing modules. At the same time, the interrupt signal can also feedback abnormal conditions, such as data flow interruption or memory access error.

[0133] 3. Image classification module

[0134] The custom CNN LeNet deep learning image classification IP core in the image classification module is an efficient image classification module based on FPGA hardware implementation, which is optimized for embedded vision systems. The IP core implements data interaction and control through six main interfaces, including three AXI4 interfaces (s_axi_CTRL_BUS, input_r_PORTA, output_r_PORTA) and three system control interfaces (ap_clk, ap_rst_n, interrupt).

[0135] Interface Function Description

[0136] AXI4 Interface

[0137] s_axi_CTRL_BUS

[0138] Based on the AXI4-Lite protocol, this interface is used to interact with the host system (such as the ARM Cortex-A series processors), mainly to implement the configuration and operation status management of the IP core. Through this interface, users can set network parameters (such as classification layer weight bias), adjust operation modes (such as reasoning mode and test mode), and monitor the status of the current task (such as classification completion flag, fault status, etc.).

[0139] input_r_PORTA

[0140] This interface is based on the AXI4 protocol and is used to receive input image data. The input data can be a formatted image tensor, such as grayscale or RGB image data generated by preprocessing a video stream or memory.

[0141] output_r_PORTA

[0142] This interface is used to output classification results, including image classification labels, confidence scores, and other data, and supports direct integration with downstream processing modules. The output data can be further processed by the host system or used for real-time display and control applications.

[0143] System control interface

[0144] ap_clk

[0145] The system clock signal drives the convolution operation and fully connected layer calculation inside the IP core. The clock frequency needs to match the computing requirements of the system to ensure the real-time and accuracy of the inference process.

[0146] ap_rst_n

[0147] Global reset signal, valid at low level, used to initialize all modules of the IP core, including the convolution layer, pooling layer, and classification layer, to ensure that the system is in a consistent state when starting or recovering from an abnormality.

[0148] interrupt

[0149] Interrupt signal, which triggers high-level output when the classification task is completed or an abnormal situation occurs, notifies the host system to read the results or perform error processing.

[0150] Extended description: interface functions and system integration

[0151] ① Functional expansion and implementation of s_axi_CTRL_BUS interface

[0152] Key features:

[0153] Task start and stop: Through register settings, the host can control the start or interruption of the classification task.

[0154] Parameter configuration: Users can dynamically load or adjust some weights and biases of the CNN network, such as the classification layer weight matrix, to adapt to different data sets or task requirements.

[0155] Operation monitoring: Through the status register, the host can obtain the operation status of the IP core in real time, including the current inference progress and error flags.

[0156] Register layout example:

[0157] 0x00: Control register, used to start, stop and reset the task.

[0158] 0x04: Status register, used to mark the task completion status or exception flag.

[0159] 0x08~0x10: Parameter register group, used to set the network weights and biases of the classification layer.

[0160] ②Flexibility and advantages of input_r_PORTA and output_r_PORTA interfaces

[0161] input_r_PORTA

[0162] Supports standardized tensor data input, such as grayscale or RGB images generated by image preprocessing modules (such as VDMA or camera interface). The interface uses a handshake protocol to ensure that the system can still correctly receive data and avoid frame loss when the input data stream is discontinuous or has jitter.

[0163] output_r_PORTA

[0164] The output classification result can be a fixed-length tensor (such as 10-class output), or expanded to a variable length to support larger classification tasks (such as 1000 classes). The output data structure of this interface includes classification labels and confidence scores, providing a convenient data interface for subsequent processing or real-time display.

[0165] ③ Timing and reset strategy of ap_clk and ap_rst_n

[0166] Clock synchronization:

[0167] The clock frequency needs to be selected according to the network complexity and inference speed requirements. For example, for a typical LeNet network, the system usually requires a clock frequency of more than 100MHz to achieve real-time performance.

[0168] The clock domain crossing design ensures data read and write operations are synchronized with external modules (such as DDR or SDRAM memory).

[0169] Reset design:

[0170] The reset signal can be controlled by the host through s_axi_CTRL_BUS or triggered by an external hardware button.

[0171] During the reset process, the internal cache, computing units (convolutional layer, pooling layer), and control logic of the IP core will be reset to ensure fast recovery after hardware failure or data flow interruption.

[0172] ④Task coordination mechanism of interrupt signal

[0173] The interrupt signal is an important mechanism for task coordination between the host and the IP core. When the classification task is completed, the interrupt signal outputs a high level to notify the host to read the result or start the next task. In multi-tasking scenarios, the interrupt signal supports multi-level priority processing, such as notifying the host system to prioritize higher priority results in batch processing mode.

[0174] The classic convolutional neural network LeNet is trained on the CIFAR-10 dataset to generate weights and perform inference based on these weights. The process includes the complete implementation of dataset preparation, model training, weight preservation, and inference steps. Finally, the system is powered on and the results are as follows: Figure 7 As shown, it can be found that the correct result can be obtained.

[0175] Collaboration and application scenarios between custom programmable image processing modules (ISP) interfaces

[0176] Through the collaborative work of the above interfaces, the custom image processing module (ISP) IP core can achieve a complete closed loop from image data stream input, image feature extraction, to result output. The system's modular design and standard interface support facilitate seamless integration with other IP cores (such as VDMA modules or HDMI display modules), and is suitable for a variety of real-time image processing scenarios.

[0177] In summary, the present invention has the following beneficial effects:

[0178] 1. Separation and collaboration of processing and transmission: In existing video processing systems, the CPU is usually responsible for both data acquisition and partial processing, which can easily cause performance bottlenecks. This system separates the processing load to the FPGA by focusing the PC's CPU on video stream acquisition and preliminary transmission. The FPGA performs efficient image processing such as edge detection or target recognition through a hardware pipeline structure. The ARM uses a lightweight IP protocol stack to quickly receive data and achieve low-latency cache management. At the same time, it makes full use of the high flexibility of the CPU, the low power consumption and high efficiency of the ARM, and the core computing acceleration of the FPGA, improving the overall system performance and task division efficiency.

[0179] 2. Insufficient modular scalability: Traditional image processing systems mostly use fixed-function hardware design, which is difficult to meet diverse application requirements. This system designs programmable and replaceable image processing modules (such as Sobel edge detection, Harris corner detection or deep learning modules) and encapsulates them into a unified interface. Users can replace the processing IP core according to different scene requirements, thereby realizing rapid expansion and customization of system functions.

[0180] 3. Optimization of data transmission and cache: The video stream transmission efficiency in the existing system is low, especially when processing high-resolution video, it is prone to frame loss or delay problems. This system uses the onboard ARM core and DDR3 / DDR4 memory to optimize the video stream cache, combined with the FPGA's VDMA data stream handling IP to ensure the full link efficiency from data reception, storage to processing, while avoiding the performance bottleneck caused by insufficient data bandwidth in traditional systems.

[0181] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A three-party collaborative video real-time transmission and processing heterogeneous system, characterized in that: It includes server CPU, ARM, FPGA, VDMA data stream handling IP, VTC timing control IP, VID OUT IP, HDMI display IP and custom image processing module, and each part is interconnected by AXI bus; The server CPU is responsible for video stream acquisition and preliminary transmission as well as preprocessing of input data. The ARM is responsible for receiving the video and transmitting it to the FPGA. The FPGA is responsible for data calculation acceleration, separating the calculation load to the FPGA, and transmitting the processed data back to the server CPU. The ARM module includes a cortex-A9 processor, an AXI communication interface, a DDR storage and an SD card. The cortex-A9 processor is connected to the AXI bus transmission protocol through the AXI communication interface, and the cortex-A9 processor is also bidirectionally connected to the DDR storage and the SD card respectively; The VDMA data stream handling IP can read data from the ARM-side DDR storage and then output it in the form of AXI-Stream; The VTC timing control IP generates RGB video timing, is responsible for generating timing signals synchronized with display output, and works in conjunction with the VDMA data stream handling IP module; VID OUT IP converts Stream data into RGB timing and gives it to HDMI display IP; The HDMI display IP is responsible for converting the RGB image data of the video output into an output signal that complies with the HDMI protocol, and transmitting the original video and the processed video data to an external display device for comparison; The custom image processing module is a free expansion module, which is encapsulated into an AXI4 standard interface and has six main input and output interfaces, including three AXI4 interfaces and three control interfaces.

2. The three-party collaborative video real-time transmission and processing heterogeneous system as claimed in claim 1, characterized in that: The custom image processing module is designed with three replaceable modules, including a SOBEL edge detection module, a Harris corner detection module, and an image classification module; The SOBEL edge detection module is used to calculate the edge features of the image, the Harris corner detection module is used to extract image feature points, and the image classification module is used for image classification based on CNN LeNet. The three modules all realize data interaction and control through six main interfaces.

3. The three-party collaborative video real-time transmission and processing heterogeneous system as claimed in claim 2, characterized in that: The three AXI4 interfaces of the SOBEL edge detection module are s_axi_AXILiteS, INPUT_STREAM, and OUTPUT_STREAM, and the three control interfaces are ap_clk, ap_rst_n, and interrupt; The s_axi_AXILiteS interface is responsible for register configuration communication with the host system for IP core initialization, parameter setting, and status reading; The INPUT_STREAM interface accepts video stream data and supports the AXI4-Stream protocol. It is used to receive raw image data or data output from the previous processing stage. The OUTPUT_STREAM interface outputs the processed edge detection image data and supports the AXI4-Stream protocol, which is convenient for connecting to subsequent processing modules or display interfaces; The ap_clk interface is responsible for the system clock signal and drives the logic operation inside the IP core; The ap_rst_n interface is responsible for the global reset signal, which is valid at low level and is used to initialize all internal states of the IP core; The interrupt interface is responsible for the interrupt signal, indicating the completion status of the current processing task.

4. The three-party collaborative video real-time transmission and processing heterogeneous system as claimed in claim 2, characterized in that: The three AXI4 interfaces of the Harris corner detection module are s_axi_control, img_inp, and img_out, and the three control interfaces are ap_clk, ap_rst_n, and interrupt; The s_axi_control interface is based on the AXI4-Lite protocol and is used to communicate with the host processor. It is responsible for the parameter configuration and operation status monitoring of the IP core. The img_inp interface is based on the AXI4-Stream protocol and is used to receive input image data streams; The img_out interface uses the AXI4-Stream protocol to output the image data stream containing the corner detection results; The ap_clk interface represents the system global clock signal, which drives all logical operations inside the IP core; The ap_rst_n interface represents the global reset signal, which is valid at low level and is used to initialize all registers and internal states of the IP core during system startup or abnormal conditions. The interrupt interface represents an interrupt signal. A high level output indicates that the current processing task is completed or an exception occurs.

5. The three-party collaborative video real-time transmission and processing heterogeneous system as claimed in claim 2, characterized in that: The three AXI4 interfaces of the picture classification module are s_axi_CTRL_BUS, input_r_PORTA, and output_r_PORTA, and the three control interfaces are ap_clk, ap_rst_n, and interrupt respectively; The s_axi_CTRL_BUS interface is based on the AXI4-Lite protocol and is used to interact with the host system to implement the configuration and operation status management of the IP core; The input_r_PORTA interface is based on the AXI4 protocol and is used to receive input image data; The output_r_PORTA interface is used to output classification results, including the classification labels and confidence score data of the image, and supports direct integration with downstream processing modules.

6. The three-party collaborative video real-time transmission and processing heterogeneous system as claimed in claim 1, characterized in that: The ARM end uses the LWIP interface as an IP protocol stack and receives the video stream data sent by the server CPU end through the UDP protocol.

7. The three-party collaborative video real-time transmission and processing heterogeneous system as claimed in claim 1, characterized in that: The FPGA ends are all externally connected to a DDR memory chip for temporarily storing video stream data.

Images