Multi-interface parallel ultra-high-definition image real-time processing and control device and method

Abstract

The invention provides a multi-interface parallel ultra-high-definition image real-time processing and control device and method, and belongs to the technical field of image processing, and the device comprises a calculation processing module, a video input interface module, a video output interface module, an industrial automation control interface module and a data storage module. The calculation processing module is connected with the video input interface module, the user control and state indication module, the video output interface module, the industrial automation control interface module and the data storage module; the calculation processing module comprises an FPGA (Field Programmable Gate Array) processing unit and is used for receiving and transmitting data through various interfaces, executing temperature control, receiving multiple paths of video streams, carrying out parallel pipeline processing, outputting a processing result in real time and controlling various automatic mobile platforms; and the video input interface module comprises four HDMI (High Definition Multimedia Interface) 2.0 input interfaces and four Ethernet input interfaces. According to the invention, multiple inputs are compatible, parallel decoding and multi-core processor cooperative processing are realized, and the processing performance is improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a multi-interface parallel ultra-high-definition image real-time processing and control device and method. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of 4K / 8K ultra-high-definition video, real-time AI analysis, edge computing, and low-latency networks, fields such as video surveillance, intelligent security, autonomous driving, and industrial inspection are placing higher demands on the real-time performance, stability, and multi-channel access capabilities of ultra-high-definition image acquisition and processing systems. These application scenarios typically require image processing equipment to simultaneously access and process multiple ultra-high-definition video streams, enabling real-time decoding, transcoding, storage, and intelligent analysis of video data, and parallel control of automated mobile platforms (such as unmanned vehicles, industrial robots, and drones) based on the processing results.

[0004] However, existing technical solutions still have significant shortcomings in meeting the above requirements. Ultra-high-definition image processing systems mainly include three architectures: The first is a PC architecture based on an industrial control computer with an external image acquisition card. This solution relies on CPU / GPU for image processing. Especially when executing deep learning algorithms, its serial processing mechanism is difficult to achieve true parallel pipeline operation, resulting in latency of up to hundreds of milliseconds, which cannot meet the real-time processing requirements of multi-channel 4K / 8K video streams. The second is a traditional embedded NVR device with a single CPU architecture. This type of system is limited by the performance of a single processor. Although some modules can be expanded through the VPX bus, it faces backplane bandwidth limitations and power consumption bottlenecks. When the number of connected high-definition cameras increases or the resolution is increased to 4K or above, the amount of computation and data throughput expands rapidly, leading to a sharp decline in system performance. The third is a development board solution based on a microcontroller or DSP. Although this solution is lower in cost, it suffers from a shortage of high-speed interfaces, difficulty in expansion, weak parallel processing capabilities, and often requires downsampling for high-speed video data, thus losing the significance of high-speed acquisition. It also has low integration. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention proposes a multi-interface parallel ultra-high-definition image real-time processing and control device and method. It provides an ultra-high-definition image processing device with high integration, supporting multiple interface inputs such as HDMI / Ethernet, based on FPGA hardware parallel decoding and multi-core processor collaborative processing, and with real-time video output and automated platform control capabilities, so as to meet the comprehensive requirements of modern ultra-high-definition image processing systems for high performance, low latency, high compatibility and high reliability.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention discloses a multi-interface parallel ultra-high-definition image real-time processing and control device, including a computing processing module, a video input interface module, a user control and status indication module, a video output interface module, an industrial automation control interface module, and a data storage module; the computing processing module is connected to the video input interface module, the user control and status indication module, the video output interface module, the industrial automation control interface module, and the data storage module; The computing and processing module includes an FPGA processing unit, which is used to receive and transmit data through various interfaces, perform temperature control, receive multiple video streams and perform parallel pipeline processing, output processing results in real time, and control various automated mobile platforms. The video input interface module includes four HDMI 2.0 input ports and four Ethernet input ports; The user control and status indication module includes buttons and an LED display unit; The video output interface module includes an Ethernet output interface, a Wi-Fi interface, and a PCIe interface; The industrial automation control interface module includes a network bus interface; The data storage module includes DDR4 SDRAM memory, SATA interface, and serial peripheral interface flash memory.

[0007] Secondly, this invention discloses a multi-interface parallel ultra-high-definition image real-time processing and control method, comprising: The device starts up, the external power supply is connected and initially supplies power, and the microcontroller initializes and controls the power sequence. Once the power supply voltage of all DC-DC converters reaches the specified level, the computing module loads the configuration information from the serial peripheral interface flash memory. The computational processing module receives video stream data from the video input interface module in parallel and processes the input video data stream in a parallel pipeline mode. The processed video stream is transmitted outward through the video output interface module and connected to various automated mobile platforms through the industrial automation control interface module.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Multi-interface centralized design: The single board integrates 8 video input interfaces (4 HDMI 2.0 + 4 Gigabit Ethernet) and PCIe, CAN, SATA, Wi-Fi and other interfaces, which can be compatible with various types of camera equipment and automation platforms without the need for external conversion modules, significantly reducing system complexity.

[0009] (2) FPGA hardware-level parallel processing capability: Based on FPGA, parallel-pipeline processing of 8 4K video streams is implemented. Each channel runs independently and the processing speed is not lower than the input speed, avoiding frame loss and the latency is as low as milliseconds, meeting the real-time processing requirements of ultra-high-definition video.

[0010] (3) Intelligent power supply and thermal management: The microcontroller controls 12 DC-DC converters in strict timing through the I²C bus, integrates temperature sensors to monitor the equipment temperature in real time, and cooperates with LED status indicators to realize power health monitoring and dynamic thermal protection, thereby improving the reliability of industrial-grade operation.

[0011] (4) Flexible dual-mode deployment: The device can operate as an independent system via USB / Wi-Fi interface, or it can work as an additional board for PC via PCIe interface, taking into account both the convenience of on-site deployment and the flexibility of development and debugging, and adapting to diverse application scenarios.

[0012] (5) Deep integration of video processing and motion control: While the FPGA processes the video stream in parallel, it directly controls the actuators and motors of platforms such as unmanned vehicles and robotic arms through the CAN-A / CAN-B bus, realizing low-latency closed-loop collaboration of video perception-decision-control without the need for an external control unit.

[0013] (6) Combining local storage and remote transmission: It supports recording 4 video streams to HDD / SSD via SATA interface, and can output the processing results in real time via Ethernet / Wi-Fi / PCIe interface to meet the dual needs of data archiving and real-time analysis.

[0014] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0016] Figure 1 This is a schematic diagram of the structure of the multi-interface parallel ultra-high-definition image real-time processing and control device described in Embodiment 1 of the present invention.

[0017] Figure 2 This is a flowchart of the FPGA operation described in Embodiment 1 of the present invention. Detailed Implementation

[0018] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0020] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0021] The existing technical solutions for ultra-high-definition image acquisition, transmission, and processing systems have the following problems: 1. Insufficient real-time parallel processing capability for multiple ultra-high-definition video streams; Traditional PC architectures rely on CPUs / GPUs for image processing. Their serial processing mechanism struggles to achieve true parallel pipelined operation, especially when executing deep learning algorithms, where latency can reach hundreds of milliseconds, failing to meet the real-time processing requirements of multi-channel 4K / 8K video streams. While microcontroller or DSP solutions are lower in cost, their parallel processing capabilities are weak, often requiring downsampling for high-speed video data, leading to frame loss. Existing NVR devices, although employing multi-core processors, lack flexible dynamic task allocation mechanisms, failing to adjust load balancing across subsystems in real-time based on workload, thus limiting overall throughput. There is an urgent need for a hardware-level parallel architecture capable of simultaneously processing eight or more ultra-high-definition video streams, with each stream's processing speed no less than the input speed.

[0022] 2. Poor compatibility and insufficient expandability of interfaces for various camera types; Existing equipment typically supports only a single video source input (such as a pure HDMI or pure network interface), and cannot simultaneously support cameras with multiple interfaces such as HDMI 2.0, Gigabit Ethernet, and MIPI-CSI. When the system needs to be expanded to connect more sensors or actuators, the lack of standardized scalable interfaces (such as FMC, PMOD) and high-speed interconnect buses (such as PCIe) leads to complex hardware reconfiguration and high costs. Although some solutions provide VPX expansion slots, their complex backplane designs are not conducive to miniaturized deployment and cannot meet the flexible and ever-changing interface requirements of industrial environments.

[0023] 3. Low system integration, making it difficult to achieve miniaturization and industrial-grade deployment; PC-based architecture solutions require external independent acquisition cards, storage arrays, control units, and other modules, resulting in bulky systems, high power consumption, and high costs, making them unsuitable for mobile platforms or space-constrained environments. Discrete embedded solutions require additional auxiliary units for power management, clock distribution, and temperature monitoring, lacking integrated design, leading to scattered PCB layouts, complex wiring, and insufficient reliability and anti-interference capabilities. Cable connections between multiple modules increase potential points of failure, making maintenance difficult and failing to meet the requirements for stable 24 / 7 operation in industrial applications.

[0024] 4. The system lacks stability and reliability, and is deficient in an integrated monitoring mechanism; Parallel processing of multiple 4K / 8K video streams significantly increases system power consumption, leading to severe overheating of core components such as FPGAs and multi-core processors. Existing solutions often employ fixed-timing power management, failing to dynamically adjust DC-DC converter output based on load and lacking health monitoring of critical modules. Without real-time temperature monitoring and dynamic power management, overheating can easily cause frequency throttling or system crashes, compromising reliability in multi-video concurrency scenarios.

[0025] 5. Difficulty in coordinating video processing with automated platform control; Existing image processing equipment typically focuses only on video encoding / decoding and storage, lacking real-time control interfaces for automated mobile platforms (such as unmanned vehicles and robotic arms). External control units are required to achieve closed-loop control based on video analysis results, increasing system latency and complexity. There is an urgent need to deeply integrate industrial control interfaces such as CAN bus with video processing pipelines to achieve low-latency collaboration between video perception and motion control.

[0026] This invention provides a multi-interface parallel ultra-high-definition image real-time processing and control device and method, which solves the above-mentioned technical problems through a highly integrated single-board architecture and hardware-level parallel processing mechanism: 1. FPGA hardware parallel pipeline processing architecture; Using an FPGA as the core computing unit and employing a parallel pipeline design, this architecture simultaneously receives eight 4K video streams from four HDMI 2.0 interfaces and four Gigabit Ethernet interfaces. An independent video processing pipeline is implemented within the FPGA, with each video stream undergoing real-time hardware processing at the input speed. Temporary video data is stored in DDR4 SDRAM to ensure zero frame loss. Compared to traditional CPU / GPU serial processing, this architecture reduces processing latency to the millisecond level, meeting the real-time requirements of ultra-high-definition video.

[0027] 2. Integrated multi-interface compatibility and hybrid expansion design; The single-board interface integrates multiple interfaces including HDMI 2.0, Gigabit Ethernet, SATA, PCIe, USB, CAN, and Wi-Fi. It supports HDMI and Ethernet input from digital video cameras, as well as PC connection (PCIe / USB). Through its integrated interface design, it is compatible with various types of camera devices without the need for external conversion modules. A standard PCIe expansion interface is reserved for flexible connection to other functional cards, meeting future interface expansion needs and solving the problem of poor expandability in traditional solutions.

[0028] 3. Highly integrated single-board system architecture; All functional modules are integrated onto a single PCB board, including FPGA, memory (DDR4 SDRAM, SPI Flash, SATA hard drive interface), power management (12-channel DC-DC converter), microcontroller, temperature sensor, and clock generator. By optimizing the PCB layout and routing, the board size and weight are reduced, allowing for direct embedding into mobile robots, drones, and other devices, achieving industrial-grade miniaturized deployment and avoiding the reliability risks associated with multi-module cable connections.

[0029] 4. Intelligent power management and thermal protection mechanism; A microcontroller monitors the output voltage of all DC-DC converters in real time via the I2C bus and sequentially powers on 12 power supplies according to a predetermined timing sequence, ensuring that the FPGA power supply strictly meets the timing requirements. An integrated temperature sensor monitors the FPGA and overall system temperature in real time and feeds the data back to the FPGA for dynamic frequency adjustment and heat dissipation control. This integrated monitoring mechanism effectively avoids overheating-induced frequency throttling and system crashes, ensuring long-term stable operation for industrial applications.

[0030] 5. Deep integration of video processing and CAN bus control; While processing eight video streams in parallel, the FPGA directly connects to the actuators, motors, and sensors of the automated mobile platform via CAN-A and CAN-B interfaces. Based on video analysis results, the FPGA can generate control commands in real time and send them out in parallel via the CAN bus, achieving closed-loop coordination of video perception, decision-making, and control. In remote operation mode, it supports the transmission of video streams, telemetry data, and control signals via Wi-Fi interface, meeting the remote control needs of scenarios such as unmanned vehicles and drones.

[0031] 6. Flexible independent / collaborative work modes; The device can operate as a standalone system (configured and monitored via USB / Wi-Fi interface) or as an add-on card to a PC via PCIe interface. In PC mode, high-speed video streaming and FPGA configuration monitoring are achieved through the PCIe bus; in standalone mode, it supports recording up to four video streams from SATA hard drives and outputting the processed results via Ethernet / Wi-Fi. This dual-mode design balances ease of development and debugging with flexibility in field deployment.

[0032] Example 1 In one or more embodiments, a multi-interface parallel ultra-high-definition image real-time processing and control device is disclosed, such as... Figure 1 As shown, it includes a computing processing module, a video input interface module, a user control and status indication module, a video output interface module, an industrial automation control interface module, and a data storage module; the computing processing module is connected to the video input interface module, the user control and status indication module, the video output interface module, the industrial automation control interface module, and the data storage module; The computing and processing module includes an FPGA processing unit 1, which is the main computer of the device. It is used to receive and transmit data through various interfaces, perform temperature control, receive multiple video streams and perform parallel pipeline processing, output processing results in real time, and control various automated mobile platforms, including cars, handcarts, robots, drones, etc. The video input interface module includes four HDMI 2.0 input interfaces 2-5 and four Ethernet input interfaces 6-9. The HDMI 2.0 input interfaces are used to connect up to four 4K resolution digital HDMI cameras and transmit their generated video streams to the FPGA for subsequent processing. The Ethernet input interfaces are used to connect up to four 4K resolution digital Ethernet cameras and transmit their video streams to the FPGA for processing. Together, they enable compatible access to multiple types of camera devices.

[0033] The user control and status indication module includes a button 10 and an LED display unit 12. The button 10 is connected to the FPGA input terminal and is used to switch the FPGA working mode. The working mode can be set before the device starts up and during operation. The LED display unit 12 is connected to the FPGA output terminal and is used to indicate the FPGA working mode, register status and various stages of video stream processing.

[0034] The video output interface module includes an Ethernet output interface 11, a Wi-Fi interface 23, and a PCIe interface 28. The Ethernet output interface 11 is connected to the FPGA input / output terminal for transmitting processed data and generated video streams. The Wi-Fi interface 23 is connected to the FPGA input / output terminal for transmitting / receiving video streams, telemetry data, and control signals in remote operation mode. The PCIe interface 28 is connected to the FPGA input / output terminal for connecting the device to a computer (PC) and operating it as a component of the PC. Through this interface, the FPGA can be configured and monitored, and video streams and power supply voltage can be transmitted.

[0035] The industrial automation control interface module includes a network bus interface; the controller area network bus interface includes CAN-A 13 and CAN-B 14. The two CAN bus interfaces are connected to the FPGA input / output terminals to connect actuators, motors, and sensors of various automated mobile platforms (cars, trolleys, robots, drones, etc.), enabling the FPGA to control these platforms based on video stream processing results, including driving actuators and motors and receiving various telemetry information.

[0036] The data storage module includes DDR4 SDRAM memory 19-22, SATA interfaces 15-18, and Serial Peripheral Interface (SPI) flash memory 29. The DDR4 SDRAM is connected to the FPGA input / output terminals to store large video stream data arrays generated during FPGA processing; the SATA interface is connected to the FPGA input / output terminals to connect a hard disk drive (HDD) or solid-state drive (SSD) to store various information (streaming video processing function parameters and settings, interface configuration, various video streams); the SPI flash memory is connected to the FPGA input / output terminals to save configuration programs and various data when power is off.

[0037] Preferably, the device also includes a system monitoring and power management module, which consists of a temperature sensor (TS) 24, a microcontroller 26, and DC-DC converters (DC-DC) 32-45. The TS is connected to the FPGA input / output terminals and to the FPGA via an I²C interface to monitor the temperature of the FPGA and the entire device. The microcontroller is connected to the FPGA via an I²C interface and is also connected to the output and enable input terminals of the DC-DC converter for power monitoring and control of the DC-DC converter. The DC-DC converter provides a stable power supply with preset voltage and maximum current to all integrated circuit devices, which is turned on sequentially by the microcontroller according to the technically required timing sequence.

[0038] Preferably, the device also includes a clock generation module, which consists of a clock generator (CLK) 25. The clock generator is connected to the FPGA input and is responsible for generating all the frequency signals required for FPGA operation, providing a unified clock reference for the entire system.

[0039] Preferably, the device also includes a configuration and debugging interface module, which includes an SWD interface 27, a JTAG interface 31, and a USB interface 30. The SWD interface is connected to the microcontroller's input / output terminals and can be programmed via a personal computer; the JTAG interface is connected to the FPGA's input / output terminals and is used to configure the FPGA and write the configuration program to the SPI flash memory; the USB interface is connected to the FPGA's input / output terminals and is used to connect the FPGA to a PC for control and monitoring operations when the device is not part of a PC.

[0040] Furthermore, the FPGA processing unit enables efficient parallel processing of 8 video streams, and its workflow is as follows: Figure 2 As shown, it includes: (1) After the FPGA is started, initialize each component.

[0041] (2) Receive commands via USB / PCIe / Wi-Fi, update the status of each component according to the commands, and then transmit the observation data to the connected device in real time via USB / PCIe / Wi-Fi or save it as a log for troubleshooting the stability of the device or each unit.

[0042] (3) Parallel detection of 8 video interfaces. The HDMI interface directly receives the source image, and the Ethernet interface receives the video stream and decodes the corresponding image using a decoder. Then, the timestamps are used to align multiple video frames. Together with the feature extraction unit, model inference unit (including correlation calculation, 3D reconstruction, object detection, etc.), and post-processing unit (including motion rule estimation, state update, image encoding, etc.), a pipelined parallel module is formed to process images / image features at different times in sequence. Finally, the results are transmitted to the designated terminal via the PCIe / Wi-Fi / SATA / Ethernet interface.

[0043] (4) The CAN interface is responsible for receiving commands from control terminals such as robots, generating corresponding control instructions based on the image processing results, and sending them to the designated controlled terminal via the PCIe / Wi-Fi / SATA / CAN interface.

[0044] (5) Steps (2), (3), and (4) run in parallel. In (3), modules located under the same double horizontal line run serially, while modules under different double horizontal lines run in parallel. New or existing computing modules can be added or removed.

[0045] The following alternative solutions exist for the above technical solutions, but they face obvious technical drawbacks: (1) Development board solutions based on traditional microcontrollers or DSPs have low integration, weak large-scale data processing capabilities, and a shortage of high-speed interfaces and difficulties in expansion; they lack parallel processing capabilities, and high-speed video needs to be downsampled, resulting in frame loss and loss of the meaning of high-speed acquisition; they cannot be compatible with multiple types of cameras such as HDMI, Ethernet, and MIPI to be connected at the same time, and the system construction is complex and has poor reliability.

[0046] (2) The solution based on the industrial computer with an external high-speed data transmission module is that the industrial computer is bulky and not suitable for mobile platforms or space-constrained scenarios; it is expensive and not suitable for large-scale deployment; it is difficult to expand the high-speed interface, and the connection of multiple modules with cables increases the number of fault points; it cannot achieve deep integration of single-board level video processing and CAN bus control, and the system latency is high.

[0047] (3) Dedicated solutions based on dedicated video codec ASICs or SoCs have long development cycles, poor flexibility, and are difficult to adapt to customized functional requirements; the interface types are fixed and cannot be flexibly expanded to standard interfaces such as FMC and PMOD; lacking hardware programmability, it is impossible to dynamically adjust task allocation and power management according to the load, and it is difficult to achieve low-latency closed-loop collaboration of video perception and motion control.

[0048] Therefore, the present invention is superior to existing alternatives in terms of real-time performance, stability, scalability and integration, and has significant technical advantages and application prospects.

[0049] Example 2 In one or more embodiments, a multi-interface parallel ultra-high-definition image real-time processing and control method is disclosed, utilizing the aforementioned ultra-high-definition image real-time processing and control device, including the following steps: Step S101: Start the equipment.

[0050] Step S101-1: External power supply connection and initial power supply; The device is powered by an external 12V DC power supply or via PCIe interface 28. The first-stage DC-DC converter 32 generates a 3.3V voltage, which is output to the microcontroller 26.

[0051] Step S101-2: Microcontroller initialization and power timing control; Once the 3.3V power supply reaches the required level, the microcontroller 26 starts and executes the program pre-written via the SWD interface 27. Following the required timing sequence, the microcontroller 26 sequentially turns on each of the DC-DC converters 33-45, continuously checking the power supply voltage provided by each converter.

[0052] Step S101-3: FPGA configuration loading; Once the power supply voltages of all DC-DC converters 33-45 reach the specified levels, FPGA1 loads the configuration information from the SPI flash memory 29 (which is pre-written into the SPI flash memory 29 via the JTAG interface 31), completes system initialization, and enters the working state.

[0053] Step S102: Equipment operation.

[0054] Step S102-1: Parallel reception of multiple video streams; FPGA 1 receives video stream data from 1 to 8 cameras (up to 4 HDMI cameras and 4 Ethernet cameras) in parallel through HDMI 2.0 input interfaces 2-5 and Ethernet input interfaces 6-9, and configures and monitors the cameras through the same interfaces.

[0055] Step S102-2: Hardware-level parallel pipeline processing; FPGA 1 processes the input video data streams in a parallel pipeline mode, with each video stream processed in an independent pipeline at its input speed. Temporary video data generated during processing is stored in DDR4 SDRAM 19-22, ensuring real-time processing of multiple 4K video streams without frame drops.

[0056] Step S102-3: Output multi-channel results; The processed video stream is transmitted externally via Ethernet output interface 11, PCIe interface 28, or Wi-Fi interface 23. Through SATA interfaces 15-18, FPGA1 can record up to 4 video streams to an external HDD or SSD for storing processing parameters, interface configurations, and various video stream data.

[0057] Step S102-4: Real-time control by the automated platform; FPGA1 connects directly to actuators, motors, and sensors of various automated mobile platforms (cars, trolleys, robots, drones, etc.) via CAN-A interface 13 and CAN-B interface 14. It drives actuators and motors in real time based on video stream processing results and receives telemetry information to achieve closed-loop coordination of video perception and motion control.

[0058] Preferably, this device also features human-computer interaction and status monitoring; Button 10 and USB interface 30 are used to switch the operating mode of FPGA 1. LED 12 indicates the operating mode, register status, and various stages of video stream processing of FPGA 1 in real time. Temperature sensor 24 continuously measures the operating temperature of the device and transmits the results to FPGA 1 via I²C interface. Clock generator 25 provides all the necessary clock pulse signals to FPGA 1.

[0059] Preferably, this device supports dual-mode operation; it can operate independently or function as part of a PC via the PCIe interface 28. In independent mode, configuration and monitoring are performed via USB / Wi-Fi interfaces; in PC mode, high-speed video transmission and FPGA configuration and monitoring are achieved via the PCIe bus, meeting the deployment requirements of different application scenarios.

[0060] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0061] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-interface parallel ultra-high-definition image real-time processing and control device, characterized in that, It includes a computing and processing module, a video input interface module, a user control and status indication module, a video output interface module, an industrial automation control interface module, and a data storage module; the computing and processing module is connected to the video input interface module, the user control and status indication module, the video output interface module, the industrial automation control interface module, and the data storage module; The computing and processing module includes an FPGA processing unit, which is used to receive and transmit data through various interfaces, perform temperature control, receive multiple video streams and perform parallel pipeline processing, output processing results in real time, and control various automated mobile platforms. The video input interface module includes four HDMI 2.0 input ports and four Ethernet input ports; The user control and status indication module includes buttons and an LED display unit; The video output interface module includes an Ethernet output interface, a Wi-Fi interface, and a PCIe interface; The industrial automation control interface module includes a network bus interface; The data storage module includes DDR4 SDRAM memory, SATA interface, and serial peripheral interface flash memory.

2. The multi-interface parallel ultra-high-definition image real-time processing and control device as described in claim 1, characterized in that, It also includes a system monitoring and power management module, which consists of a temperature sensor, a microcontroller, and a DC-DC converter. The temperature sensor is connected to the input / output terminals of the FPGA processing unit and is connected to the FPGA processing unit via an I²C interface to monitor the temperature of the FPGA processing unit and the entire device. The microcontroller is connected to the FPGA processing unit via an I²C interface and is also connected to the output terminal and enable input terminal of the DC-DC converter for power monitoring and control of the DC-DC converter. The DC-DC converter provides a stable power supply with preset voltage and maximum current to all integrated circuit devices, which is turned on sequentially by the microcontroller according to the technical requirements.

3. The multi-interface parallel ultra-high-definition image real-time processing and control device as described in claim 1, characterized in that, It also includes a clock generation module, which consists of a clock generator; the clock generator is connected to the input of the FPGA processing unit and is responsible for generating all the frequency signals required for FPGA operation, providing a unified clock reference for the entire device.

4. The multi-interface parallel ultra-high-definition image real-time processing and control device as described in claim 1, characterized in that, It also includes a configuration and debugging interface module, which includes an SWD interface, a JTAG interface and a USB interface; the SWD interface is connected to the microcontroller input / output terminal; the JTAG interface is connected to the FPGA input / output terminal and is used to configure the FPGA and write the configuration program to the SPI flash memory. The USB interface connects to the FPGA's input / output terminals, allowing the FPGA to be connected to a PC for control and monitoring when the device is not part of a PC.

5. The multi-interface parallel ultra-high-definition image real-time processing and control device as described in claim 1, characterized in that, The workflow of the FPGA processing unit includes: (1) After the FPGA is started, initialize each component; (2) Receive commands via USB / PCIe / Wi-Fi, update the status of each component according to the commands, and then transmit the observation data to the connected device in real time via USB / PCIe / Wi-Fi; (3) Parallel detection of 8 video interfaces; HDMI directly receives the source image, Ethernet interface receives the video stream and decodes the corresponding image using the decoder, and then uses timestamps to align multiple video frames; it forms a pipeline parallel module with the feature extraction unit, model inference unit and post-processing unit to process images or image features at different times in sequence; the results are transmitted to the designated terminal via PCIe, Wi-Fi, SATA or Ethernet interface; (4) The CAN interface is responsible for receiving commands from the control terminal, generating corresponding control instructions based on the image processing results, and sending them to the designated controlled terminal via PCIe, Wi-Fi, SATA or CAN interface; (5) Steps (2), (3), and (4) are run in parallel.

6. The multi-interface parallel ultra-high-definition image real-time processing and control device as described in claim 1, characterized in that, The HDMI 2.0 input interface is used to connect up to 4 4K resolution digital HDMI cameras, transmitting their generated video streams to the FPGA for further processing; the Ethernet input interface is used to connect up to 4 4K resolution digital Ethernet cameras, also transmitting their video streams to the FPGA for processing. Together, they enable compatible access to multiple types of camera devices.

7. The multi-interface parallel ultra-high-definition image real-time processing and control device as described in claim 1, characterized in that, The button is connected to the input of the FPGA processing unit and is used to switch the FPGA's operating mode. The operating mode can be set before the device starts up and during operation. The LED display unit is connected to the FPGA output and is used to indicate the FPGA's operating mode, register status, and various stages of video stream processing.

8. The multi-interface parallel ultra-high-definition image real-time processing and control device as described in claim 1, characterized in that, The Ethernet output interface is connected to the input / output terminals of the FPGA processing unit for transmitting processed data and generated video streams; The Wi-Fi interface is connected to the input / output terminals of the FPGA processing unit and is used to transmit and receive video streams, telemetry data and control signals in remote operation mode. The PCIe interface connects to the input / output terminals of the FPGA processing unit, and is used to connect the device to a PC and operate as a component of it. The PCIe interface is used to configure and monitor the FPGA, transmit video streams, and supply power voltage.

9. The multi-interface parallel ultra-high-definition image real-time processing and control device as described in claim 1, characterized in that, The network bus interface includes two CAN bus interfaces connected to the input / output terminals of the FPGA processing unit, used to connect actuators, motors and sensors of various automated mobile platforms.

10. A multi-interface parallel ultra-high-definition image real-time processing and control method, characterized in that, include: The device starts up, the external power supply is connected and initially supplies power, and the microcontroller initializes and controls the power sequence. Once the power supply voltage of all DC-DC converters reaches the specified level, the computing module loads the configuration information from the serial peripheral interface flash memory. The computational processing module receives video stream data from the video input interface module in parallel and processes the input video data stream in a parallel pipeline mode. The processed video stream is transmitted outward through the video output interface module and connected to various automated mobile platforms through the industrial automation control interface module.

Images