Intelligent optoelectronic device heterogeneous computing system and video transmission and control method thereof

Abstract

The application discloses an intelligent optoelectronic device heterogeneous computing system and a video transmission and control method thereof. The heterogeneous computing system comprises a main control processor and a programmable logic device, and a video transmission link and a control transmission link are arranged between the main control processor and the programmable logic device. The video transmission link comprises a first video transmission channel with a high rate and a second video transmission channel with a low rate, and both are special for video; the control transmission link is independent of the video transmission link and is special for control instructions. When the first video transmission channel is abnormal, the system is automatically switched to the second video transmission channel and the transmission is performed at a reduced resolution and frame rate; the nanosecond-level hard trigger synchronous acquisition can be realized through a control transmission link chip selection pin; and the software configuration of multiple working modes and multiple video sub-channels is supported. The application solves the problems of single video link, insufficient real-time control and lack of flexible configuration in the prior art, and can be applied to airborne optoelectronic pods, unmanned aerial vehicle / vehicle vision systems and other high-reliability intelligent optoelectronic devices.

Description

Technical Field

[0001] This invention relates to the field of embedded heterogeneous computing system technology, specifically to a multiprocessor heterogeneous computing system architecture for intelligent optoelectronic devices (such as vehicle-mounted optoelectronic turrets, high-performance multi-light fusion systems, multimodal optoelectronic pods, intelligent vision observation systems, etc.), and a video data transmission and real-time control method based on this architecture. Background Technology

[0002] As intelligent optoelectronic devices rapidly evolve towards greater intelligence, miniaturization, and high reliability, the demands on computing power, data transmission bandwidth, real-time control precision, and time synchronization accuracy are increasing dramatically. A single ARM processor or a single FPGA is insufficient to meet the multiple requirements of real-time multi-channel video acquisition, AI inference acceleration, industrial-grade real-time control, and multi-sensor fusion. The ARM+FPGA heterogeneous computing architecture, combining the flexible control capabilities of ARM with the parallel data processing capabilities of FPGA, has become the mainstream hardware solution for intelligent optoelectronic devices.

[0003] In the prior art, patent CN119865570A discloses a multi-channel video transmission system and method based on FPGA and RK3588. The system includes an FPGA platform and an RK3588 platform, with the FPGA platform connected to the RK3588 platform. The FPGA platform includes a multi-mode LVDS decoding module, a data interface conversion module, a read / write cache component, DDR memory, and a PCIe bridge module. The RK3588 platform includes a PCIe controller, a video buffer pool, a video compression encoding module, and a data transmission processing module. The PCIe controller is connected to the PCIe bridge module, and multiple video streams are transmitted to the RK3588 for compression encoding and transmission via a single PCIe bus. This solution achieves the compression and transmission of multiple video streams, but it has the following technical defects:

[0004] Defect 1: The video transmission link is singular, lacking redundancy and degradation mechanisms, resulting in insufficient system reliability. In existing solutions, multiple video data acquired by the FPGA are typically transmitted to the ARM processor via a single PCIe bus. If the PCIe link fails due to loose physical connections, electromagnetic interference, temperature changes, or chip malfunction, the entire video acquisition system will be completely paralyzed. In high-reliability applications (such as airborne optoelectronic pods, unmanned vehicle vision systems, and security monitoring systems), this "single point of failure" is fatal. Although existing technologies (such as patent CN121193718A) disclose dual-channel redundancy schemes based on abnormal switching between primary and secondary transmission channels in the field of communication networks, these are applied to audio and video communication at the channel level of IP networks such as VoLTE / VoIP, representing redundancy at the network link level, rather than redundancy design for physical video transmission links between heterogeneous chips within a single board. Redundancy design for video transmission links in single-board-level heterogeneous computing systems remains a technological gap.

[0005] Defect 2: Control commands and video data share the same bus, compromising hard real-time performance. In existing ARM+FPGA systems, ARM configuration commands to the FPGA are typically transmitted via the PCIe or AXI bus. When multiple video streams fill the PCIe bandwidth, control commands must queue, making it impossible to meet nanosecond or microsecond-level hard real-time control requirements—such as synchronous exposure triggering of multiple video sensors or real-time control of servo mechanisms. While existing technologies involve optimization of the SPI interface communication between the ARM and FPGA, these are limited to efficiency improvements of the SPI communication protocol itself and do not address system-level architecture design that uses SPI as a dedicated control channel independent of the video transmission link, achieving physical isolation between the control and data planes. In existing technologies, video data and control commands share a single PCIe bus; when the video stream fills the bandwidth, control commands queue, failing to meet nanosecond-level hard real-time control requirements.

[0006] Defect 3: Lack of flexible configuration and adaptive bandwidth management capabilities for video channels. Different product models have different video transmission bandwidth requirements—high-end models require simultaneous transmission of multiple 4K video streams, while low-end models only require a single 1080p video stream. Existing solutions cannot achieve different video channel combinations through software configuration on the same hardware architecture, nor can they automatically execute degraded transmission strategies when video link bandwidth is insufficient or fails. Although existing technologies (such as patent CN115842900A) disclose methods for service degradation by reducing resolution and frame rate in wireless channel communication scenarios, their adjustment of encoding transmission parameters for network congestion and packet loss scenarios is not channel switching and degraded operation under hardware link failure.

[0007] Defect 4: Lack of a high-precision global clock synchronization and unified timestamp mechanism. In intelligent optoelectronic devices, the fusion processing of multi-source sensor data (multi-channel video, industrial control data, etc.) places extremely high demands on time synchronization accuracy. Although existing solutions involve clock synchronization between FPGA and ARM or sensor timestamp annotation, they typically only achieve single-level clock synchronization, lacking a complete synchronization chain from "external time synchronization → FPGA clock discipline → FPGA-ARM synchronization → sensor data timestamp annotation," making it difficult to meet the time synchronization requirements of multi-sensor fusion and system-level collaborative work.

[0008] In summary, existing ARM+FPGA heterogeneous computing systems, when applied to high-reliability intelligent optoelectronic devices, urgently need to address the following technical issues: a lack of redundancy in the single video transmission link; insufficient hard real-time performance due to the shared bus between control commands and video data; a lack of flexible configuration and adaptive bandwidth management capabilities for video channels; and a lack of a high-precision global clock synchronization mechanism. Summary of the Invention

[0009] This invention aims to provide a heterogeneous computing system for intelligent optoelectronic devices and its video transmission and control method, in order to solve the technical problems in the prior art, such as the lack of redundancy in the single video transmission link, insufficient hard real-time performance due to the shared bus between control commands and video data, lack of flexible configuration and adaptive bandwidth management capabilities for video channels, and lack of a high-precision global clock synchronization mechanism.

[0010] This invention provides a heterogeneous computing system for intelligent optoelectronic devices, including a main control processor and a programmable logic device. The main control processor is connected to a first memory; the programmable logic device is connected to a second memory. Optionally, the first memory is DDR4 4GB*2; the first non-volatile memory is eMMC 64GB and a TF expansion card; the second memory is DDR3 2GB*2, and the second non-volatile memory is FLASH 16MB.

[0011] The core innovations of the intelligent optoelectronic device heterogeneous computing system of this invention are as follows:

[0012] 1. This invention employs a dual-video transmission channel redundancy and degradation architecture. A video transmission link is established between the main control processor and the programmable logic device, comprising a first video transmission channel and a second video transmission channel. The data transmission rate of the first video transmission channel is higher than that of the second video transmission channel, and both channels are dedicated to video data. In a preferred embodiment, the first video transmission channel is a PCIe bus, and the second video transmission channel is a MIPI bus. The programmable logic device integrates a video stream distribution management module, a link status monitoring module, and a channel switching control module: the video stream distribution management module is connected to the two video transmission channels and allocates video data to the corresponding channels for transmission according to a preset link strategy; the link status monitoring module monitors the link status of the first video transmission channel in real time (including physical layer activity status and bandwidth utilization); the channel switching control module switches the video data to the second video transmission channel for transmission when the first video transmission channel is abnormal. The video stream distribution management module supports three working modes: a first working mode that only enables the first video transmission channel, a second working mode that only enables the second video transmission channel, and a third working mode that enables both channels simultaneously. Furthermore, the second video transmission channel can be configured as one or more MIPI buses. Its number and bit width are flexibly set based on the MIPI interface resources supported by the main control processor and the product's bandwidth requirements, through configuration commands issued by the control transmission link. The video stream distribution management module then distributes video data to each MIPI sub-channel in either MIPI-only or dual-channel mode. When the first video transmission channel malfunctions, the channel switching control module executes a degraded transmission strategy—switching the video data to the second video transmission channel and transmitting it at a reduced resolution and / or frame rate to adapt to the bandwidth limitations of the second video transmission channel. To ensure no frame loss during the switching process, the programmable logic device also integrates a double-buffered queue for video data, maintaining continuous writing and reading of video data during switching to achieve seamless switching.

[0013] 2. This invention employs physical-level forced isolation between the control plane and the data plane. The invention provides a control transmission link independent of the video transmission link, serving as a dedicated channel for control commands between the main control processor and the programmable logic device (FPGA). In a preferred embodiment, the control transmission link is an SPI bus. Control commands include, but are not limited to: state machine configuration commands for the FPGA, synchronous trigger commands for multiple video inputs, reconstruction commands for the FPGA's internal parameters, and switching commands for three operating modes. Specifically, the main control processor sends a hard-triggered synchronization signal to the FPGA via the chip select (CS) signal pin of the SPI bus. In response to this hard-triggered synchronization signal, the FPGA performs synchronous acquisition operations on the multiple video input interfaces with the highest priority. The response delay of the hard-triggered synchronization signal is in the nanosecond range (no more than 5 clock cycles). The SPI control channel and the video channel are completely independent at both the physical layer (PCB routing) and the logic layer (FPGA internal bus). Congestion or failure of the video link will not affect the transmission of SPI control commands under any circumstances. As an optional implementation, the control transmission link can be any interface other than the SPI bus, such as the UART bus, I2C bus, general purpose input / output (GPIO) bus, or a standalone AXI bus, as long as it is independent of the video transmission link and serves as a dedicated channel for control instructions between the main control processor and the programmable logic device.

[0014] This invention employs a four-level global clock synchronization and unified timestamp mechanism. The front end of the programmable logic device (PLD) is connected to an external clock synchronization interface, supporting at least one of pulse-per-second (PPS), IEEE 1588 Precision Time Protocol (PTP), and / or B-code time synchronization. The local clock is calibrated via an external time signal. A clock synchronization link is established between the PLD and the main control processor to synchronize the calibrated local clock to the main control processor, enabling both to share a unified time base. The PLD integrates a global timestamp generation module, which appends a globally unified timestamp to each frame of sensor data based on the unified time base, achieving end-to-end time synchronization from the system level to the data level. Alternatively, the local clock can be calibrated by obtaining an external time signal from the local area network via the main control processor. The calibrated local clock is then synchronized to the PLD via the clock synchronization link between the main control processor and the PLD, allowing both to share a unified time base.

[0015] 4. This invention employs multi-source heterogeneous sensor front-end aggregation. The front-end of the programmable logic device is connected to multiple video input interfaces (including at least LVDS and / or MIPI interfaces) and multiple industrial control interfaces (including one or more of CAN, RS485, RS422, and UART), enabling unified access to multi-source video and industrial control data.

[0016] This invention also provides a video transmission and control method based on the above-mentioned heterogeneous computing system, comprising: S1, a programmable logic device receiving multiple video data streams through a front-end multi-channel video input interface and receiving industrial control data through a front-end industrial control interface; S2, real-time monitoring of the link status of a first video transmission channel; S3, when the first video transmission channel is normal, transmitting the multiple video data streams to the main control processor through the first video transmission channel; S4, when the first video transmission channel is abnormal, executing a degradation transmission strategy, switching the multiple video data streams to a second video transmission channel for transmission to the main control processor; S5, receiving control commands from the main control processor through a control transmission link independent of the two video transmission channels, and performing corresponding operations according to the control commands. Further, the method also includes: S6, the programmable logic device receiving an external timing signal through an external clock synchronization interface and calibrating its local clock; S7, synchronizing the calibrated local clock to the main control processor through a clock synchronization link; S8, based on the local clock, appending a globally unified timestamp to each frame of the multiple video data streams and the industrial control data.

[0017] Compared with prior art, the present invention has the following advantages:

[0018] (1) High reliability: The dual video channel redundancy design ensures that the system automatically degrades to operation when a single link fails, and video acquisition is not interrupted. The system reliability is improved from the "single point of failure" of a single link to system-level fault tolerance. (2) Hard real-time control: The SPI control channel and the video channel are completely isolated at the physical layer and logic layer. The transmission of control commands is not affected by the full load or failure of the video link. The hard trigger response reaches the nanosecond level.

[0019] (3) Flexible product configuration: The same hardware PCB can be configured with multiple working modes such as PCIe only, MIPI only or dual channels through FPGA configuration. The MIPI video channel can be configured as one or more sub-channels to cover the needs of different grades of products and realize "one board for multiple uses".

[0020] (4) Bandwidth Adaptive: When the PCIe link is congested, some video streams are automatically transferred to the MIPI channel to achieve dual-channel load balancing and effectively utilize all available bandwidth resources;

[0021] (5) Seamless video switching: The dual-buffered queue mechanism ensures that video data is not lost or interrupted during channel switching;

[0022] (6) High-precision time synchronization: The four-level global clock synchronization mechanism ensures that all sensor data in the system have a unified time reference, with timestamp accuracy reaching the nanosecond level, meeting the requirements of multi-sensor fusion and system-level collaborative work. Attached Figure Description

[0023] Figure 1 This is a block diagram of the overall architecture of the heterogeneous computing system of the present invention, showing the main control processor, the programmable logic device, and the first video transmission channel (PCIe), the second video transmission channel (MIPI), the control transmission link (SPI), and the clock synchronization link between them. It also shows the back-end output interface of the main control processor and the front-end interface and external clock synchronization interface of the programmable logic device.

[0024] Figure 2 This is a schematic diagram of the internal functional modules of the programmable logic device of the present invention, showing the connection relationship between the front-end interface receiving layer, the interface protocol parsing module, the multi-channel video synchronous acquisition module, the data stream classification module, the video data double buffer queue, the video stream distribution management module, the PCIe transmitting interface, the MIPI transmitting interface, the SPI control interface, the control command transparent transmission module, the link status monitoring module, the channel switching control module, and the global clock synchronization module.

[0025] Figure 3 This is a flowchart of the video transmission and control method of the present invention, showing steps S1 to S5 and their branch processes.

[0026] Figure 4 This is a timing diagram of the SPI hard trigger synchronization signal of the present invention, showing the timing relationship of the SPI chip select (CS) signal falling edge triggering the hard trigger synchronization signal, the FPGA responding with the highest priority and performing timestamp latching, frame capture enable, and timestamp appending.

[0027] Figure 5 This is a schematic diagram of the operation of the video data dual-buffered queue of the present invention, showing the seamless switching mechanism of alternating writing and reading of buffer A and buffer B, and the effect of channel switching commands in the inter-frame interval.

[0028] Figure 6 This is a block diagram illustrating the principle of the four-level global clock synchronization mechanism of the present invention. It shows the structure of the four-level synchronization link, which includes external time synchronization, clock discipline, FPGA and main control processor clock synchronization, and sensor data timestamp annotation, as well as the additional timestamps on each sensor data. Detailed Implementation

[0029] The present invention will be further illustrated below with reference to specific embodiments. These embodiments are for illustrative purposes only and do not constitute a limitation on the scope of protection of the present invention.

[0030] like Figure 1 As shown, this embodiment provides a heterogeneous computing system for intelligent optoelectronic devices, including a main control processor and a programmable logic device.

[0031] The main control processor uses an ARM architecture processor (specifically model RK3588), which is connected to a first memory (DDR4, 2GB×2) and a first non-volatile storage device (eMMC, 64GB, and a TF card expansion interface). The programmable logic device uses an FPGA (specifically model XC7A100T, belonging to the Artix-7 series), which is connected to a second memory (DDR3, 2GB×2) and a second non-volatile storage device (FLASH, 16MB).

[0032] Four independent physical links are deployed between the main control processor and the programmable logic device, as shown in Table 1 below:

[0033]

[0034] Table 1 Independent physical links between the main control processor and the programmable logic device.

[0035] As an optional implementation, the control transmission link can be any interface other than the SPI bus, such as the UART bus, I2C bus, general purpose input / output (GPIO) bus, or a standalone AXI bus, as long as it is independent of the video transmission link and serves as a dedicated channel for control commands.

[0036] In the PCB layout, PCIe differential pairs, MIPI differential pairs, and SPI single-ended signal lines belong to different signal layers and are physically isolated by ground plane partitioning to ensure that crosstalk between each link meets signal integrity requirements.

[0037] The programmable logic device (PLD) features multiple video input interfaces (two LVDS interfaces and one MIPI interface) and multiple industrial control interfaces (one CAN interface, one RS485 interface, two RS422 interfaces, and two UART interfaces). The PLD also includes a JTAG debugging interface and an external clock synchronization interface (supporting PPS and / or PTP time synchronization). The main control processor's rear interface includes a video output interface (HDMI interface), a network interface (100Mbps Ethernet port), and general-purpose expansion interfaces (USB 2.0 and / or USB 3.1).

[0038] In addition to the basic configuration (1×PCIe 2.0×4 + 1×MIPI 4 lane) mentioned above, the second video transmission channel can also be flexibly configured into multiple MIPI sub-channels, forming various optional configuration combinations, depending on the MIPI interface resources of the main control processor (such as RK3588). For example, a configuration of 1×PCIe 2.0×4 + 2×MIPI 4 lanes is suitable for high-bandwidth parallel transmission scenarios with dual video sources; a configuration of 2×MIPI 4 lanes + 4×MIPI 2 lanes is suitable for scenarios with multi-channel video source split transmission. The number and bit width combination of the MIPI sub-channels are set by the main control processor according to the product model requirements, through configuration commands (such as MIPI_LANE_MODE) issued via the SPI control link. The FPGA's internal video stream distribution management module then allocates each video stream to the corresponding MIPI transmission interface. The above configuration is also dynamically adjusted via SPI commands during system operation, without the need to reprogram the FPGA bit stream. When multiple MIPI sub-channels are configured, the total effective bandwidth of the second video transmission channel increases proportionally to the number of channels and the bit width, further enhancing the flexibility of channel configuration and providing greater bandwidth margin for degraded transmission in the event of PCIe failure.

[0039] As an optional implementation, depending on the product model requirements, the PCB may only have differential traces corresponding to the first video transmission channel (PCIe), only have differential traces corresponding to the second video transmission channel (MIPI), or have differential traces corresponding to both the first and second video transmission channels simultaneously. When only a single channel is deployed, the corresponding transmit interface of the undeployed channel remains disabled within the FPGA, and the system operates with a single channel; when both channels are deployed simultaneously, both channels can operate, achieving the aforementioned redundancy and degradation.

[0040] PCIe 2.0 uses a ×4 configuration, with a line rate of 5GT / s per lane per direction, for a total link rate of 20GT / s across the four lanes. PCIe uses 8b / 10b encoding, with the payload accounting for 80%, resulting in an effective data rate of 16Gbps. After deducting TLP protocol overhead (packet header, ECRC checksum, flow control, etc., accounting for approximately 8%), the actual effective one-way bandwidth is approximately 14.7Gbps (approximately 1.8GB / s).

[0041] The MIPI CSI-2 uses a 4-lane D-PHY v1.2 interface with a line rate of 2.5Gbps per channel, for a total link rate of 10Gbps across all four channels. The D-PHY also uses 8b / 10b encoding, resulting in an effective data rate of 8Gbps. After deducting the CSI-2 protocol overhead (frame header, packet header, ECC / CRC checksum, approximately 3%), the actual effective bandwidth is approximately 7.76Gbps (approximately 970MB / s).

[0042] It can be seen that the effective bandwidth of the first video transmission channel (PCIe 2.0×4) is about 1.8 times that of the second video transmission channel (single MIPI 4 lane), and the data transmission rate of the first video transmission channel is higher than that of the second video transmission channel. When the second video transmission channel is configured as multiple MIPI sub-channels, its total effective bandwidth increases exponentially with the number of channels.

[0043] Taking YUV422 8-bit format video as an example, the bandwidth requirements for each video stream are as follows: UHD 4K@60fps is approximately 995MB / s (approximately 7.96 Gbps); 1080P@60fps is approximately 248.8MB / s (approximately 1.99Gbps); 720P@60fps is approximately 110.6MB / s (approximately 0.88Gbps); 1080P@30fps is approximately 124.4MB / s (approximately 1.0Gbps); and 720P@30fps is approximately 55.3MB / s (approximately 0.44Gbps).

[0044] Accordingly, the effective bandwidth of PCIe 2.0×4 can support approximately 7 channels of 1080P@60fps (7×1.99Gbps≈13.9Gbps<14.7Gbps), meeting the high bandwidth requirements of simultaneous transmission of multiple 4K / 1080p channels; the effective bandwidth of a single MIPI 4 lane can support approximately 2 channels of 1080P@60fps (2×1.99Gbps≈3.98Gbps<7.76Gbps). If two MIPI 4 lanes are configured, approximately 4 channels of 1080P@60fps can be supported. If two MIPI lanes are configured, the carrying capacity will be further improved.

[0045] Figure 2 The following core functional modules are integrated within the programmable logic device. The front-end interface receiver layer includes video and control interfaces such as LVDS×2 / MIPI×1 / CAN / RS485 / RS422×2 / UART×2. The interface protocol parsing module includes modules for LVDS decoding / MIPI CSI-2 reception / industrial control protocol parsing. Other modules include:

[0046] The video stream distribution management module connects to the first video transmission channel (PCIe bus) and the second video transmission channel (MIPI bus), and is used to allocate video data to the corresponding channel for transmission according to a preset link strategy. The video stream distribution management module supports three operating modes:

[0047] (1) First working mode (single PCIe mode): Only the PCIe bus is enabled as the video transmission channel. The main control processor sends the configuration command "MODE_PCIE_ONLY" to the FPGA via SPI. The FPGA's internal video stream distribution management module turns off the clock of the MIPI transmission interface and routes all video streams to the PCIe bus. This mode is suitable for high-bandwidth scenarios where multiple 4K video streams are transmitted simultaneously.

[0048] (2) Second working mode (single MIPI mode): Only the MIPI bus is enabled as the video transmission channel. The main control processor sends the configuration command "MODE_MIPI_ONLY" to the FPGA via SPI. The FPGA's internal video stream distribution management module disables the PCIe hard core IP (to reduce power consumption) and routes all video streams to the MIPI bus. This mode is suitable for low-power and low-cost scenarios. The MIPI bus in this mode can be one or more MIPI sub-channels, and the specific number and bit width can be flexibly set according to the aforementioned multi-channel MIPI configuration.

[0049] (3) Third working mode (dual-channel mode): The PCIe bus and MIPI bus are used as video transmission channels simultaneously. The main control processor sends the configuration command "MODE_DUAL" to the FPGA via SPI. The FPGA's internal video stream distribution management module enables two links simultaneously. The PCIe carries the main video stream, and the MIPI carries the auxiliary video stream or serves as a dynamic backup channel. The MIPI can be one or more sub-channels, and the auxiliary video stream carried is distributed among the MIPI sub-channels by the video stream distribution management module.

[0050] The three modes can be dynamically switched during system operation via SPI instructions, without the need to reprogram the FPGA bit stream, thus realizing the software definition of the hardware architecture.

[0051] The link status monitoring module is used to monitor the link status of the PCIe bus in real time. Specifically, it monitors the link status signal "pcie_link_up" (high level indicates L0 activity) output by the PCIe hard core IP, sampling the signal at a period of 1ms; at the same time, it monitors the bandwidth utilization of the PCIe bus (obtained through the PCIe performance counter).

[0052] The channel switching control module connects the link status monitoring module and the video stream distribution management module. When the link status monitoring module detects a PCIe link anomaly (the "pcie_link_up" signal is low for three consecutive sampling cycles), it sends a "switch_req" switching request signal to the channel switching control module. The channel switching control module waits for the current video frame to complete transmission (detecting the end-of-frame marker) and then performs a switching operation at the start of the next frame—switching the output target of the video stream distribution management module from the PCIe bus to the MIPI bus. After the switch, the video stream distribution management module automatically initiates a degradation strategy: reducing the acquisition resolution from 1920×1080 to 1280×720 and the frame rate from 60fps to 30fps to adapt to the bandwidth limitations of the MIPI bus. Furthermore, when the link status monitoring module detects that the PCIe bus bandwidth utilization exceeds a preset threshold (e.g., 85%), it triggers the channel switching control module to switch newly added video streams to the MIPI bus for transmission, achieving load balancing between the two video channels.

[0053] The control command pass-through module is connected to the control transmission link (SPI bus) to receive and parse control commands from the main control processor. These control commands include, but are not limited to: state machine configuration commands for programmable logic devices, synchronous trigger commands for multiple video inputs, configuration commands and data reception commands for multiple heterogeneous sensors, reconstruction commands for internal parameters of programmable logic devices, switching commands for three operating modes, and configuration commands for the number and bit width of MIPI sub-channels.

[0054] like Figure 5 As shown, the video data double-buffered queue connects the multi-video input interface and the video stream distribution management module. The double-buffered queue contains two independent frame buffers—buffer A and buffer B. Under normal operating conditions, while a frame of video data is written to buffer A, the data of the previous frame, already completed in buffer B, is being read out and sent to the video stream distribution management module. When the channel switching control module performs a channel switch from PCIe to MIPI, the switching command only takes effect during the frame gap (i.e., the moment when the current frame transmission is complete and the next frame has not yet started), ensuring no data loss during the switching process.

[0055] The global clock synchronization module includes the following sub-modules: an external time synchronization interface sub-module, which connects to an external clock synchronization interface and supports the reception and decoding of PPS, PTP, and / or B-code time synchronization signals; a clock discipline sub-module, which uses digital phase-locked loop technology to calibrate the frequency and phase of the local crystal oscillator with the external time synchronization signal as a reference; a clock synchronization distribution sub-module, which distributes the calibrated high-precision local clock to the main control processor through a clock synchronization link; and a global timestamp generation sub-module, which generates a 64-bit global timestamp counter based on the calibrated local clock and adds a unified timestamp to each sensor data stream. Figure 4 This embodiment demonstrates the specific implementation of how the main control processor (such as RK3588) sends a hard trigger synchronization signal to the FPGA via the CS (chip select) pin of the SPI bus, as described in detail in this embodiment, and provides a principle derivation of its response delay.

[0056] Under normal operating conditions, the SPI CS pin is high (inactive). When the main controller needs to trigger the FPGA to perform multi-channel video synchronous acquisition, it pulls the CS pin low (active). Internally, the FPGA connects the CS signal to a dedicated interrupt input pin, which has the highest priority within the FPGA (higher priority than all video data processing tasks). In response to a falling edge interrupt on the CS pin, the FPGA triggers the multi-channel video synchronous acquisition module to perform the following operations within no more than 5 clock cycles (50ns for a 100MHz clock): ① latch the current global timestamp counter value; ② simultaneously enable frame capture operations on all front-end video interfaces; ③ append a unified timestamp to the video frame data of each channel.

[0057] In this embodiment, the SPI clock is 50MHz (clock cycle 20ns). However, it should be noted that the hard trigger path of the CS pin is completely independent of the SPI clock and protocol stack. The SPI clock frequency only affects the transmission rate of ordinary control instructions on the SPI data lines (MOSI / MISO), and does not affect the response speed of the CS hard trigger. The path of the CS hard trigger signal is as follows: the main control processor GPIO pulls the CS pin low → PCB trace propagation → FPGA input pin → FPGA internal IO input buffer and cross-clock domain synchronization → triggering the synchronous acquisition action. The estimated delays for each segment are as follows: the PCB trace propagation delay is approximately 1-3ns (calculated based on a trace length of tens of centimeters, approximately 0.1-0.2ns per centimeter); the FPGA internal IO input buffer and cross-clock domain synchronization require approximately 2-3 clock cycles (the FPGA operates at a 100MHz clock, i.e., 20-30ns); the three actions of latching the timestamp, enabling frame capture, and appending the timestamp require approximately 2 clock cycles (i.e., 20ns). The total delay is approximately 50ns (approximately 5 clock cycles), consistent with the design specifications.

[0058] The hardware-level timing alignment value of this mechanism lies in the fact that the CS hard trigger signal does not go through PCIe's TLP packetization, DMA transfer, host interrupt, and software driver parsing, nor does it go through the SPI protocol stack. Its response delay is a fixed value that is independent of the amount of video data. In contrast, if control commands are transmitted through the PCIe shared channel, they need to go through TLP packetization (approximately 1-10µs), DMA transfer, host interrupt, and software parsing (typically 10-100µs). Moreover, when multiple video streams fill the PCIe bandwidth, control commands need to queue and wait, further increasing the delay and making it uncertain. Therefore, the hard trigger response delay of this invention does not increase with the increase of video data volume. This is the essential difference between the SPI dedicated control channel and the PCIe shared control channel, ensuring nanosecond-level timing alignment accuracy for synchronous exposure / acquisition of multiple video sensors.

[0059] like Figure 3 As shown, this embodiment provides a video transmission and control method based on the above-mentioned heterogeneous computing system, which includes the following steps:

[0060] Step S1: Receiving multiple video data streams

[0061] The programmable logic device (PLD) receives multiple video data streams through two LVDS interfaces and one MIPI interface on the front end. Simultaneously, the PLD receives industrial control data through the front-end CAN, RS485, RS422, and UART interfaces.

[0062] Step S2: Real-time monitoring of PCIe link status

[0063] The link status monitoring module inside the programmable logic device monitors the link status of the PCIe bus in real time, including the PCIe physical layer activity status (“pcie_link_up” signal) and the PCIe bus bandwidth utilization.

[0064] Step S3: Video transmission under normal conditions

[0065] When the PCIe bus is functioning normally ("pcie_link_up" is high), the video stream distribution management module of the programmable logic device transmits multiple video data streams to the main control processor via the PCIe bus. Simultaneously, the link status monitoring module continuously monitors the bandwidth utilization of the PCIe bus. When the bandwidth utilization exceeds a preset threshold, newly added video data streams are transmitted via the MIPI bus, achieving load balancing between the two video channels.

[0066] Step S4: Degradation Switching and Transmission in Case of PCIe Anomaly

[0067] When the link status monitoring module detects an anomaly on the PCIe bus (the “pcie_link_up” signal is low for three consecutive sampling cycles), the channel switching control module performs the following operations: ① Wait for the current video frame transmission to complete (detect the frame end flag); ② At the beginning of the next frame, switch the output target of the video stream distribution management module from the PCIe bus to the MIPI bus; ③ Use a double-buffered queue for video data to ensure that no data is lost during the switching process; ④ After the switching is completed, automatically execute a degradation transmission strategy—reduce the resolution of the video data (from 1080p to 720p) and / or reduce the frame rate of the video data (from 60fps to 30fps) to adapt to the bandwidth limitations of the MIPI bus.

[0068] Step S5: Independent transmission of the SPI control channel

[0069] Throughout the video data transmission process, the programmable logic device (FPGA) receives control commands from the main controller in real time via an SPI control link independent of PCIe and MIPI, and executes corresponding operations based on these commands. When the main controller sends a hard-triggered synchronization signal via the CS pin of the SPI bus, the FPGA responds with the highest priority, performing synchronous acquisition of multiple video inputs within nanoseconds.

[0070] Taking 30fps video as an example, the single frame period is approximately 33.3ms (1 / 30), and the frame gap (vertical blanking period) is typically 1-2ms. The typical time for enabling and starting the MIPI transmission interface (D-PHY initialization, PLL locking, and interface timing establishment) is less than 100us. During the switching process, detecting the end-of-frame flag takes less than 1us, and then the MIPI interface enabling and video stream distribution target switching are completed within the frame gap (less than 100us). This time is much shorter than the frame gap (1-2ms), so the switching action can be completed within the frame gap without causing any perceptible interruption to the video output. Simultaneously, while frame N is written to buffer A in the double-buffered queue, frame N-1 is read from buffer B. The switching command only takes effect during the frame gap. During the switching process, the target channel is switched on the writing side while the data stream on the reading side remains continuous, ensuring, from a mechanism perspective, that the video data has no gaps and no dropped frames.

[0071] Using the above link bandwidth data, the bandwidth requirement for 1080P@60fps video is approximately 248.8 MB / s, for 720P@60fps it's approximately 110.6 MB / s, for 1080P@30fps it's approximately 124.4 MB / s, and for 720P@30fps it's approximately 55.3 MB / s. When downgrading the video from 1080P@60fps to 720P@60fps, the bandwidth requirement decreases from 248.8 MB / s to 110.6 MB / s, a reduction of approximately 55.6%; when downgrading from 1080P@60fps to 720P@30fps, the reduction is approximately 77.8%. Therefore, when multiple video streams switch to the MIPI channel after a PCIe failure, for example, if three 1080P@60fps videos are fully switched and downgraded to 720P@60fps, the total bandwidth requirement is 3 × 110.6 ≈ 331.8 MB / s, which is still lower than the effective bandwidth of a single MIPI 4-lane (970 MB / s). This can fully support the MIPI channel, ensuring that critical video data is not interrupted during a PCIe failure. If the second video transmission channel is expanded into multiple MIPI sub-channels (such as two MIPI 4-lane channels with a total effective bandwidth of approximately 1.16 GB / s) according to the aforementioned configuration, the bandwidth margin for downgraded transmission is further increased, allowing it to support multiple critical video streams with higher resolutions / frame rates.

[0072] The effective bandwidth of PCIe 2.0×4 is approximately 1.8GB / s. Assuming a preset threshold of 85%, the threshold bandwidth is approximately 1.53GB / s. When the system carries one UHD 4K@60fps video stream (995MB / s), the total bandwidth requirement is 1×995≈1GB / s, accounting for approximately 56% of the effective PCIe bandwidth, which is still below the threshold. When a second UHD 4K@60fps video stream is added, if all streams are transmitted via PCIe, the total bandwidth requirement is approximately 2GB / s, exceeding the threshold. At this point, the link status monitoring module triggers the channel switching control module to switch the second video stream to MIPI bus transmission and reduces the frame rate of the UHD 4K@60fps video stream to UHD 4K@30fps video stream (occupying 497.5MB / s, lower than the available bandwidth of 970MB / s for a single MIPI stream), achieving load balancing between the two video transmission channels and effectively utilizing all available bandwidth resources.

[0073] like Figure 6 As shown, the specific implementation of the four-level global clock synchronization mechanism is as follows:

[0074] Level 1 – External Clock Synchronization: The FPGA front end has a clock synchronization interface (INT) that connects to an external timing device (such as a GNSS receiver or a PTP master clock). The FPGA internally implements the following logic: PPS second pulse processing – The FPGA latches the current high-precision counter value using the rising edge of the PPS signal as a trigger, calculates the deviation from the ideal value, and adjusts the local clock frequency control word through a digital phase-locked loop to achieve clock discipline; PTP protocol processing – The FPGA identifies PTP event messages (Sync, Delay_Req, etc.) at the physical layer. Timestamps are directly recorded by hardware when messages enter or leave the FPGA, avoiding delay jitter introduced by software processing. The main processor runs the PTP protocol stack, calculates the link delay and clock offset based on the timestamps recorded by the FPGA, and writes the correction values ​​back to the FPGA's clock adjustment register.

[0075] The second level – Clock synchronization between the FPGA and the main control processor: The FPGA and the main control processor synchronize their clocks via a clock synchronization link (which can be shared with the SPI control channel). The FPGA internally maintains a 64-bit high-precision clock counter, driven by a locally tamed clock at a frequency of 100MHz (10ns precision). Every 1ms, the FPGA sends the current clock counter value to the main control processor via the clock synchronization link. The operating system on the main control processor side converts the received hardware clock value into system time and corrects the local system time. When external time synchronization signals are lost, the FPGA continues to keep time using its locally tamed crystal oscillator, maintaining high time accuracy for a short period (e.g., within 24 hours).

[0076] Level 3 – Timestamp Labeling of Sensor Data: The FPGA integrates a global timestamp generation module. The core of this module is a 64-bit global timestamp counter, driven by the FPGA's 100MHz local clock. For the various sensor data received from the front end: LVDS cameras (2 channels) – When the FPGA detects the rising edge of the LVDS frame synchronization signal (VSYNC), it latches the current global timestamp counter value as the timestamp of that frame image, appending it to the header of the image data; MIPI cameras (1 channel) – When the FPGA receives the MIPI CSI-2 frame start packet (FS), it latches the current global timestamp counter value as the timestamp of that frame image; CAN / RS485 / RS422 / UART industrial control data – When the FPGA receives a complete data frame, it latches the current global timestamp counter value as the timestamp of that data frame. All timestamped data is encapsulated internally within the FPGA and then transmitted to the main control processor via PCIe or the MIPI video channel. The main control processor performs time alignment and fusion processing on the multi-source data based on the timestamps.

[0077] Level Four – System-Level Time Coordination: Through the three-level synchronization mechanism described above, the entire heterogeneous computing system achieves the following synchronization accuracy: Level 1 external synchronization is sub-microsecond (PPS / PTP); Level 2 internal synchronization is microsecond; and Level 3 data annotation is nanosecond (10ns resolution). The final result is that all sensor data within the system has a high-precision unified timestamp, consistent with the time base of the external host and collaborating devices, meeting the time synchronization requirements for multi-sensor fusion and system-level collaborative work.

[0078] Combination Figure 1 The heterogeneous computing system shown illustrates the system-level application of a dual-video-channel redundant architecture in intelligent optoelectronic devices. Taking an airborne optoelectronic pod as an example, the system needs to continuously acquire multiple video data streams (visible light, infrared, etc.) and process them in real time during flight missions. If a traditional single PCIe video transmission scheme is used, the entire video acquisition system will be completely paralyzed and the mission will fail if the PCIe link fails due to vibration, temperature changes, or electromagnetic interference.

[0079] After adopting the dual-video channel redundancy architecture of the present invention: (1) During normal operation, 3 video channels (2 LVDS + 1 MIPI) are transmitted to RK3588 for AI inference processing at full resolution and full frame rate through the PCIe 2.0×4 bus. The total bandwidth of the 3 channels of 1080p60 is about 5.97Gbps, which is lower than the effective bandwidth of PCIe 14.7Gbps, leaving sufficient margin; (2) When the PCIe link is abnormal, the system automatically switches to the MIPI bus within microseconds and continues to transmit key video data in a reduced frame rate manner (60fps→30fps). The total bandwidth of the 3 channels of 1080p30 is about 2.99Gbps, which is lower than the bandwidth of a single MIPI 4. The effective bandwidth of lane is 7.76Gbps, ensuring that pilots or ground stations can still obtain key visual information; (3) The SPI control channel always works independently. Even if the PCIe and MIPI video channels are abnormal, the SPI control channel can still work normally. The main control processor can query the FPGA status, perform system-level fault diagnosis and reset operations through SPI; (4) The global clock synchronization continues to run. Even if the video channel is switched, the global clock synchronization mechanism of the FPGA is not affected, and the timestamps of all sensor data always maintain a unified reference.

[0080] This architecture improves the system's reliability from "a single point of failure can lead to complete failure" to a fault-tolerant level of "system degrades operation without interruption of tasks in the event of a single point of failure," while ensuring the temporal consistency of multi-sensor data, which has significant engineering value for high-reliability intelligent optoelectronic devices.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, various modifications and variations can be made to the present invention without departing from its spirit and scope. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A heterogeneous computing system for intelligent optoelectronic devices, characterized in that, include: The main control processor is connected to the first memory; A programmable logic device connected to a second memory; A video transmission link and a control transmission link are provided between the main control processor and the programmable logic device; The video transmission link includes a first video transmission channel and a second video transmission channel, wherein the data transmission rate of the first video transmission channel is higher than the data transmission rate of the second video transmission channel. Both the first video transmission channel and the second video transmission channel are dedicated video data channels; The control transmission link is independent of the video transmission link and is a dedicated channel for control instructions between the main control processor and the programmable logic device. The front end of the programmable logic device is connected to multiple video input interfaces and multiple industrial control interfaces; The programmable logic device integrates: A video stream distribution management module is connected to the first video transmission channel and the second video transmission channel, and is used to distribute video data to the first video transmission channel and / or the second video transmission channel for transmission according to a preset link strategy; The link status monitoring module is used to monitor the link status of the first video transmission channel in real time. The channel switching control module is connected between the link status monitoring module and the video stream distribution management module. When the first video transmission channel is abnormal, the video data is switched to the second video transmission channel for transmission. A control command pass-through module is connected to the control transmission link and is used to receive and parse control commands from the main control processor.

2. The heterogeneous computing system for intelligent optoelectronic devices according to claim 1, characterized in that, The first video transmission channel is a PCIe bus, the second video transmission channel is a MIPI bus, and the control transmission link is an SPI bus; the multi-channel video input interface includes at least one LVDS interface and / or at least one MIPI interface; the multi-channel industrial control interface includes one or more of CAN interface, RS485 interface, RS422 interface, and UART interface; the main control processor is an ARM architecture processor, and the programmable logic device is an FPGA.

3. The intelligent optoelectronic device heterogeneous computing system according to claim 2, characterized in that, The video stream distribution management module has the following operating modes: a first operating mode that only enables the PCIe bus as a video transmission channel, a second operating mode that only enables the MIPI bus as a video transmission channel, and a third operating mode that simultaneously enables the PCIe bus and the MIPI bus as video transmission channels; when the PCIe bus is abnormal, the channel switching control module executes a degradation transmission strategy, switches the video data to the MIPI bus, and transmits it at a reduced resolution and / or a reduced frame rate.

4. The intelligent optoelectronic device heterogeneous computing system according to claim 2, characterized in that, The link status monitored by the link status monitoring module includes the physical layer activity status of the PCIe bus and / or the bandwidth utilization of the PCIe bus. When the link status monitoring module detects that the bandwidth utilization of the PCIe bus exceeds a preset threshold, it triggers the channel switching control module to switch the new video stream to the MIPI bus for transmission, thereby achieving load balancing between the first video transmission channel and the second video transmission channel.

5. The heterogeneous computing system for intelligent optoelectronic devices according to claim 1, characterized in that, The control transmission link is an SPI bus. The main control processor sends a hard-triggered synchronization signal to the programmable logic device through the chip select signal pin of the SPI bus. In response to the hard-triggered synchronization signal, the programmable logic device performs the synchronous acquisition operation of the multi-channel video input interface with high priority. The response delay of the hard-triggered synchronization signal is on the nanosecond level.

6. The heterogeneous computing system for intelligent optoelectronic devices according to claim 1, characterized in that, The programmable logic device also integrates a video data double-buffered queue, which is connected between the multi-channel video input interface and the video stream distribution management module. When the channel switching control module performs a switching operation from the first video transmission channel to the second video transmission channel, the double-buffered queue maintains continuous writing and reading of video data during the switching period, thereby achieving seamless switching of video data.

7. The heterogeneous computing system for intelligent optoelectronic devices according to claim 1, characterized in that, The programmable logic device (PLD) is also connected to an external clock synchronization interface. The PLD receives external time signals through the external clock synchronization interface and calibrates its local clock according to the external time signals. A clock synchronization link is also provided between the PLD and the main control processor. The PLD synchronizes its calibrated local clock to the main control processor through the clock synchronization link, so that the main control processor and the PLD share a unified time base. The PLD integrates a global timestamp generation module. Based on the unified time base, the global timestamp generation module adds a globally unified timestamp to each frame of data received by the multi-channel video input interface and the multi-channel industrial control interface.

8. A video transmission and control method based on the heterogeneous computing system of intelligent optoelectronic devices according to any one of claims 1-7, characterized in that, Includes the following steps: S1: The programmable logic device receives multiple video data through the front-end multi-channel video input interface and industrial control data through the front-end industrial control interface; S2: The programmable logic device monitors the link status of the first video transmission channel in real time; S3: When the first video transmission channel is normal, the programmable logic device transmits the multiple video data to the main control processor through the first video transmission channel; S4: When the first video transmission channel is abnormal, the programmable logic device executes a downgrade transmission strategy, switching the multi-channel video data to the second video transmission channel for transmission to the main control processor. The data transmission rate of the second video transmission channel is lower than that of the first video transmission channel. S5: The programmable logic device receives control instructions from the main control processor through a control transmission link independent of the first video transmission channel and the second video transmission channel, and performs corresponding operations according to the control instructions.

9. The video transmission and control method according to claim 9, characterized in that, Step S3 further includes: real-time monitoring of the bandwidth utilization of the first video transmission channel; when the bandwidth utilization exceeds a preset threshold, transmitting the newly added video data stream through the second video transmission channel to achieve load balancing between the first and second video transmission channels; the degradation transmission strategy in step S4 includes: reducing the resolution of the video data and / or reducing the frame rate of the video data to adapt to the bandwidth limitations of the second video transmission channel; and during channel switching, the programmable logic device maintains continuous writing and reading of video data through a video data double buffer queue to achieve seamless switching of video data.

10. The video transmission and control method according to claim 9, characterized in that, The control transmission link is an SPI bus. In step S5, the main control processor sends a hard trigger synchronization signal to the programmable logic device through the chip select signal pin of the SPI bus. The programmable logic device responds to the hard trigger synchronization signal and performs synchronous acquisition of the multi-channel video input interface with high priority.

Images