Image transmission system and image transmission method

Abstract

The invention discloses an image transmission system and an image transmission method, and relates to the technical field of image transmission. The system comprises an FPGA (Field Programmable Gate Array) module which is used for being directly connected with a plurality of Cameralink cameras, skipping a decoding chip and directly decoding image data of the plurality of Cameralink cameras; the image processing module is used for carrying out cutting, frame extraction, superposition, black and white processing and image format conversion on the original image; and the virtual camera module is used for coding the original or processed image into a Cameralink format video stream and outputting the Cameralink format video stream to the downstream sub-module through a standard connector. According to the invention, multiple paths of data are processed in a centralized manner through the FPGA, the number of independent driving and hardware modules is reduced, and the lightweight and integrated requirements of aerospace equipment are met; a plurality of sets of decoding chips and redundant cables are omitted, so that the hardware cost and the power consumption are reduced; through the design of the virtual camera, the downstream module can reuse the same hardware interface to adapt to different algorithm requirements, and the expansibility is high.

Description

Technical Field

[0001] The present invention relates to the technical field of image transmission, and in particular to an image transmission system and an image transmission method. Background Art

[0002] Cameralink is a high-speed machine vision interface standard, widely used in industrial cameras. It is classified by resolution: ordinary resolution and high-resolution cameras; by output color: monochrome (black and white) and color cameras; by response frequency range: visible light, infrared and ultraviolet cameras; by scanning mode: progressive scanning and interlaced scanning cameras; by frame rate: low frame rate and high frame rate cameras. The traditional Cameralink camera system faces the following problems: high system complexity: each camera requires independent drive and dedicated decoding chip (such as DS90CR287 / 288), resulting in hardware redundancy; high cost and power consumption: multiple decoding chips, multiple cables and multiple image encoding and decoding modules greatly increase cost and power consumption; poor flexibility: each submodule requires a customized interface and cannot dynamically adapt to different algorithm requirements.

[0003] At present, the image acquisition systems of aerospace and military products are becoming more and more complex. The video data required by each sub-module is not the same. Therefore, it is necessary to select the corresponding camera according to the needs. Each camera is matched with its own driver and processed by a dedicated decoding chip, which limits the integration and real-time performance, and also leads to increased power consumption and cost. Summary of the invention

[0004] The purpose of the present invention is to provide an image transmission system and an image transmission method, which can reduce system complexity, cost and power consumption and improve flexibility and compatibility through hardware architecture optimization and function virtualization.

[0005] To achieve the above object, the present invention provides the following technical solution: an image transmission system, comprising: FPGA module, which is used to directly connect multiple Cameralink cameras and skip the decoding chip to directly decode the image data of multiple Cameralink cameras; DDR3 memory, which is used to partition and multi-frame cache the multi-channel image data of the Cameralink camera; Image processing module, which is used to crop, extract frames, overlay, black and white process the original image and convert the image format; Standard video interface, which is used to send pre-processed image data to the CPU or display control host computer; The virtual camera module is used to encode the original or processed image into a Cameralink format video stream and output it to the downstream sub-module through a standard connector.

[0006] Preferably, the FPGA module comprises: A camera interface unit, which is used to support multiple parallel accesses in CameralinkBase, Medium or Full modes; A protocol parsing unit, which is used to parse the Cameralink serial data protocol directly through FPGA logic; The pre-processing control unit is used to dynamically configure the cropping area, frame extraction frequency and format conversion rules.

[0007] Preferably, the camera interface unit supports simultaneous connection of at least 4 Cameralink cameras, with each camera having a bandwidth of ≥ 850 Mbps.

[0008] Preferably, the partition rule of the DDR3 memory is: Divide independent buffer areas by camera channel; Each buffer stores multiple consecutive frames of images, and the number of frames is configurable.

[0009] Preferably, the image processing module includes: An image cropping unit, which is used to extract the effective area of ​​the image according to a preset ROI; A frame extraction unit, which is used to reduce the frame rate according to a set ratio; The format conversion unit is used to convert image data from RGB to grayscale, perform Bayer interpolation, and convert YUV422 to RGB.

[0010] Preferably, the virtual camera module comprises: An encoding logic unit, which is used to encapsulate the image data into a serial signal according to the Cameralink protocol; The interface simulation unit is used to output signals through a physical connector to simulate the camera models supported by the downstream module.

[0011] The present invention also provides an image transmission method, comprising the following steps: Directly connect to multiple Cameralink cameras via FPGA to directly parse and decode video data; Store multiple image data into DDR3 memory by channel partition; Trim, extract frames, and convert formats of cached data; Distribute the processed data to the CPU or display and control device through a standard interface; Encode the raw or processed data into a Cameralink signal and output it to the downstream submodule in the form of a virtual camera.

[0012] Preferably, the multi-channel image data is stored in the DDR3 memory by channel partition, and the buffer area of ​​each channel is independently allocated to support multi-frame cyclic overwriting writing.

[0013] Preferably, the output is sent to a downstream submodule in the form of a virtual camera, and the encoded parameters are dynamically adjusted to match the Cameralink protocol version of the downstream module.

[0014] Compared with the prior art, the present invention has the following beneficial effects: Centralized processing of multiple data channels through FPGA reduces the number of independent drivers and hardware modules, meeting the requirements of lightweight and integrated aerospace equipment. By eliminating multiple decoding chips and redundant cables, hardware costs and power consumption are reduced; The virtual camera design allows downstream modules to reuse the same hardware interface, adapt to different algorithm requirements, and has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of the structure of an image transmission system provided by an embodiment of the present invention; Figure 2 A schematic diagram of an application of an image transmission system provided by an embodiment of the present invention; Figure 3 A main flow chart of an image transmission method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figure 1 and Figure 2 The present invention provides an image transmission system 100, comprising: The FPGA module 110 is used to directly connect to multiple Cameralink cameras and skip the decoding chip to directly decode the image data of the multiple Cameralink cameras; DDR3 memory 120, which is used to partition and multi-frame cache the multi-channel image data of the Cameralink camera; An image processing module 130 is used to perform cropping, frame extraction, superposition, black-and-white processing and image format conversion on the original image; A standard video interface 140, which is used to send the pre-processed image data to the CPU or the display control host computer, for example, to transmit the processed data to the CPU through the PCIe3.0 interface and output it to the display control host computer through the HDMI2.0 interface; The virtual camera module 150 is used to encode the original or processed image into a Cameralink format video stream and output it to the downstream sub-module through a standard connector.

[0018] In an optional embodiment, the FPGA module 110 includes: The camera interface unit is used to support multiple parallel access in CameralinkBase, Medium or Full mode. Its interface type supports CameralinkBase (4 channels), Medium (2 channels) or Full (1 channel) mode. The LVDS differential signal receiving circuit is configured through the BankI / O of the FPGA. When multiple parallel accesses are performed, a single FPGA can simultaneously connect to 4 CameralinkBase mode cameras (each with a bandwidth of ≥850Mbps). Signal isolation is achieved through the physical layer (PHY) pin allocation, and an impedance matching circuit (such as a 100Ω termination resistor) is integrated to reduce signal reflection interference; DC balanced encoding (such as 8B / 10B encoding) is supported to ensure data integrity; for example, the I / O level is dynamically adjusted through the SelectIO technology of the FPGA to be compatible with the electrical standards of different Cameralink camera manufacturers (such as Basler, Dalsa, etc.); The protocol parsing unit is used to directly parse the Cameralink serial data protocol through FPGA logic, skip the traditional decoding chip (such as DS90CR287), and use the built-in high-speed SerDes (serial deserializer) of FPGA to convert the Cameralink serial data into 28-bit parallel data (24-bit pixel data + 4-bit control signal); separate the timing signals such as pixel clock (PixelClock), line synchronization (LineValid), frame synchronization (FrameValid) from the data stream; at the same time, the protocol parsing unit also detects transmission errors through CRC (cyclic redundancy check) and triggers the data retransmission mechanism; for example, when parsing the CameralinkFull mode protocol of the Baslerace camera, the data format can be set to 80-bit transmission (including 24-bit RGB data, 4-bit control signal and 52-bit reserved bit); the clock frequency is 85MHz (corresponding to 2048×1088@60fps resolution); The preprocessing control unit is used to dynamically configure the preprocessing algorithm, cropping area, frame extraction frequency and format conversion rules, and adjust the preprocessing parameters (such as ROI coordinates, frame extraction ratio) in real time through the host computer software or register mapping; and use the parallel computing capability of FPGA to realize multi-channel data synchronous processing (such as processing ROI cropping of 4 cameras at the same time).

[0019] In an optional embodiment, the FPGA module 110 further includes: Status monitoring and feedback unit, which is used to monitor the FPGA operation status in real time and feedback error information and performance data; The status monitoring and feedback unit includes: The status register subunit is used to record the data throughput, buffer occupancy, preprocessing parameters, etc. of each channel; Error alarm subunit, which is used to detect abnormalities such as signal loss and CRC check failure, and trigger an interrupt to notify the host computer; The power management subunit is used to dynamically shut down idle modules (such as unused camera channels) to reduce overall power consumption.

[0020] In an optional embodiment, the partition rule of the DDR3 memory 120 is: Divide independent buffer areas by camera channel; Each buffer area stores multiple consecutive frames of images, and the number of frames is configurable.

[0021] It can be understood that the DDR3 memory 120 can be divided into 4 independent buffer areas according to the camera channel, and each area stores 5 consecutive frames of images (resolution 2048×2048, 8-bit grayscale); At the same time, the cache area supports cyclic overwrite writing to ensure that real-time data is not lost.

[0022] In an optional embodiment, the image processing module 130 includes: Image cropping unit, which is used to extract the effective area of ​​the image according to the preset ROI, reduce invalid data transmission and processing, such as setting the starting coordinates (X, Y) and width and height (Width, Height) of the ROI through FPGA registers or host computer software. The image cropping unit also generates a hardware mask based on the coordinate counter to retain only the pixel data within the ROI. The amount of data after cropping is reduced, reducing storage and transmission pressure (for example, cropping from a 2048×2048 full frame to a 1024×1024 ROI reduces the data volume by 75%). The frame extraction unit is used to reduce the frame rate according to the set ratio to adapt to the computing power limitation or specific algorithm requirements of the back-end processing unit. For example, it extracts frames according to the 1 / 2, 1 / 3, 1 / 4 mode (such as retaining 1 frame for every 3 frames). At the same time, it can also dynamically adjust the frame extraction strategy. For example, the frame extraction ratio is switched in real time according to the CPU load feedback (such as enabling 1 / 3 frame extraction when the load is >80%), and the continuity of the line and field synchronization signals is maintained when discarding frames to avoid timing errors of downstream modules; The format conversion unit is used to convert image data from RGB to grayscale, perform Bayer interpolation, and convert YUV422 to RGB, as well as truncate high-bit-depth data (such as 12-bit) to 8-bit, compatible with standard interfaces; Among them, the Bayer pattern raw data (such as RGGB) is converted into RGB format using a bilinear interpolation algorithm.

[0023] In an optional embodiment, the image processing module 130 further includes: An image stacking unit is used to stack images from multiple cameras into a single data stream for multi-sensor information fusion, including: merging multiple images through an Alpha blending algorithm (such as transparency stacking); spatially aligning multiple ROIs based on calibration parameters (such as translation and rotation compensation); and using FPGA's DSPSlice to accelerate floating-point operations (such as the Alpha blending formula: Pixel_out=α×Pixel_A+(1-α)×Pixel_B, where Pixel_out represents the output pixel value after blending, α is the blending factor, Pixel_A is the pixel value of the upper layer or foreground, and Pixel_B is the pixel value of the lower layer or background). For example, in multispectral imaging, by fusing visible light and infrared images, specific target features can be highlighted. The black and white processing unit is used to convert color images into grayscale images, reduce the amount of data and adapt to the requirements of monochrome algorithms. For example, grayscale processing is performed through the standard grayscale formula: Y=0.299R+0.587G+0.114B (ITU-RBT.601), and simplification is performed through shift and addition approximate calculations (such as Y=(R+2G+B)>>2), where >>2 means shifting right by two bits, Y is the output grayscale value, R is the pixel value of the red channel, G is the pixel value of the green channel, and B is the pixel value of the blue channel. In practical applications, monochrome images are more suitable for edge detection and feature extraction, and the amount of grayscale image data is reduced by 66% compared to RGB.

[0024] In an optional embodiment, the virtual camera module 150 simulates the output behavior of a real physical camera, so that downstream modules (such as display control devices, algorithm processing units, etc.) do not need to perceive the difference in data sources, and directly reuse the original interface and driver. Through hardware-level protocol encapsulation and signal simulation, the system compatibility is significantly improved and the hardware dependency cost is reduced. Specifically, it includes: An encoding logic unit, which is used to encapsulate the image data into a serial signal according to the Cameralink protocol; Among them, the corresponding data packet structure is generated according to the Cameralink protocol version (Base / Medium / Full), including: Pixel data: 24-bit RGB or 8 / 10 / 12-bit grayscale data; Control signals: line synchronization (LineValid), frame synchronization (FrameValid), data valid (DataValid); Reserved bits: fill the reserved fields specified by the protocol (such as the 52-bit reserved bits in CameralinkFull mode); It can be understood that the encoding logic unit ensures the timing synchronization of downstream modules by generating a pixel clock (PixelClock) and synchronization signal that is completely consistent with the physical camera. At the same time, the encoding logic unit also supports dynamic switching of protocol versions (such as switching from Base mode to Full mode) to be compatible with different downstream device requirements.

[0025] The interface simulation unit is used to output signals through a physical connector to simulate the camera models supported by the downstream module.

[0026] In an optional embodiment, the interface simulation unit further includes: The electrical characteristics simulation subunit is used to generate differential signals that comply with the Cameralink electrical standard and match the output characteristics of the physical camera, including generating signals that comply with the Cameralink electrical specification through the LVDS (low voltage differential signal) output pin of the FPGA: differential voltage: ±350mV (typical value); impedance matching: 100Ω termination resistor to reduce signal reflection; pre-emphasis and equalization to compensate for signal attenuation in long-distance transmission; and eye diagram test to verify signal quality (such as eye diagram opening ≥80%UI); Dynamic parameter configuration subunit, which is used to adjust the output parameters of the virtual camera in real time and simulate the behavior of different models of physical cameras, including: configuring the output resolution (such as 2048×2048) and frame rate (such as 60fps) through registers, supporting external trigger (ExternalTrigger), continuous acquisition (Free-Run) and other modes, responding to external synchronization signals, and dynamically switching the output data format (such as RGB888, YUV422, 8-bit grayscale); The multi-protocol compatible subunit is used to support the private protocol extensions of mainstream Cameralink camera manufacturers and improve downstream compatibility, including parsing private control commands (such as camera parameter configuration instructions) of manufacturers such as Basler and Dalsa, and emulating manufacturer-specific data packet structures (such as Basler's metadata embedding mechanism). Therefore, the target brand can be switched through host computer instructions without hardware modification; The error test subunit is used to artificially inject error signals for robustness testing of downstream modules. The controllable error types include: data frame loss: randomly discarding frames in proportion; signal jitter: introducing controllable jitter (such as ±5% UI) in the pixel clock; CRC error: intentionally generating an erroneous checksum; when injecting an error signal, the error injection can be started by an external instruction or a timer; The data optimization subunit is used to compress or downsample the output data to adapt to low-bandwidth downstream modules. For lossless compression, data redundancy is reduced by using run-length encoding (RLE) or differential encoding. For lossy compression, the amount of data is reduced by quantization or downsampling (such as compressing from 12 bits to 8 bits).

[0027] In this embodiment, multiple channels of data are centrally processed through FPGA to reduce the number of independent drives and hardware modules, meeting the requirements of lightweight and integrated aerospace equipment. By eliminating multiple sets of decoding chips and redundant cables, hardware costs and power consumption are reduced. Through virtual camera design, downstream modules can reuse the same hardware interface to adapt to different algorithm requirements and have strong scalability.

[0028] Based on the above embodiments, Figure 3 As shown, the present invention also provides an image transmission method for supporting the image transmission system of the above embodiment, and the image transmission method comprises the following steps: Step S210, directly connecting multiple Cameralink cameras via FPGA to directly parse and decode video data.

[0029] Optionally, step S210 includes: Step S211, directly connecting at least 4 Cameralink cameras via FPGA to receive serial data in real time; Step S212, parsing the Cameralink protocol to extract pixel clock, horizontal and vertical synchronization signals and image data.

[0030] Step S220: storing the multiple channels of image data into the DDR3 memory according to the channel partitions.

[0031] Optionally, step S220 further includes: The image data of each camera is stored in the corresponding DDR3 buffer area by channel; the buffer area adopts a "ping-pong operation" mechanism to achieve seamless switching of data reading and writing.

[0032] Among them, Ping-Pong operation is an efficient data management technology based on double buffering, which is mainly used to achieve seamless switching of data reading and writing processes. Its name comes from the fact that two buffers (BufferA and BufferB) work alternately like ping-pong balls, ensuring continuous data processing while avoiding read-write conflicts.

[0033] In this embodiment, the cache area adopts a "ping-pong operation" mechanism to perform data reading and writing steps as follows: Step S221: writing data into BufferA and reading data from BufferB at the same time; Step S221: When BufferA is full or BufferB is empty, switch operation: Stop writing to BufferA and start writing to BufferB; Stop reading from BufferB and start reading from BufferA; Step S223, looping the above steps to form a continuous "ping-pong" switching.

[0034] It can be understood that the ping-pong operation mechanism realizes efficient parallel processing of data reading and writing on hardware platforms such as FPGA through intelligent switching of double buffers, which significantly improves the throughput and real-time performance of the system.

[0035] Step S230, trimming, extracting frames, and converting formats of the buffered data.

[0036] Optionally, step S230 includes: Step S231, extracting data from the buffer area and performing ROI clipping (retaining the central 1024×1024 area); Step S232, extract frames from the trimmed data (reducing the frame rate to 1 / 2), and convert it into YUV420 format; Step S240, distributing the processed data to the CPU or display control device through a standard interface; Step S250 , encoding the original or processed data into a Cameralink signal, and outputting it to a downstream submodule in the form of a virtual camera.

[0037] Optionally, step S250 further includes: Step S251, sending the pre-processed data to the CPU via the PCIe interface for algorithm processing; Step S252, encode the full frame raw data into a CameralinkFull mode signal, and output it to the downstream module through the MDR26 interface.

[0038] Furthermore, the multi-channel image data is stored in the DDR3 memory by channel partition, and the buffer area of ​​each channel is independently allocated, supporting multi-frame cyclic overwrite writing.

[0039] Furthermore, it is output to the downstream submodule in the form of a virtual camera, and the encoded parameters are dynamically adjusted to match the Cameralink protocol version of the downstream module.

[0040] In this embodiment, multiple channels of data are centrally processed through FPGA to reduce the number of independent drives and hardware modules, meeting the requirements of lightweight and integrated aerospace equipment. By eliminating multiple sets of decoding chips and redundant cables, hardware costs and power consumption are reduced. Through virtual camera design, downstream modules can reuse the same hardware interface to adapt to different algorithm requirements and have strong scalability.

[0041] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be considered exemplary and non-restrictive in all respects, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present invention.

Claims

1. An image transmission system, characterized in that: include: FPGA module, which is used to directly connect multiple Cameralink cameras and skip the decoding chip to directly decode the image data of multiple Cameralink cameras; DDR3 memory, which is used to partition and multi-frame cache the multi-channel image data of the Cameralink camera; Image processing module, which is used to crop, extract frames, overlay, black and white process the original image and convert the image format; Standard video interface, which is used to send pre-processed image data to the CPU or display control host computer; The virtual camera module is used to encode the original or processed image into a Cameralink format video stream and output it to the downstream sub-module through a standard connector.

2. The system according to claim 1, characterized in that The FPGA module includes: A camera interface unit, which is used to support multiple parallel accesses in CameralinkBase, Medium or Full modes; A protocol parsing unit, which is used to parse the Cameralink serial data protocol directly through FPGA logic; The pre-processing control unit is used to dynamically configure the cropping area, frame extraction frequency and format conversion rules.

3. The system according to claim 2, characterized in that The camera interface unit supports simultaneous connection of at least 4 Cameralink cameras, with each channel bandwidth ≥ 850Mbps.

4. The system according to claim 1, characterized in that The partition rules of the DDR3 memory are: Divide independent buffer areas by camera channel; Each buffer stores multiple consecutive frames of images, and the number of frames is configurable.

5. The system according to claim 1, characterized in that The image processing module comprises: An image cropping unit, which is used to extract the effective area of ​​the image according to a preset ROI; A frame extraction unit, which is used to reduce the frame rate according to a set ratio; The format conversion unit is used to convert image data from RGB to grayscale, perform Bayer interpolation, and convert YUV422 to RGB.

6. The system according to claim 1, characterized in that The virtual camera module includes: An encoding logic unit, which is used to encapsulate the image data into a serial signal according to the Cameralink protocol; The interface simulation unit is used to output signals through a physical connector to simulate the camera models supported by the downstream module.

7. An image transmission method, characterized in that: The following steps are involved: Directly connect to multiple Cameralink cameras via FPGA to directly parse and decode video data; Store multiple image data into DDR3 memory by channel partition; Trim, extract frames, and convert formats of cached data; Distribute the processed data to the CPU or display and control device through a standard interface; Encode the raw or processed data into a Cameralink signal and output it to the downstream submodule in the form of a virtual camera.

8. The method according to claim 7, characterized in that The multi-channel image data is stored in the DDR3 memory according to channel partitions, and the buffer area of ​​each channel is independently allocated, supporting multi-frame cyclic overwriting and writing.

9. The method according to claim 7, characterized in that: The output is sent to the downstream submodule in the form of a virtual camera, and the encoded parameters are dynamically adjusted to match the Cameralink protocol version of the downstream module.

Images