Multi-camera image processing system and its data processing method
By configuring a DMA controller in the image processing system, a point-to-point direct connection channel is established to directly transmit unencoded raw image data, solving the problem of long data transmission paths in existing technologies. This achieves low-latency and high-efficiency image data transmission, making it suitable for applications such as real-time monitoring and autonomous driving.
Patent Information
- Application Number
- CN202611418952.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2026-08-07
- Filing Date
- 2026-09-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing digital image surveillance technologies suffer from lengthy data transmission paths between edge computing devices and image processing devices, resulting in significant computational resource consumption and high latency, which cannot meet the real-time requirements of applications such as security protection and autonomous driving.
The image processing system employing multiple cameras establishes a point-to-point physical direct connection channel between the image processing device and the edge computing device by configuring a DMA controller in the image processing device, directly transmitting unencoded raw image data and using a specific transmission format for high-speed transmission.
It significantly reduces the latency between edge computing devices and image processing devices, reduces the time delay of image encoding/decoding and data encapsulation, meets real-time requirements, increases data throughput, and improves system efficiency.
Smart Images

Figure CN123123741A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a technique for transmitting raw video data, and more particularly to a multi-camera image processing system and its data processing method. Background Technology
[0002] Digital image surveillance and image processing technologies are advancing rapidly. Traditional surveillance technology involves one or more cameras capturing images, which are then processed and stored by a back-end host (or digital video recorder, DVR) to provide functions such as split-screen, historical video playback, moving object detection, and remote online viewing.
[0003] On the other hand, with the rise of artificial intelligence, moving object detection and edge computing functions can be performed by a device other than the DVR. Specifically, the DVR encodes the image data (e.g., H.264 / H.265 / AV1, etc.), then encapsulates the encoded image data using data transmission protocols (e.g., Real-time Streaming Protocol (RTSP), HTTP, etc.) before transmitting it to another device. This other device then needs to decapsulate and decode the image data, resulting in a lengthy path from image acquisition to AI analysis, consuming significant computing resources and causing high latency. This is a fatal flaw if this digital image monitoring technology is used in applications requiring real-time responses (such as security and autonomous driving). Summary of the Invention
[0004] This invention provides a multi-camera image processing system and its data processing method, which can significantly reduce the peer-to-peer latency between edge computing devices and image processing devices.
[0005] The multi-camera image processing system of the present invention includes multiple cameras, an image processing unit, and an edge computing unit. The image processing unit is coupled to the cameras, wherein each camera provides its own raw image data to the image processing unit. The image processing unit communicates with the edge computing unit via a cable, enabling the edge computing unit to obtain the raw image data of each camera from the image processing unit. The image processing unit includes a buffer memory, a data receiving circuit, and a direct memory access (DMA) controller. The data receiving circuit is coupled to the buffer memory. The data receiving circuit temporarily stores the raw image data of each camera in the buffer memory. The DMA controller is coupled to the buffer memory and the cable. The DMA controller reads the raw image data from the buffer memory and uses the cable to transfer the raw image data of each camera to the raw image buffer of the edge computing unit.
[0006] This invention proposes a data processing method for an image processing system. The image processing system includes multiple cameras and an edge computing device. The data processing method includes: obtaining raw image data for each of the cameras; and communicating with the edge computing device via a cable, enabling the edge computing device to obtain the raw image data for each camera from the image processing device. The step of communicating with the edge computing device via the cable, enabling the edge computing device to obtain the raw image data for each camera from the image processing device, includes: temporarily storing the raw image data for each camera in a buffer memory of the image processing device using a data receiving circuit; and reading the raw image data from the buffer memory using a DMA controller of the image processing device, and transmitting the raw image data for each camera to a raw image buffer of the edge computing device via the cable.
[0007] Based on the above, in this embodiment of the invention, when the data receiving circuit in the image processing device receives raw image data from the camera and temporarily stores it in the buffer memory, it does not perform the original video encoding and data encapsulation process. Instead, it uses a DMA controller to directly transmit the raw image data in the buffer memory to the raw image buffer of the edge computing device via a physical cable and a specific transmission format. This significantly reduces the point-to-point latency between the edge computing device and the image processing device in the image processing system, and reduces the time delay for hardware to perform image encoding / decoding, data encapsulation, and decapsulation of the raw image data. This meets the requirements of artificial intelligence applications with high real-time requirements, such as real-time monitoring and autonomous driving assistance systems. Attached Figure Description
[0008] Figure 1This is a schematic diagram of an image processing system according to an embodiment of the present invention.
[0009] Figure 2 yes Figure 1 A detailed block diagram of an image processing system.
[0010] Figure 3 This is a flowchart of a data processing method for an image processing system according to an embodiment of the present invention.
[0011] Explanation of symbols in the attached drawings:
[0012] 100: Image Processing System
[0013] 110-1~110-N: Camera
[0014] 120: Image processing device
[0015] 122: Raw data acquisition device
[0016] 130: Edge computing device
[0017] 140: Physical Direct Connection Channel
[0018] 221: Data receiving circuit
[0019] 222: Buffer memory
[0020] 223: Direct Memory Access Controller
[0021] 225: Image Encoder
[0022] 226: Network Packaging Processor
[0023] 231: Raw Image Buffer
[0024] 232: Artificial Intelligence Analysis Device
[0025] 233: Depackaging the processor
[0026] 234: Jitter Buffer
[0027] 235: Image Decoder
[0028] 240: Network Interface
[0029] S310~S326: Steps of the data processing method of the image processing system Detailed Implementation
[0030] Figure 1This is a schematic diagram of an image processing system 100 according to an embodiment of the present invention. The image processing system 100 includes one or more cameras (e.g., cameras 110-1 to 110-N), an image processing device 120, and an edge computing device 130. The image processing device 120 includes a raw data acquisition device 122. The edge computing device 130 includes an artificial intelligence analysis device 232. The image processing device 120 may be a digital video recorder (DVR).
[0031] The image processing system 100 can process and store images or videos acquired by one or more cameras to provide functions such as screen splitting, historical video playback, object detection, and remote online viewing. In object detection technology, the raw data acquisition device 122 in the image processing device 120 can transmit image data acquired from cameras 110-1 to 110-N to the edge computing device 130, so that the artificial intelligence (AI) analysis device 232 in the edge computing device 130 can perform AI analysis functions such as edge detection and object detection on the image data.
[0032] The image processing system 100 of this embodiment can significantly shorten the peer-to-peer latency between the edge computing device 130 and the image processing device 120, and reduce the time delay of hardware in image encoding and decoding, data encapsulation and decapsulation of raw image data, thereby meeting the needs of artificial intelligence application scenarios with high real-time requirements, such as real-time monitoring, autonomous driving assistance, security protection, security monitoring, industrial inspection, etc.
[0033] In this embodiment, the raw data acquisition device 122 utilizes a cable to establish a point-to-point (P2P) physical direct connection channel 140 from the image processing device 120 to the edge computing device 130, thereby achieving low-latency transmission of raw image data. Users of this embodiment can utilize different technologies to establish the point-to-point (P2P) physical direct connection channel 140 from the image processing device 120 to the edge computing device 130, such as physical cables, fiber optic cables, optical communication modules, etc., depending on their needs.
[0034] Figure 2 yes Figure 1 A detailed block diagram of the image processing system 100. The image processing device 120 mainly includes a data receiving circuit 221, a buffer memory 222, and a direct memory access (DMA) controller 223. Figure 1 Raw data acquisition device 122 in Figure 2 This is implemented using a DMA controller 223. The image processing device 120 may also include an image encoder 225 and a network encapsulation processor 226.
[0035] The edge computing device 130 mainly includes a raw image buffer 231 and an artificial intelligence (AI) analysis device 232. The edge computing device 130 may also include a depackaging processor 233, a jitter buffer 234, and an image decoder 235.
[0036] In common applications, the image processing device 120 and the edge computing device 130 can use the network interface 240 to transmit image data, but image encoding and decoding, data encapsulation and decapsulation of the image data are required, which causes time delay.
[0037] In detail, in common applications, the image processing device 120 receives uncompressed raw image data from multiple cameras 110-1 to 110-N via a specific signal transmission format (e.g., Serial Digital Interface (SDI)). This raw image data is processed by the data receiving circuit 221 (e.g., an SDI decoder) and then temporarily stored directly in the buffer memory (also known as main memory) of the image processing device 120. The image encoder 225 performs video encoding (e.g., H.264 / H.265 / AV1, etc.) on the temporarily stored raw image data, reducing the data size. Furthermore, the network encapsulation processor 226 encapsulates the video-encoded image data into multiple packets for easy transmission via the network interface 240.
[0038] However, when multiple cameras 110-1 to 110-N are connected to an image processing device 120 at the same time, the images of each camera 110-1 to 110-N need to be encoded, encapsulated and transmitted over the network independently, which may lead to low overall transmission efficiency, forming a multi-channel transmission bottleneck and making it difficult to meet the needs of multi-channel real-time monitoring.
[0039] The depacketizing processor 233 in the edge computing device 130 obtains these packets from the network interface 240 and uses a jitter buffer 234 to reduce network jitter, then restores these packets to video-encoded image data. To ensure smooth image playback, the receiving end must configure the jitter buffer 234 to smooth out inconsistencies in data arrival times. However, the jitter buffer 234 introduces tens to hundreds of milliseconds of additional latency, a fatal flaw for AI applications requiring extremely real-time responses (such as security and autonomous driving).
[0040] On the other hand, the image decoder 235 obtains the video-encoded image data from the jitter buffer 234 and performs image decoding, thereby temporarily storing the original image data in the original image buffer 231. The artificial intelligence analysis device 232 performs image analysis based on the original image data in the original image buffer 231. Since the aforementioned video encoding formats (e.g., H.264 / H.265) often utilize inter-frame prediction technology, the decoding of predicted frames (e.g., P-frames) and bidirectional predicted frames (e.g., B-frames) depends on keyframes (e.g., I-frames) or other frames. This increases the complexity and waiting time of the image decoder 235 of the edge computing device 130 when performing image decoding, making it impossible to achieve independent real-time processing of a single frame.
[0041] To address the aforementioned challenges, the image processing apparatus 120 of this embodiment is equipped with a DMA controller 223 dedicated to point-to-point transmission between the image processing apparatus 120 and the edge computing device 130, thereby establishing a physical direct connection channel 140. The image processing apparatus 120 can be implemented using a system-on-a-chip (SoC) or a field-programmable gate array (FPGA).
[0042] The DMA controller 223 is responsible for directly retrieving unencoded raw image data (e.g., YUV or RGB format image data) from the buffer memory 222, and then transmitting this unencoded raw image data via cable to the raw image buffer 231 of the edge computing device 130. This process completely bypasses the hardware image encoder 225, image decoder 235, network encapsulation processor 226 (e.g., Real-Time Streaming Protocol (RTSP) / Web Application Programming Interface (WEB API)), decapsulation processor 233, and related software processing flows, forming a high-speed point-to-point transmission channel. In this embodiment, the raw image data is unencoded and unencapsulated image data.
[0043] In other words, the data receiving circuit 221 uses the Serial Digital Interface (SDI) protocol to temporarily store the raw image data of each of the cameras 110-1 to 11-N in the buffer memory 222. The DMA controller 223 is coupled to the buffer memory 222 and the cable. The DMA controller 223 reads the raw image data from the buffer memory 222 and uses the cable to transmit the raw image data of each of the cameras 110-1 to 110-N to the raw image buffer 231 of the edge computing device 130. The artificial intelligence analysis device 232 performs image analysis based on the raw image data in the raw image buffer 231. Therefore, the image decoder 235 and the network encapsulation processor 226 are not necessary components of the image processing device 120.
[0044] The DMA controller 223 reads the raw image data of each of the cameras 110-1 to 110-N from the buffer memory 222 and encapsulates this raw image data into a specific transmission format. The artificial intelligence analysis device 232 uses this specific transmission format to obtain the raw image data of each of the cameras 110-1 to 110-N from the data temporarily stored in the raw image buffer 231.
[0045] This embodiment refers to this specific transmission format as Lightweight Raw Video Payload (LRVP). This specific transmission format establishes a custom frame header and raw payload (i.e., the body of the raw image data) for each frame captured in the raw image data. The raw payload is an uncompressed array of continuous pixel data, and this raw payload is transmitted immediately after the aforementioned frame header.
[0046] The frame header may contain only the necessary information for image analysis in the artificial intelligence analysis device 232, such as: a synchronization string consisting of one or more synchronization words, camera ID, absolute frame sequence number, timestamp, and image format identifier. The image format identifier includes at least one or a combination of image width, image height, image color format, header checksum, and total data length. Table 1 lists the fields in the frame header as examples.
[0047] Table 1:
[0048]
[0049] This embodiment can directly push raw image data with a specific transmission format to the edge computing device 130 in a point-to-point (P2P) manner through the high-speed physical layer output interface (e.g., a direct PCIe interface or a network chip with RDMA functionality) configured in the image processing device 120. This physical direct connection channel 140 is a physical line, completely free from interference from the network protocol stack or operating system layer (OS layer), ensuring high-speed and low-latency data transmission.
[0050] The image processing unit 120 continuously monitors the data input from cameras 110-1 to 110-N. Once an image frame acquired by any of cameras 110-1 to 110-N is ready in the buffer memory 222 to become raw image data, the DMA controller 223 immediately initiates transmission, sending this raw image data along with its frame header in a specific transmission format to the raw image buffer 231 via a physical line. In other words, the DMA controller 223 does not wait for multiple frames to be batched, thereby ensuring real-time image transmission in multi-camera scenarios.
[0051] After receiving raw image data from a physical line, the edge computing device 130 can directly write this raw image data into the raw data bitmap area of the raw image buffer 231. The AI engine interface of the AI analysis device 232 can immediately read data from the raw image buffer 231 and perform image analysis without additional decapsulation, video decoding, or anti-shock buffer processing. Therefore, the decapsulation processor 233, the jitter buffer 234, and the image decoder 235 are not necessary components of the edge computing device 130.
[0052] Figure 3 This is a flowchart of a data processing method for an image processing system according to an embodiment of the present invention. Figure 3 Data processing methods are applicable Figure 2 Image processing system 100. Please also refer to... Figure 2 and Figure 3 In step S310, Figure 2 The image processing device 120 obtains the original image data of each of the multiple cameras 110-1 to 110-N. In step S320, the image processing device 120 communicates with the edge computing device 130 via a cable, so that the edge computing device 130 obtains the original image data of each of the multiple cameras 110-1 to 110-N from the image processing device 120.
[0053] Step S320 may include steps S322 to S326. In step S322, the data receiving circuit 221 of the image processing device 120 temporarily stores the raw image data of each of the cameras 110-1 to 110-N in the buffer memory 222 of the image processing device 120. In step S324, the DMA controller 223 of the image processing device 120 reads the aforementioned raw image data from the buffer memory 222 and transmits the raw image data of each of the multiple cameras 110-1 to 110-N to the raw image buffer 231 of the edge computing device 130 via a cable. In step S326, the artificial intelligence analysis device 232 of the edge computing device 130 performs image analysis based on the raw image data in the raw image buffer 231. The detailed flow of each step in the data processing method can be referred to the above embodiments.
[0054] Based on the above, in this embodiment of the invention, when the data receiving circuit in the image processing device receives raw image data from the camera and temporarily stores it in the buffer memory, it does not perform the original video encoding and data encapsulation process. Instead, it uses a DMA controller to directly transmit the raw image data in the buffer memory to the raw image buffer of the edge computing device via a physical cable and a specific transmission format. This significantly reduces the point-to-point latency between the edge computing device and the image processing device in the image processing system, minimizing the time delay for hardware image encoding / decoding, data encapsulation, and decapsulation of the raw image data. This meets the requirements of real-time artificial intelligence applications, such as real-time monitoring and autonomous driving assistance systems. For example, point-to-point transmission between the edge computing device and the image processing device can achieve ultra-low latency at the millisecond level.
[0055] On the other hand, since uncompressed raw image data is transmitted and a specific transmission format defined in this embodiment is used, the effective bandwidth utilization is maximized. This allows more image data to be transmitted per unit time, significantly improving the overall data throughput of the image processing system 100. Furthermore, neither the image processing device 120 nor the edge computing device 130 needs to perform heavy image encoding / decoding and network packet encapsulation / decapsulation processing. The computing resources of the image processing device 120 and the edge computing device 130 can be freed from these repetitive tasks, allowing the image processing device 120 to focus more on the recording function, while the hardware performance of the edge computing device 130 can be more focused on the image analysis of the artificial intelligence analysis device 232, improving the overall system efficiency. Moreover, since point-to-point direct transmission is carried out using dedicated physical lines, data transmission is stable and predictable, without relying on network protocols to handle jitter or packet loss. Therefore, the data transmission path does not need to pass through the jitter buffer 234 of the edge computing device 130, further eliminating the waiting time of tens to hundreds of milliseconds introduced by transmission via the network interface 240.
[0056] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.
Claims
1. A multi-camera image processing system, comprising: Multiple cameras; An image processing device coupled to the camera, wherein the camera provides its own raw image data to the image processing device; as well as An edge computing device, wherein the image processing device communicates with the edge computing device via a cable, enabling the edge computing device to obtain the raw image data of each camera from the image processing device. The image processing device includes: Buffer memory; A data receiving circuit, coupled to the buffer memory, wherein the data receiving circuit temporarily stores the raw image data of each of the cameras in the buffer memory; and A direct memory access (DMA) controller is coupled to the buffer memory and the cable, wherein the DMA controller reads the raw image data from the buffer memory and uses the cable to transfer the raw image data of each camera to the raw image buffer of the edge computing device.
2. The image processing system as claimed in claim 1, wherein the image processing device is a digital video recorder (DVR), and the data receiving circuit is a serial digital interface (SDI) decoding circuit based on the serial digital interface protocol.
3. The image processing system of claim 1, wherein the edge computing device includes the original image buffer and the artificial intelligence analysis device. The artificial intelligence analysis device performs image analysis based on the raw image data in the raw image buffer.
4. The image processing system of claim 3, wherein the DMA controller reads the raw image data from the buffer memory and encapsulates the raw image data into a specific transmission format. The artificial intelligence analysis device thereunder uses the specific transmission format to obtain the original image data of each camera from the data in the original image buffer.
5. The image processing system of claim 4, wherein the specific transmission format includes a frame header and a raw payload, wherein the raw payload is an uncompressed array of consecutive pixel data and the raw payload immediately follows the frame header. The frame header includes a synchronization string, camera channel number, absolute frame number, timestamp, and image format identifier. The image format identifier includes at least one or a combination of image width, image height, image color format, header check code, and total data length.
6. The image processing system of claim 1, wherein the cable is either a physical cable or an optical fiber cable.
7. The image processing system as claimed in claim 1, wherein the image processing device is a system-on-a-chip or a field-programmable gate array.
8. The image processing system of claim 1, wherein the raw image data is unencoded and unencapsulated image data.
9. A data processing method for an image processing system, wherein the image processing system includes multiple cameras and an edge computing device. The data processing method includes: Obtain the raw image data of each of the cameras; as well as The edge computing device communicates with the image processing device via a cable, enabling the edge computing device to obtain the raw image data from each camera. The step of communicating with the edge computing device via the cable, enabling the edge computing device to obtain the original image data of each camera from the image processing device, includes: Using the data receiving circuit of the image processing device, the original image data of each camera is temporarily stored in the buffer memory of the image processing device; and The raw image data is read from the buffer memory using the direct memory access (DMA) controller of the image processing device, and the raw image data of each camera is transmitted to the raw image buffer of the edge computing device using the cable.
10. The data processing method of claim 9, wherein the data processing method is implemented by an image processing device in the image processing system. The image processing device is a digital video recorder (DVR), and the data receiving circuit is a serial digital interface (SDI) decoding circuit.
11. The data processing method of claim 9, wherein the edge computing device includes the original image buffer and the artificial intelligence analysis device. The data processing method further includes: The artificial intelligence analysis device is used to perform image analysis based on the original image data in the original image buffer.
12. The data processing method as described in claim 11, wherein, The step of transmitting the raw image data of each of the cameras to the raw image buffer of the edge computing device using the cable includes: The DMA controller reads the raw image data from the buffer memory and encapsulates the raw image data into a specific transmission format; and The artificial intelligence analysis device obtains the original image data of each camera from the data in the original image buffer based on the specific transmission format.
13. The data processing method of claim 12, wherein the specific transmission format includes a frame header and a raw payload, wherein the raw payload is an uncompressed array of consecutive pixel data and the raw payload immediately follows the image frame header. The frame header includes a synchronization string, camera channel number, absolute frame number, timestamp, and image format identifier. The image format identifier includes at least one or a combination of image width, image height, image color format, header check code, and total data length.
14. The data processing method of claim 13, wherein the step of communicating with the edge computing device via the cable, enabling the edge computing device to obtain the respective raw image data of the cameras from the image processing device, further comprises: Continuously monitor the data input of the camera; as well as When an image frame acquired by any of the cameras is ready in the buffer to become the raw image data, transmission is immediately initiated to transmit the raw image data, along with the frame header of the specific transmission format, through the line to the raw image buffer.
15. The data processing method of claim 9, wherein the cable is either a physical cable or an optical fiber cable.
16. The data processing method of claim 9, wherein the original image data is unencoded and unencapsulated image data.
