Method and apparatus for synchronizing signals in an ethernet bridge architecture with timing replication

Abstract

This application relates to time sensitive multimedia endpoint interface bridging over Ethernet transport. An Ethernet bridge architecture enables timing replication. The Ethernet bridge receives data packets from a sensor (e.g., a video sensor) and immediately marks each data packet with a transmitter time code. The marked data packets are then forwarded over a digital data network or link that can exhibit packet delivery time variations and reordering to the appropriate receiver. The receiver identifies the data packets that include a local time code and delays processing (display) of the data packets until the time code local to the receiving node matches the transmitter time code plus a certain delay. The receiver also restores the original order of the packets by observing packet sequence numbers and placing them in the appropriate place in a memory buffer. By delaying processing, the Ethernet bridge compensates for any variations in network latency.

Description

Technical Field

[0001] This application relates to bridging of time-sensitive multimedia endpoint interfaces delivered over Ethernet. Background Technology

[0002] The availability of high-speed data links, switches, and other infrastructure devices enables the transmission of uncompressed pixel data between video endpoints (image sensors and displays) and the host processors that consume or generate this data. However, attaching such sensors to high-speed Ethernet currently requires several off-the-shelf components containing a MAC, PHY, microcontroller, and some logic to implement a bridge between the traditional image sensor and the Ethernet on the transmission side. This logic can be implemented using a Field-Programmable Gate Array (FPGA). This solution is expensive and power-intensive, and therefore impractical for mass deployment. On the receiving side, there is no image signal processor that can be directly attached to Ethernet, and many available components are designed to interface directly with the sensor via dedicated interfaces. For camera applications, these are Mobile Industry Processor Interface (MIPI) and Camera Serial Interface (CSI) interfaces. For displays, these are HDMI, VESA, OpenLDI, and MIPIDSI interfaces. If a non-integrated Ethernet bridge is implemented, it will require a PHY, MAC, and logic, which is also expensive.

[0003] Ethernet-based video delivery becomes feasible if a highly integrated dedicated bridge and Ethernet cable between the sensor and display interfaces are available. However, issues related to data segmentation at the receiving end, packet encapsulation, decapsulation, reordering, and timing recovery must still be addressed, especially if the bridge must be functionally invisible to the application—that is, truly replicating the sensor interface signals and timing on the image signal processor side. Summary of the Invention

[0004] On one hand, embodiments of the inventive concept disclosed herein relate to an Ethernet bridge architecture with timed replication. The Ethernet bridge receives data packets from a sensor (e.g., a video sensor) and immediately tags each packet with a transmitter timecode. The tagged data packets are then forwarded to an appropriate receiver. The receiver identifies data packets containing local timecodes and delays the processing (display) of the packets until the local timecode of the receiving node matches the transmitter timecode plus a certain delay. Through this delay processing, the Ethernet bridge compensates for any variations in network latency. Furthermore, the delay is greater than the worst-case delay defined by the network architecture.

[0005] The Ethernet bridge allows for distributed multi-camera and multi-display systems based on high-bandwidth Ethernet infrastructure, while still using non-Ethernet sensors, displays, and application processors.

[0006] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and should not limit the scope of the claims. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate exemplary embodiments of the inventive concepts disclosed herein and, together with the general description, serve to explain the principles. Attached Figure Description

[0007] By referring to the accompanying drawings, those skilled in the art can better understand the numerous advantages of the embodiments of the inventive concepts disclosed herein, in which:

[0008] Figure 1 A block diagram of a system according to an exemplary embodiment is shown;

[0009] Figure 2 A block diagram illustrating a data packet according to an exemplary embodiment;

[0010] Figure 3 A block diagram illustrating a system for time-coding and transmitting data packets according to an exemplary embodiment;

[0011] Figure 4 A block diagram illustrating a system for receiving and processing data packets according to an exemplary embodiment;

[0012] Figure 5 A block diagram illustrating a system for transmitting and receiving data packets in multiple virtual channels, according to an exemplary embodiment. Detailed Implementation

[0013] Before explaining in detail at least one embodiment of the inventive concept disclosed herein, it should be understood that the application of the inventive concept is not limited to the details of the construction and arrangement of the components, steps, or methodologies set forth in the following description or illustrated in the accompanying drawings. In the following detailed description of embodiments of the inventive concept, numerous specific details are set forth to provide a more thorough understanding of the inventive concept. However, it will be apparent to those skilled in the art to which this disclosure pertains that the inventive concept may be practiced without these specific details. In other instances, well-known features may not have been described in detail to avoid unnecessarily complicating this disclosure. The inventive concept disclosed herein can have other embodiments or can be practiced or implemented in various ways. Moreover, it should be understood that the phrases and terms used herein are for descriptive purposes and should not be considered limiting.

[0014] As used herein, the letters following element symbols are intended to refer to embodiments of features or elements that may be similar to, but not necessarily identical to, those of the aforementioned elements or features bearing the same element symbol (e.g., 1, 1a, 1b). Such shorthand notation is for convenience only and should not be construed as limiting the inventive concepts disclosed herein in any way, unless expressly stated otherwise.

[0015] Furthermore, unless explicitly stated to the contrary, "or" refers to inclusive or rather, not exclusive or. For example, conditions A or B satisfy any of the following: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); and both A and B are true (or exist).

[0016] Additionally, the use of "a" or "an" is to describe elements and components of embodiments of the inventive concept. This is done merely for convenience and to give a general meaning to the inventive concept, and "a" and "an" are intended to include one or at least one, and the singular includes the plural, unless they clearly mean otherwise.

[0017] Furthermore, while various components may be described as being directly connected, direct connection is not mandatory. Components may communicate data with intermediary components that are not specified or described. It should be understood that "data communication" refers to both direct and indirect data communication (e.g., the possibility of intermediary components).

[0018] Finally, as used herein, references to “one embodiment” or “some embodiments” mean that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the inventive concept disclosed herein. The appearance of the phrase “in at least one embodiment” in the specification does not necessarily refer to the same embodiment. Embodiments of the disclosed inventive concept may include one or more of the features expressly described herein or inherently present, or any combination or sub-combination of two or more such features.

[0019] In summary, embodiments of the inventive concepts disclosed herein relate to an Ethernet bridge architecture with realistic timing relationship replication. The Ethernet bridge receives data packets from a sensor (e.g., a video sensor) and immediately tags each packet with a transmitter timecode. The tagged data packets are then forwarded to the appropriate receiver. The receiver identifies data packets containing local timecodes and delays the processing (display) of these packets until the local timecode of the receiving node matches the transmitter timecode plus a certain delay. Through this delay processing, the Ethernet bridge compensates for any variations in network latency. This delay can be greater than the worst-case latency defined by the network architecture. The Ethernet bridge allows for distributed multi-camera and multi-display systems based on high-bandwidth Ethernet infrastructure while still using non-Ethernet sensors, displays, and application processors.

[0020] refer to Figure 1 The diagram illustrates a block diagram of a system according to an exemplary embodiment. The system includes a transmitting or source node 100 and a receiving or aggregation node 102. The transmitting node 100 (e.g., a network-connected camera, microphone, or other audio sensor, or the like) receives 106 data packets from a sensor 104 (e.g., a video sensor, audio sensor, or the like) and tags each data packet 108 with a timecode local to the transmitting node. The transmitting node 100 then transmits the tagged data packets 110 to the receiving node 102 (e.g., a display, audio monitor, application that uses the tagged data packets for data processing, or the like).

[0021] Receiver node 102 receives 112 tagged data packets, identifies the timecodes in the data packets, compares them with the local timecode of receiver node 102, and delays 114 the processing of the data packets (e.g., display). When the data packet timecode plus some predefined delay matches the receiver's local timecode, receiver node 102 processes 116 that data packet; for example, it displays it on monitor 118.

[0022] It should be understood that each of the transmitting node 100 and the receiving node 102 includes a processor (transmitting node processor and receiving node processor, respectively) for performing the steps described herein. In the context of this disclosure, "processor" may refer to a dedicated processor, general-purpose programmable central processing unit (CPU), field-programmable gate array, microprocessor, microcontroller, embedded processor, digital signal processor, and other such data processing technologies that are hardwired for the described purposes. Where the processor includes means configurable by software or firmware, this software or firmware may be embodied in non-transitory memory; this memory may be a PROM, EPROM, EEPROM, flash memory, dynamic random access memory, or the like. The controller 600 is configured to perform certain process steps 606, 608, 610, 612, as described more fully herein.

[0023] In the event of a certain level of latency caused by the network connection between transmitting node 100 and receiving node 102, embodiments of this disclosure ensure that the variable delay in packet delivery does not affect the timing recovery and synchronization of the conventional display at receiving node 102. In at least one embodiment, if the network has some known worst-case latency, a predefined delay can be defined as greater than that worst-case latency.

[0024] refer to Figure 2This illustrates the structure of a data packet stream 200 according to an exemplary embodiment. In at least one embodiment, the data packet 200 includes a MIPI CSI data stream, a long packet 202 containing less time-critical pixel data, and a short packet 204 carrying time-sensitive synchronization events. Figure 2 The diagram illustrates a single virtual channel packet stream on a MIPI CSI port, containing a mixture of short packets 204 and long packets 202. The ingress port should record a timecode when the time-critical short packet 204 arrives. When the timecode is captured, the corresponding short packet 204 along with the corresponding timecode is sent over the network connection. Any suitable encapsulation method can be used; in at least one embodiment, a tuple of {type, length, value} is used as a descriptor for the subsequent data payload, which is relatively easy to parse in hardware.

[0025] While the specific exemplary embodiments relate to CSI, it should be understood that the concepts described herein apply to other interfaces. Such interfaces may include, but are not limited to, video serial or parallel interfaces, audio interfaces, USB interfaces with isochronous services, etc. Furthermore, while the specific exemplary embodiments relate to video streaming, the inventive concepts are equally applicable to any stream in which packet delivery is time-sensitive, including video, audio, time-sensitive control events, etc.

[0026] If long packets cannot be fitted into a single Ethernet network packet, they can be segmented into multiple smaller packets, each containing a sequence number and a data payload. These packets can be reordered during propagation over the network link, which can also cause delivery time variations. By observing the sequence number and placing the data payload in the receive buffer at the appropriate location, the payload order can be reconstructed at the receiving node. Reordering restores the original order of the data payloads within the long packet.

[0027] In at least one embodiment (e.g., MIPI CSI), short packet 204 comprises four bytes and carries a stream ID. The occurrence time of the short packet indicates the time when pixel data was captured and is used to synchronize processing with the display. Long packet 202 can carry up to 64k bytes and carries a large amount of pixel data. In at least one embodiment, packets 202 and 204 use stream IDs to represent virtual channels and allow data streams from different sources to be interleaved.

[0028] When the MIPI CSI interface is used in vision and ADAS applications, sensor timing is represented by the time positions of the vertical and horizontal synchronization signals and the corresponding FSYN and LSIN short packets 204 in the MIPI CSI data stream. The hardware detects the arrival time of these short packets 204 and correlates them with a shared time base by recording local timecodes and generating presentation timestamps (PTS).

[0029] The timecode is transmitted to the receiving end in the video element data packet and is then used by the receiver's output interface to create corresponding FSIN and LSYN packets at the receiver-side time, which is equal to the original PTS plus a certain delay. This delay can be greater than all possible delays in network transmission, including delays in the MAC and PHY of the network interface, transmission delays, and packet forwarding delays in network switches and bridges. The same method can be used in multi-stream mode, where independent streams are distinguished by corresponding virtual channel identifiers in the Ethernet packets, and independent timestamps are used for packets belonging to different virtual channels.

[0030] refer to Figure 3 This diagram illustrates a possible system for time-coding and transmitting data packets according to an exemplary embodiment. The MIPI CSI interface component 300 captures a data stream containing video data or the like. It should be understood that the MIPI CSI interface component 300 may include any device with a CSI interface, including CSI-1, CSI-2, CSI-3, etc. The MIPI CSI stream is examined by a CSI packet parser 302, which detects the arrival of time-sensitive short packets and signals. When the CSI packet parser detects a time-sensitive short packet, a timestamp 304 immediately records a local timecode. Recording the local timecode may include sampling a synchronization network time base such as that defined by the IEEE 1588 Virtual NCO and tick counter or other time synchronization protocols of this network. This sample becomes the presentation timestamp (PTS) of the corresponding data packet. A Type Length Value (TLV) builder 306 receives the data stream (containing pixel and control data) from the CSI packet parser 302 and the timecode from the time sampler 304.

[0031] The time-coded packet stream is sent from TLV builder 306 to packetizer FIFO 308 for elastic buffering, and then passed to video protocol packetizer 310. Video protocol packetizer 310 constructs Ethernet packets according to the desired delivery protocol. Video protocol packetizer 310 sends the created video stream packets to Ethernet transport component 314. Ethernet transport component 314 may include an Ethernet physical interface (PHY) configured to arbitrate between video packets and control plane packets from control plane processing element 312, which are transmitted between source and sink nodes and used for remote control and status communication. The Ethernet MAC may implement enhancements such as credit-based traffic shaping or leaky bucket traffic shaping.

[0032] refer to Figure 4This diagram illustrates a block diagram of a system for receiving and processing data packets according to an exemplary embodiment. A receiving node receives packets from a network via an Ethernet component 400 and directs video stream packets to a unpacker 402 and then to a unpacker FIFO buffer 406. Data packets contain a TLV structure with timecode captured when the packet appears on a transmitter node. A PTS extractor 408 extracts the timecode from each data packet and delivers it to a PTS comparator 410; the PTS extractor 408 also delivers pixel and control data to a packet release element 412. In at least one embodiment, the PTS extractor 408 may add a predefined delay to the timecode.

[0033] PTS comparator 410 continues to compare the timecode with a predefined delay to the synchronization network time base. When the synchronization network time base equals the timecode plus the predefined delay, PTS comparator 410 signals packet release element 412 to feed pixel and control data to MIPI CSI interface component 414, such as an image signal processor (ISP) or display device. The signal on MIPI CSI interface 414 replicates the input timing from the transmission node, but with an additional fixed delay.

[0034] Because the latency is constant for all MIPI packets, and the time base is synchronized between the receiver and transmitter nodes, the timing relationship between short packets carrying the start of frame and start of line signals is the same as that of the originating sensor. Therefore, application processors can use them to recover the line rate, frame rate, and other video-related clocks at the receiving node. Ethernet also provides a bidirectional communication channel for control plane 404 messages used for remote node management and time synchronization.

[0035] refer to Figure 5 This document illustrates a block diagram of a system for transmitting and receiving data packets across multiple virtual channels, according to an exemplary embodiment. The principles described herein can be extended to receiving streams from multiple virtual channels at one or more transmitter nodes and routing them as interleaved MIPI CSI packets to an output interface. The packets are received by a packet filter 500, which identifies packets requiring timecode synchronization and forwards those packets to a depacketizer 502. The data streams are then assigned to virtual channel FIFO buffers 506 by a virtual channel demultiplexer 504. Each virtual channel data stream is then released via a packet release element 510. The packet release element 510 compares the timecode in each data stream with a synchronization network time base 508 containing a predefined delay and releases the corresponding data to a multiplexer and packet builder 512. The resulting data packets are then delivered to a CSI multiplexing element 514 for release to the corresponding processing interface (e.g., a MIPI image processing interface).

[0036] It should be understood that, in the context of this disclosure, a "virtual channel" refers to a data connection architecture between a transmitting node and a receiving node, where a single physical channel, such as an Ethernet connection, supports multiple independent logical connections, for example, defined via separate data ports. Alternatively or additionally, a virtual channel may be defined by multiple different data connections between the transmitting node and the receiving node.

[0037] In at least one embodiment, packet release element 510 can determine that more than one data stream should be released simultaneously; that is, the timecode plus the delay is the same for data in more than one virtual channel FIFO buffer 506. In that case, a suitable arbitration scheme can be used to release the packet. For example, round-robin arbitration can be used.

[0038] Embodiments of this disclosure implement a method for maintaining a common time base from an image sensor to a network-connected display, which can be used in industrial machine vision, studio fabrication, or ADAS automotive technology. The network-connected device detects and measures time-critical events on the transmission side, transmits the desired time base along with the data to the receiving end, and delays the release of time-sensitive data on the receiving side until the original timecode is added with a predefined delay, which may be greater than all worst-case possible transmission delays in the Ethernet infrastructure.

[0039] The implementation benefits from Ethernet's ability to extend, broadcast, and multicast streams for use by more than one receiver by using switches interconnected via potentially different speed links, and to aggregate and interleave multiple independent data streams from multiple sources for delivery on a common segment. Routing between source and aggregation endpoints can depend on application and system state and can be dynamically changed as needed during vehicle operation. Conversely, the display interface on the application processor, including its timing, must be replicated on the Ethernet bridge of the remote display, as the display panel uses timing to synchronize its internal circuitry.

[0040] It is believed that the inventive concept disclosed herein and its many incidental advantages will be understood from the foregoing description of embodiments of the inventive concept, and it will be apparent that various changes can be made to the form, construction, and arrangement of its components without departing from the broad scope of the inventive concept disclosed herein or without sacrificing all its substantial advantages; and individual features from various embodiments can be combined to obtain other embodiments. The forms previously described herein are merely illustrative embodiments, and the appended claims are intended to cover and include such changes. Furthermore, any of the features disclosed with respect to any of the individual embodiments may be incorporated into any other embodiment.

Claims

1. A method for synchronizing signals, comprising: Data packets are received via the transmission node processor; Record the local timecode of the transmission node; The data packet is marked with the time code local to the transmission node; The data packet is transmitted to the receiving node; The data packet is received via the receiving node processor; The data packet is processed with a delay until the local timecode of the receiving node matches the local timecode of the transmitting node plus the delay; The data packet is processed when the local timecode of the receiving node matches the local timecode of the transmitting node plus the delay; as well as Define multiple virtual channels that carry independent data streams; When the timecode in the tagged data packet is matched with the timecode local to the transmission node plus a delay, the data in the corresponding virtual channel is released to the interface.

2. The method according to claim 1, wherein the timecode local to the transmitting node and the timecode local to the receiving node are synchronized via a network time synchronization protocol.

3. The method according to claim 1, wherein the data packet includes a Short Synchronous Camera Serial Interface (CSI) data packet.

4. The method of claim 1, further comprising: For each of the plurality of virtual channels, perform the following steps independently: Data packets are received via the transmission node processor; Record the local timecode of the transmission node; The data packet is marked with the time code local to the transmission node; and The data packet is transmitted to the receiving node.

5. The method of claim 4, further comprising: For each of the plurality of virtual channels, it is determined when the timecode local to the receiving node matches the timecode local to the transmitting node in the tagged data packet plus a delay.

6. The method of claim 1, wherein the delay is greater than the worst-case delay defined by the network architecture.

7. A node in a network, comprising: At least one processor configured to: The receiving node receives data packets from the transmitting node; Identify the local time code of the transmission node marked in the data packet; The data packet is processed with a delay until the local timecode of the receiving node matches the local timecode of the transmitting node plus the delay; and The data packet is processed when the local timecode of the receiving node matches the local timecode of the transmitting node plus the delay.

8. The node of claim 7, wherein the at least one processor is further configured to: Receive data packets from sensors that communicate with the at least one processor; Record the local timecode of the transmission node; The data packet is marked with the local time code; as well as The data packet is transmitted to the receiving node; Define multiple virtual channels; For each of the plurality of virtual channels, the at least one processor is configured to: Receive data packets from the sensor; Record the local timecode of the transmission node; The data packet is marked with the local timecode of the transmission node.

9. The node according to claim 7, wherein the timecode local to the transmitting node and the timecode local to the receiving node are synchronized via a network time synchronization protocol.

10. The node of claim 7, wherein the latency is greater than the worst-case latency defined by the network architecture.

11. The node of claim 8, wherein the sensor includes a device having a CSI interface.

12. The node according to claim 8, wherein: For each of the plurality of virtual channels, the at least one processor is configured to: The data packet is transmitted to the receiving node.

13. The node of claim 12, wherein the at least one processor is further configured to: For each of the plurality of virtual channels, determine whether the local timecode of the receiving node matches the local timecode of the transmitting node plus a delay; and Release the data in the corresponding virtual channel to the interface.

14. An apparatus comprising: A depacker configured to receive sensor data packets; A timestamp, configured to record a local time code, the timestamp being configured to record the local time code whenever a data packet is received; A memory buffer, configured to: Receive the sensor data packet; and Extract timecode data from the sensor data packet; as well as Encapsulator, configured to: Receive sensor data packets from the memory buffer; If the data packet is determined to be a long packet, the recorded local timecode is discarded. Generate a new data packet from the sensor data packet data and the local time code; as well as The new data packet is sent to the receiving node.

15. The device according to claim 14, wherein: The unpacker is configured to: Receive data packets from the transmission node; and The data in the data packets from the transmission node are unpacked and reordered to restore the original order; It further includes: A timestamp extractor configured to extract timecodes from the data packets; A timestamp comparator configured to compare the data packet timestamp with the receiver timestamp locally on the computer device; and A packet release element, configured to perform local-to-network synchronization based on the packet timecode to replicate event timings from the transport node.

16. The device of claim 15, wherein the data packet timecode and the receiver timecode are synchronized via a network time synchronization protocol.

17. The apparatus of claim 15, wherein the packet release element is further configured to add a delay to the packet timecode.

18. The device of claim 17, wherein the latency is greater than the worst-case latency defined by the network architecture.

19. The apparatus according to claim 15, wherein: The control plane processor is configured to define multiple virtual channels; and For each of the plurality of virtual channels: The unpacker receives sensor data packets; The timestamp recorder records a local timestamp; and The encapsulator: Generate a new data packet based on the local timecode; and The new data packet is sent to the receiving node.

Images