System and method for controlling high-speed video

Abstract

The present disclosure relates to systems and methods for controlling high-speed video. The systems and methods transmit video data from a video source to an image processing system. Pixel data is received in a local buffer of a network interface controller and provided in a video transport packet. The video transport packet includes the pixel data, a media access control header, and a video header. The video transport packet is received by another network interface controller, which provides the pixel data directly to a video frame buffer of the image processing system.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for implementing digital video delivery in packet data network environments, including but not limited to environments using high-speed video delivery over Ethernet networks. Some embodiments of the present disclosure relate to a network interface controller configured for efficient video delivery between a digital video source and a networked video processor. Background Art

[0002] Packet networks are used to distribute data packets based on address information contained within each data packet. For example, Ethernet networks use packets to transmit digital data, containing segmented picture payload data in the form of pixel brightness values, as well as framing and other supplemental information between the picture payload data. Processing a pixel stream received from an Ethernet network through a conventional network interface controller (NIC) into a set of network packet data buffers in an image processing system can be inefficient, at least in part due to the need to copy the segmented payloads from the network buffers into the pure video buffers expected by video applications. Summary of the Invention

[0003] In one aspect, the present disclosure relates to a network interface controller for use with a packet network for transmitting video data to an image processing or display system, the network interface controller comprising: a media access controller (MAC); a host agent interface configured to transmit non-video transport packets to a network buffer for further processing in a network stack; and a video transport accelerator configured to receive video transport packets from the media access controller, extract a video payload from the video transport packets, and control the host agent interface to write pixel data directly into a video frame buffer of the image processing system.

[0004] On the other hand, the present disclosure relates to a network interface controller for transmitting pixel data from a video source, the network interface controller comprising: a media access controller; an input interface configured to be directly or indirectly coupled to the video source through a shared resource; an input / output interface for network data packets for non-video communication; and a video transport accelerator configured to receive the pixel data into a local buffer and create a video transport packet stream containing a fragment sequence of video primitives in a payload, and provide the stream to the media access controller.

[0005] In yet another aspect, the present disclosure relates to a method for transmitting video data from a video source to an image processing system, the method comprising: receiving pixel data in a local buffer of a first network interface controller and creating a video transport packet stream, wherein the video transport packet includes a media access control header, a video streaming protocol header, and pixel payload data; receiving the video transport packet from a second network interface controller and providing the pixel data to a video frame buffer of the image processing system without recopying or reformatting the pixel data by host processor software. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The various objects, aspects, features and advantages of the present disclosure will become more apparent and better understood by reference to the detailed description taken in conjunction with the accompanying drawings, in which reference characters identify corresponding elements throughout. In the drawings, like reference numerals generally indicate identical, functionally similar and / or structurally similar elements.

[0007] Figure 1 is a generally schematic block diagram of a video delivery system according to some embodiments;

[0008] Figure 2 According to some embodiments Figure 1 A generally schematic block diagram of a video frame buffer in a host system memory in a video delivery system as described in;

[0009] Figure 3 According to some embodiments Figure 1 A schematic block diagram generally illustrating a set of buffer layouts and a buffer descriptor table in a host system memory in a video delivery system described in;

[0010] Figure 4 According to some embodiments, Figure 1 A schematic block diagram generally illustrating a packet sequence in a video delivery system as described in;

[0011] Figure 5 According to some embodiments Figure 1 A schematic block diagram of a video streaming protocol packet structure of a video delivery system as described in;

[0012] Figure 6 is a method for tracking the Figure 1 The video transmission system used in Figure 4 Schematic block diagram of a picture buffer mapping of a reception progress of a video transport packet as described in;

[0013] Figure 7 According to some embodiments Figure 1 A schematic block diagram of a video source host processing system and a network interface controller used in the video delivery system described in;

[0014] Figure 8 According to some embodiments Figure 1 A schematic block diagram of a video source host processing system and a network interface controller used in the video delivery system described in;

[0015] Figure 9 According to some embodiments Figure 1 A schematic block diagram of a video destination host processing system and a network interface controller used in the video delivery system described in ; and

[0016] Figure 10 According to some embodiments Figure 1 Schematic block diagram of a network interface controller used in the video delivery system described in .

[0017] The details of various embodiments of the methods and systems are set forth in the accompanying drawings and the description below. DETAILED DESCRIPTION

[0018] The following IEEE standards (including any draft versions of such standards) are hereby incorporated by reference in their entirety and made a part of this disclosure for all purposes: IEEE P802.3 TM and the IEEE Ethernet standard system, including but not limited to LRM, VSR, SR, MR, LR, ZR, and KR. Although the present disclosure may refer to aspects of these standards, the present disclosure is in no way limited by these standards.

[0019] Some embodiments relate to a network interface controller for transmitting pixel data from a video source. The network interface controller includes a host agent interface configured to couple to a video source, and a video transport accelerator configured to receive the pixel data into a local buffer and provide a video transport packet. In some embodiments, the video transport packet includes the pixel data, a media access control header, and a video header.

[0020] Some embodiments relate to a network interface controller for transmitting video data to an image processing system. The network interface controller includes a host agent interface configured to couple to the image processing system directly or via a shared memory resource; and a video transport accelerator configured to receive video transport packets, identify the video transport packets, and control the host agent interface to write pixel data directly to a video frame buffer in a memory of the image processing system. In some embodiments, the video transport packets include pixel data, a media access control header, and a video header.

[0021] Some embodiments relate to a method for transmitting video data from a video source to an image processing system. The method includes receiving pixel data in a local buffer of a first network interface controller and providing a stream of video transport packets. The video transport packets include the pixel data, a media access control header, and a video header. The method also includes receiving video transport packets from a second network interface controller and providing the pixel data to a video frame buffer of the image processing system without involving a host processor of the image processing system and / or requiring additional software to recopy or reformat the pixel data.

[0022] A. Video delivery network

[0023] refer to Figure 1 , a video delivery network includes a video transmitter endpoint 101, a packet data network infrastructure 102 that routes and carries data between connected devices, and a video receiver endpoint 103 that consumes video data. Additional endpoints, such as additional transmitters and receivers, may be part of the video delivery network.

[0024] In some embodiments, video transmitter endpoint 101 includes a video source 111 in the form of a camera or media storage device, and a video enhanced network interface controller (NIC) 112 connected to a physical network medium. Video source 111 is any device or system for providing data (e.g., video streaming data). In some embodiments, video source 111 is any video source, including but not limited to a media player, cable provider, internet subscription service, headend, video camera, set-top box, sensor, storage media server, satellite provider, video recorder, computer, or other source of video material. In some embodiments, video source 111 can be coupled to NIC 112 via a parallel interface, cable, or other connection.

[0025] In some embodiments, video source 111 is a camera or image sensor implemented as a single chip or system on chip (SOC). In some embodiments, video source 111 includes one or more decoding units, a display engine, a transcoder, a processor, and a storage unit (e.g., a frame buffer, a memory, etc.). In some embodiments, video source 111 may also include one or more microprocessors, digital signal processors, general-purpose CPUs, application-specific integrated circuits (ASICs), programmable logic devices, and / or one or more other integrated circuits. Video source 111 may include Figure 1 For example, video source 111 may include additional buffers (eg, input buffers), network interfaces, controllers, memory, input and output devices, and other components for audio / video / data processing.

[0026] Packet data networks can be implemented as direct connections (null networks) or as an infrastructure integrating packet switches, repeaters, and bridges, depending on the desired topology and performance goals. Other devices, such as peripheral units and application- or function-specific processors, can also be part of the network infrastructure and used for application-specific communications.

[0027] Receiver endpoint 103 is a consumer or recipient of video data and contains a video enhancement NIC 113, which transmits the data for image processing and display functions in video host 114. In some embodiments, NIC 112 and / or NIC 113 are capable of extracting pixel payloads and arranging the data into a format suitable for direct processing by an image signal processor, video encoder, or video display (e.g., video receiver endpoint 103). In some embodiments, NIC 112 and / or NIC 113 are also capable of maintaining video data integrity and correcting or concealing data corrupted due to packet errors or loss. In some embodiments, NIC 112 and / or NIC 113 operate with minimal or no host CPU intervention to facilitate packet data processing at multi-gigabit data rates. In some embodiments, the physical layer of NICs 112 and 113 is coupled to packet data network infrastructure 102 via Ethernet twisted pair cabling, fiber optic cabling, or other connections.

[0028] Video host 114 is any device or system that consumes or generates picture data. In some embodiments, video host 114 is an image processing system that includes a monitor, which can be any type of screen or viewing medium for the video signal from video source 111. Video host 114 can be part of or connected to an industrial machine vision system, a studio production system, an advanced driver assistance system (ADAS) system, a home theater, a set-top box unit, a computer, a smartphone, a smart TV, a gaming system, an augmented reality system, a virtual reality system, or other video system.

[0029] In some embodiments, the video host 114 may be implemented as a single-chip integrated circuit or a system on a chip (SOC). In some embodiments, the video host 114 may include one or more decoding units, a display engine, a transcoder, a processor, and a storage unit (e.g., a frame buffer, a memory, etc.). In some embodiments, the video host 114 may include one or more microprocessors, digital signal processors, CPUs, application specific integrated circuits (ASICs), programmable logic devices, or one or more other integrated circuits. The video host 114 may include Figure 1 Other components not shown.

[0030] The video source 111 and the video host 114 can process video streams in a number of formats, resolutions, and bit rates supported by a particular implementation without changing the conceptual hardware architecture and protocol (e.g., different picture resolutions (e.g., 1080p, 4K, or 8K), frame rates (e.g., 60 fps vs. 30 fps), bit precision (e.g., 10-bit vs. 8-bit), Bayer raw pixel frames, or other video characteristics). In some embodiments, the video host 114 is coupled to the video enhanced NIC 113 via wires or other connections.

[0031] The functionality of a video transmitter and receiver, as well as a general-purpose data endpoint, can be combined within the same physical device embodiment, depending on the needs of each application. Different embodiments can address a variety of practical applications. In some embodiments, the protocol exploits the ability of packetized networks to interleave and carry different types of data between the same or different endpoints using the same network infrastructure and physical media.

[0032] B. Ethernet Packet Network

[0033] Some embodiments of the communication system use the following physical layers: IEEE P802.3ch for multi-gigabit (10 Gbps, 5 Gbps, and 2.5 Gbps) systems, IEEE 802.3bp for 1000BASE-T1 (1 Gbps) systems, 802.3bw for 100BASE-T1 (100 Mbps) systems, automotive Ethernet communication systems, or IEEE Ethernet standard systems including but not limited to LRM, VSR, SR, MR, LR, ZR, and KR. Networking endpoints ( Figure 1 The endpoints 101 and 103 of the UE communicate by transmitting data frames (packets) that contain a network-specific header, payload data, and a trailer for verifying the integrity of the frame data. In some applications (e.g., video applications), the payload comprises a pixel data stream. For example, pixel data streams are transmitted in various applications, including industrial machine vision, studio production, advanced driver assistance systems (ADAS), drone technology, simulators, gaming and entertainment systems, augmented or virtual reality systems, and other video applications. In some embodiments, the network or media can be designed to efficiently use Ethernet infrastructure (e.g., Figure 1 The packet data network infrastructure 102 in the IEEE 8025 specification is used as a common and inexpensive medium to carry multiple streams of video and still pictures along with other traditional data and control items simultaneously over the same wiring in industrial and automotive systems.

[0034] The packet data network infrastructure 102 may include any type and / or form of network and may include any of the following: a point-to-point network, a broadcast network, a telecommunications network, a data communications network, or a computer network. The network topology may be a bus, star, or ring network topology. The network may be any such network topology capable of supporting the operations described herein.

[0035] In some embodiments, Figure 1 The network infrastructure 102 does not interpret the content of the data packets. In some embodiments, the network infrastructure only optionally applies traffic shaping or prioritization to a particular packet, but does not modify its specific application content. Therefore, in some embodiments, standard components can be used to implement the network infrastructure 102.

[0036] In contrast, video enhancement NICs 112 and 113 do have knowledge of the video content and implement dedicated hardware or a combination of a programmable controller unit (MCU) and firmware for implementing the operating methods defined in the video transport protocol as explained below according to some embodiments. Some embodiments use the methods discussed below in Section C to augment the packet layout so that the payload pixel stream can be easily extracted from the packet buffer and copied to the frame buffer for use by an image processing system (e.g., Figure 1 The video host 114 in is further utilized.

[0037] When the NIC hardware (for example, Figure 1 When implemented in a video enhancement NIC (e.g., 112 and 113), the systems and methods of some embodiments eliminate or reduce the processing overhead associated with copying from / to frame buffers and operate Ethernet video transport at high network data rates (e.g., 10 Gbit / s and above required for high-definition video). In some embodiments, the video enhancement NIC (e.g., Figure 1 The NICs 112 and 113 in the system offload the general-purpose CPU on the system from the segmentation and reassembly operations and error correction of the video data in software. In some embodiments, the same protocol and hardware can be used to transfer large binary buffers, with all headers automatically stripped and payloads concatenated (e.g., for multi-dimensional binary buffers).

[0038] C. Native Pixel Buffer and Communication Protocol

[0039] In some embodiments, the system connects a video endpoint (e.g., an image sensor or display) to a host processor (e.g., Figure 1 The data is transferred from / to a natively laid out pixel data buffer in memory (e.g., in the form of a continuous video buffer where data is stored in sequential memory locations, representing a rectangular two-dimensional array of pixels). Figure 2 , and compared to other types of data, videos and pictures are typically arranged as large continuous buffers in memory, which in some embodiments has a memory that is more suitable for use by video host 114 ( Figure 1 ) of the image processing and display hardware. In some embodiments, the video frame buffer 350 stores pixel data 354 for the entire image or frame of the video host 114.

[0040] Figure 2 The exemplary and conventional buffer structure shown in FIG includes pixel data 354 corresponding to a raster scan line of length 352, which is placed in sequential locations in memory starting from a software-defined starting address 360. At the end of the scan line, additional padding locations 363 may be reserved to optimize transactions on the memory interface, thereby creating a scan line to row address offset equal to the step size 362. Therefore, the starting address of the next row in memory will be equal to the buffer starting address plus the step size. Memory allocation for subsequent rows will proceed similarly. The allocation of pixel components (RGB, YUV, NV12, etc.) and the number of pixel bits may vary depending on the application, but applications may adhere to a two-dimensional array representation and store pixel data in sequential memory locations. In many embodiments, image signal processors, displays, graphics, and vision processors expect this type of data layout. Alternative data layouts may be utilized.

[0041] According to some embodiments, video frame buffer 350 is larger than the maximum network packet size (e.g., Ethernet packet size), does not contain a required header, and contains contiguous pixels in pixel data 354. In some embodiments, the size of display (e.g., television) line length 352 may also not match the size of the payload.

[0042] refer to Figure 3In contrast to the layout of video frame buffer 350, conventional Ethernet NICs transfer data between packet buffers 308a-f allocated by the network stack software. The set 306 of packet buffers 308a-f is a mix of empty and filled buffers 308a-f managed by the network software. The software maintains a buffer descriptor table 300, whose entries 304a-f point to specific packet buffers 308a-f. The NIC reads buffer descriptor table 300 one by one, and on the transmit path, retrieves packets from buffers 308a-f and passes the contents to the transmitted network packet. On the receive path, the NIC reads the addresses of empty buffers 308a-f from descriptor table 300 and writes data from the received packet to the buffers 308a-f pointed to by the descriptors in entries 304a-f. While this structure is suitable for network data transmission, adjacent buffers 308a-f may contain data for unrelated data streams, and buffers 308a-f may internally contain auxiliary headers and data specific to the communication protocol. Buffers 308a to f cannot be consumed directly by the video processing unit, and the network software needs to balance the set of network buffers 308a to f with the two-dimensional video buffer (e.g., Figure 2 Data is copied between video buffers 350 in .

[0043] In some embodiments, a video-enhanced network interface controller (e.g., NIC 112 and / or 113) Figure 1 )) As described herein, the data carried in the payload of a video transport packet is directly taken from / placed into a continuous frame buffer without copying or reformatting the data in software. The method for encapsulating video and control data into network packets is defined by the Video Streaming Protocol (VSP). Figure 5 and 6 Some embodiments of the protocol mapped to Ethernet IP / UDP transport compatible with the GigE Vision specification are described. Other embodiments of the protocol with similar functionality over other packet network types are also possible. In some embodiments, the hardware implementation of the NIC matches the data encapsulation method defined in the protocol.

[0044] Pixel data is sent in blocks so that the entire block fits within a single Maximum Transmission Unit (MTU) frame of Ethernet, which is traditionally 1500 bytes long. When the video scanline payload is larger than the allowed network block, a sequence of packets is sent that includes a data leader packet 411, one or more video payload blocks or packets 412, and a data trailer packet 413 ( FIG. 4 ). In some embodiments, the block of packets 412, from the data leader packet 411 to the data trailer packet 413, is mapped to a video primitive, which can be a scanline, a block of scanlines, or an entire picture.

[0045] In some embodiments of a VSP packet, the packet begins with a conventional Ethernet UDP header 401, followed by a sequence packet identifier 402, VSP specific protocol data 403, and ends with a standard Ethernet checksum in field 409, as shown in FIG. Figure 5 The VSP payload packet has a similar structure, but carries pixel data in the pixel payload 405.

[0046] The standard IP / UDP connectionless packet structure allows packets to pass through standard Ethernet routers and switches, and uses the UDP header to implement Quality of Service (QoS) policies for video delivery within the network infrastructure. Sequence numbers in the packets allow detection of packet loss within the network infrastructure and, optionally, implementation of retransmission as corrective action.

[0047] If a packet is lost, the next undamaged packet needs to be placed into the video frame buffer 350 ( Figure 2 ) to its desired location in memory, rather than just being placed in the next available memory location. If the VSP payload size is L bytes, and the line length is K bytes, then the address for Group N payload placement will be:

[0048] Address = Start_Addr + int(N*L / K)*stride+frac(N*L / K),

[0049] Where int() is the integer part of the argument, and frac() is the fractional part of the argument, and packets are numbered starting at 0. In other words, first calculate how many whole rows were previously transferred (int(N*L / K)), multiply the stride offset by the factor in the padding bytes, and then add the offset of the fractional data block from the last packet.

[0050] In some embodiments, the ratio R=K / L may be an integer (an integer number of VSP packet payloads fits into one scan line) to remove divider hardware and eliminate fractional offsets for the added benefit of aligned memory access speed improvement.

[0051] Then, the address where the payload of any arbitrary Nth VSP packet is placed will be:

[0052] Address = Start_Addr + N / R * stride

[0053] When R is a power of 2, further simplification is achieved and the division operator can be replaced by a binary shift.

[0054] In some embodiments, damaged packets should not create visible artifacts in the picture if dropped.

[0055] The discovery and configuration of streaming, starting, stopping, and tearing down of video streams is supported by another protocol, the Video Stream Control Protocol (VSCP). In some embodiments, VSCP can be implemented by connectionless messaging at the IP / UDP level. However, other implementations are possible. In some embodiments, a stream receiver (e.g., endpoint 103 ( Figure 1 )) will send control messages in UDP datagrams to obtain a description of the transmitter (e.g., endpoint 101) in XML or other descriptive format, specify the IP address and port of the stream destination (receiver), specify the start time, and trigger the transmission from the video endpoint. Other supplementary commands may be implemented to set parameters of the endpoint and control the camera sensor.

[0056] In some embodiments, VSCP is used to request retransmission of lost packets in the unlikely event of packet loss in the network infrastructure. The entire video primitive consists of block packets, the number of which is incremental ( Figure 4 , groups 411 to 413). Figure 6 In some embodiments, a video primitive is an entire picture represented by the data in the frame buffer 331. However, a video primitive can be smaller than an entire picture. Regions (portions of scan lines) are mapped to a receive status table 332, where each block is represented by a transmit status flag 333.

[0057] The mapping is guided by the ratio R, and the index in the table can be mapped to the packet number N as index = N / R. For example, for two VSP packets per row (R = 2), the index of the tag entry in the table is as follows Figure 4 As displayed.

[0058] The status flag 333 can be "invalid", "OK" or "lost". Before the video primitive transmission starts, all status flags in the transmission status table are initialized to "invalid".

[0059] When the VSP delivers a block packet 412 with packet_ID=N and its FCS field ( Figure 5 , FCS field 409) is correct, the status flag corresponding to position N of this block in the video primitive is set to "OK". If the FCS field 405 is incorrect, or a new packet with a packet_ID greater than N+1 is received, packet loss is detected and the flag 335 of the block is set to the "lost" state. If packet correction by retransmission is enabled, the receiver NIC 113 sends a retransmission request for packet N to the transmitter NIC 112. The latter will attempt to retransmit the lost block packet.

[0060] 4. When this packet is received, the FCS field 409 will be checked and the flags 335 will be updated. When the data trailer packet 413 is received, the video primitive reception is considered complete and the receive status table will end with all "OK" status (table 332) or some flags will remain in the "lost" state (table 334), which will signal that the received video primitive contains uncorrectable errors. The video processing method may optionally use error concealment to reduce the effects of data corruption. The method of concealment is application specific and is outside the scope of this disclosure.

[0061] Common time base and synchronization of pixel clocks on Ethernet video endpoints and hosts can be implemented using the methods and hardware described in the IEEE 1588 standard.

[0062] D. Endpoint Implementation

[0063] According to some embodiments, Figure 4 As shown in the figure, the efficient video network transmission controller builds a valid Ethernet data packet stream from the fragments of the frame buffer, pre-sets the header and control structure specific to the transport method, and appends padding and data integrity check information. Figure 5 to extract the pixel payload using the control structures found in the Ethernet packet shown in

[15] , place the data into a video-specific buffer to recreate the same concatenated contents as the source buffer, and verify the integrity of the delivered pixel data.

[0064] Camera Streaming Endpoint( Figure 7 ) comprises an image sensor 501 with an optical assembly, a video capture interface 502 in the form of a MIPI CSI-2 or alternative camera connection, a data buffer 503 or FIFO with a storage size sufficient to temporarily hold one or more payload blocks, Figure 5 The VSP controller 505 creates packets and passes them to the hardware of the Ethernet media access controller (MAC) and the physical layer (PHY) 506. The PHY 506 is attached to the network medium 508. The buffer 503 receives data from the image sensor 501 with the optical assembly and temporarily stores the data in such a way that the first pixel data will be transmitted when at least the entire video block is stored and can be quickly sent to the network at a network data rate that can be different from and higher than the data rate on the image sensor interface.

[0065] The VSCP controller 504 monitors the fullness of the data buffer 503 and triggers the VSP controller 505 to build a VSP protocol packet when there is enough data available to fill a block. However, although the packet is sent to the network, it is not removed from the buffer 503, but is stored in the shaded portion of the buffer for a period of time to allow for a time window when the block may be retransmitted based on a request from the receiving endpoint.

[0066] Controller 504 also receives VSCP messages from the network. Some messages, such as retransmission requests, can be interpreted in hardware and trigger a retransmission of a lost block if the retransmission request arrives within a time window while the block is still in the shaded area of data buffer 503. Since the size of buffer 503 is limited, the time window for retransmission will also be limited. Some embodiments may have a size sufficient to store 4 to 6 blocks, allowing enough time for round-trip delay in the transmitter and receiver buffers.

[0067] Embedded Memory Streaming Endpoint ( Figure 8 ) is an alternative embodiment and includes a system memory 511 that temporarily stores images generated or acquired by some other element of the system, a VSP controller 515 that feeds data to the network controller, and a PHY 516. The VSCP controller 514 operates in the same manner as the VSCP controller 504. For example, the images in the system memory 511 may be generated by the graphics controller. Although in Figure 8 The pixel data is delivered to the VSP controller 515 from system memory, but the actual transfer mechanism may include direct memory access (DMA) or a system fabric transaction master agent to pull the pixel data from shared memory or resources in the system.

[0068] Some embodiments of the receiver are Figure 9 The data packets are received from the physical media 601 by the PHY and NIC 602. The PHY and NIC 602 demodulates, deserializes, and classifies the packets into VSP 622 and other network 623 categories, and presents them on the corresponding interfaces. The video stream controller 603 will obtain the VSP block packet observing its packet_ID and place the video block payload into the corresponding location in the system memory 605. The frame buffer 350 ( Figure 2 ) to calculate the position. The video stream controller 603 will also use the layout parameters of the reference image (e.g., starting address 360, line length 252, step size 362, number of pixels per frame, etc.). Figure 6 The described tags 333 and 335 simultaneously fill the receive state table 604 ( Figure 6 332 in the hardware embodiment). Upon receiving the data tail packet 413 ( Figure 4), the controller 603 will indicate to the CPU 609 that a new video primitive is available for use by the interrupt request 614. At this point, the software running on the CPU will use the data from the receive status table 604 via connection 615 to make a decision on the need for error concealment. The CPU can then instruct other system components, such as the image processor 607, to operate the desired algorithms on the video primitives and, for example, display a picture on the screen 608.

[0069] The data interfaces 610, 611, and 612 between system components may use, but are not limited to, system interfaces such as direct bus connections, AMBAAXI, PCI Express, HDMI, and DisplayPort. Interfaces for control, status, and interrupt requests may be implemented in various ways, including, but not limited to, direct wired connections, register read / write access in software, or message signaling interrupts. When the receiver is implemented as a chip or board, the configuration may have functional units on the chip or module, and Figure 9 The other functional units shown in the figure may be part of a computer system or a host device.

[0070] In some embodiments, such controllers are designed to build standard Ethernet packets while transmitting and extract video data from received packets into a video buffer with minimal or no host CPU intervention, eliminating the need to copy data between the network and the video buffer. In some embodiments, the video-enhanced network interface controller can be designed to further be able to conceal or correct errors on the data transport medium without intervention by a host processor (e.g., a central processing unit (CPU)).

[0071] refer to Figure 10 , a network interface controller (NIC) 900 includes a network physical layer interface 902, a network media access and controller (MAC) 904, a transmission control protocol / Internet protocol (TCP / IP) accelerator 906, a video transport accelerator 908, and a host agent interface 912. In some embodiments, the network interface controller (NIC) 900 is provided to provide high-speed video transport. In some embodiments, the network interface controller 900 may be provided with or coupled to the video host 114 or the video source 111. In some embodiments, a separate network interface controller is associated with each of the video host 114 or the video source 111.

[0072] The video enhancements described herein may be implemented using a video transport accelerator 908 within an Ethernet network controller, with interface 902 and network media access controller 904 used for p-serial data conversion, address filtering, QoS policy enforcement, and FCS generation and checking, and reusing host agent interface 912. Accelerator 908 may implement video transmitter or video receiver functionality, or both, depending on application requirements.

[0073] The media or network is attached to the physical layer interface 902, which is configured for the type of transmission media and the speed associated with the media. The physical layer interface 902 isolates physical media dependencies (PMDs) and allows the rest of the network interface controller 900 to render all other layers media-independent. In some embodiments, the physical layer interface 902 includes transmit and receive ports coupled to a 10GBaseT Ethernet or other network. The host agent interface 912 provides system interconnection to the video host 114 or video source 111. The host agent interface 912 can be a peripheral component interconnect (PCI) or PCI Express.

[0074] The media access controller 904 is a layer for converting data structures into signals of the physical layer interface 902 for interacting with the network. The media access controller 904 is integrated with the physical layer interface 902 and is implemented in hardware in some embodiments.

[0075] refer to Figure 5 and 10 The network media access controller 904 provides a video data path including a video delivery accelerator 908 for processing video data in the internal buffer of the network interface controller 900. The video delivery accelerator 908 controls the host agent interface 912 to receive and transmit specific delivery packets (e.g., video stream data packets, such as Figure 5 The video stream data packet 407 in the video frame buffer 350 is read and written directly to the video frame buffer 350, such as Figure 2 . The video frame buffer 350 contains pixel data 354 (represented as, but not limited to, RGB or YUV samples) as a sequence of television scan lines. Pixel data 354 is placed in sequential locations within the video frame buffer 350. N pixel rows constitute the entire video frame buffer 350, where N is an integer. In some embodiments, a more efficient hardware implementation may require padding with padding locations 363 at the end of the row length 352 so that the initial bus transaction is aligned on a power-of-two boundary. The video frame buffer 350 includes a starting address 360 (e.g., the address of the first pixel in memory), a scan line length (e.g., the number of pixels in a scan line), a stride 362 (the offset from the starting address of the first row to the starting address of the next row), and a number of rows (e.g., 1 to N). When software configures the video delivery accelerator 908, the starting address 360, scan line length 352, stride 362, and number of rows (e.g., 1 to N) are provided to the video delivery accelerator 908.

[0076] refer to Figure 5 and 10For example, a video transport packet, such as video stream data packet 407, includes a media access controller header, a UPD header, a video stream header 402, a pixel payload 405, and a frame checksum field 409. The video stream header 402 includes a protocol and version field, a sequence number field, and an extension field. Pixel data is carried in the pixel payload 405. In some embodiments, the video stream data packet 407 can be distinguished from other Ethernet packets by its destination address, protocol ID, IP port, or similar attributes, which instruct the network media access controller 904 to route the video stream data packet 407 to the video transport accelerator 908 rather than, in some embodiments, routing it to a non-video path for data packets.

[0077] In the delivery path, the video transport packet is generated by the video transport accelerator 908 on the transmission side by reading the data block for transmission from the frame buffer, building a network packet (e.g., a video transport packet) in hardware, and passing the network packet to the media access controller 904 ( Figure 2 ) is established. The receiving video delivery accelerator 908 strips off the header and passes only the pixel payload 405 to the receiving frame buffer. In some embodiments, the standard Ethernet packet payload contains a video header (header 402), which further defines the video delivery accelerator protocol version and flag, packet sequence number, and future extension bits. For example, the protocol may define a version and sub-version. The protocol may have associated negotiation parameters such as pixel format (RGB, NV12, YUV4:4:4, etc.), number of bits per pixel, and number of pixels per row. Using these parameters, in some embodiments, the position of each pixel in the Ethernet packet payload can be easily identified. The selected mapping will be used to configure the video stream controller hardware.

[0078] In a video primitive, the sequence number in the sequence number field defines the current number of the video stream data packet 407. In some embodiments, the sequence number starts at zero at the beginning of each video primitive, which can be an entire picture. In some embodiments, the payload length is mapped to the number of pixels per line to start each new line in a new packet, for example, dividing a scan line into two video transport packets. The total number of video packets in a frame buffer will be M, where M is an integer. In some embodiments, the user selects the most appropriate number of pixels per packet based on the pixel data format.

[0079] If any packets are lost (e.g. Figure 6If the video delivery accelerator 908 detects this condition and proceeds to take corrective action, the corrective action may include: automatic retransmission request: The receiving video delivery accelerator 908 sends a request to the transmitter to resend the lost packet. In some embodiments, this request is sent after a predefined timeout to clear any queues in the network infrastructure. The retransmitted packet replaces the empty space in the frame buffer, and the successfully received frame buffer can be reported to the software via an asserted interrupt. In some embodiments, the corrective action may include: correction via forward error correction: Redundant information can be used to recreate the lost packet. In some embodiments, methods with overlapping transmissions (such as a digital fountain) can be used with or without software assistance. In some examples, the packet loss probability is expected to be low enough not to burden the CPU with this task. In some embodiments, the corrective action may include: error concealment: Visually sufficient error concealment can be implemented by replacing the lost pixel block with the average of adjacent scan lines. When the entire buffer is received and (optionally) corrected, an interrupt may be asserted to the CPU to indicate the need for action, such as a display buffer rollover.

[0080] In some embodiments, the video delivery accelerator 908 in the transmission path receives the video frame buffer 350 ( Figure 2 ), where the parameters are the starting address 360, the scan line length 352, the step size 362, and the number of lines (e.g., 1 to N). When the video frame buffer 350 is ready, the host processor (e.g., CPU) of the video source 111 will start the transfer by writing the sequence number (=0) into the configuration register.

[0081] The video delivery accelerator 908 sequentially transfers the data to a local buffer and constructs a video stream data packet 407 by prepending the media access control header 401 and the video stream header 402. This operation can be implemented in hardware in a sequential streaming mode. Once the header is passed to the media access controller 904, the data segments read from the frame buffer are transmitted to the media access controller 904. At the end of the segment, a cyclic redundancy check (e.g., CRC32) is appended to the packet in field 409, and the transmission is temporarily stopped. If the sequence number is less than the maximum sequence number M, the transmission cycle is repeated for the next frame buffer segment. At the end of the buffer transfer, an interrupt is asserted to the processor, indicating that the video delivery accelerator 908 is ready to transmit the next frame.

[0082] The transmit and receive paths of the video delivery accelerator 908 optionally contain an approved virtual lossless compression engine, such as VESA DSC standard (DSC v.1.2) compression. In some embodiments, compression is more efficient in a YUV 4:2:2 frame buffer layout. Compression can be negotiated by an initialization protocol implemented in software via a conventional network stack.

[0083] The video delivery accelerator 908 may optionally include content protection hardware (e.g., compliant with the latest HDCP specifications). The minimum hardware should then include an approved encryptor in the transmit path and a matching decryptor in the receive path. The security and other parameters of these engines can be provided by software running on the host or an appropriate embedded microcontroller, and their operation is outside the scope of this document.

[0084] While the examples of communication systems described above may include devices operating according to Ethernet and other standards, it should be understood that embodiments of the described systems and methods may operate according to alternative standards and utilize wireless communication devices other than those configured as devices. For example, multi-unit communication interfaces associated with cellular networks, satellite communications, vehicular communication networks, wired networks, and networks may utilize the systems and methods described herein to achieve improved overall capacity and / or link quality without departing from the scope of the systems and methods described herein.

[0085] It should be noted that certain paragraphs of this disclosure may refer to terms such as "first" and "second" that relate to devices, operating modes, transmission chains, etc., to identify or distinguish one from another or from another. These terms are not intended to relate entities (e.g., a first device and a second device) solely in time or according to sequence, although in some cases, these entities may include such a relationship. These terms also do not limit the number of possible entities (e.g., devices) that can operate in a system or environment. The terms coupled or connected include indirect and direct couplings and connections.

[0086] It should be understood that the systems described above may provide multiple or each of any of those components, and that these components may be provided on a standalone machine or, in some embodiments, on multiple machines in a distributed system. Additionally, the systems and methods described above may be provided as one or more computer-readable programs or executable instructions embodied on or in one or more articles of manufacture. The articles of manufacture may be a floppy disk, hard disk, CD-ROM, flash memory card, PROM, RAM, ROM, or magnetic tape. Generally, the computer-readable program may be implemented in any programming language (e.g., LISP, PERL, C, C++, C#, PROLOG) or any bytecode language (e.g., JAVA). The software program or executable instructions may be stored as object code on or in one or more articles of manufacture.

[0087] While the foregoing written description of the methods and systems enables one skilled in the art to make and use what is presently believed to be the best mode thereof, it should be understood and appreciated by those skilled in the art that variations, combinations, and equivalents of the specific embodiments, methods, and examples herein exist. The methods and systems should therefore not be limited by the embodiments, methods, and examples described above, but rather by all embodiments and methods within the scope and spirit of the present disclosure. The headings provided herein are non-limiting.

[0088] The network interface controller has been described above with the aid of functional building blocks that illustrate the performance of certain important functions. For ease of description, the boundaries of these functional building blocks have been arbitrarily defined. Functions and structures may be integrated together across such boundaries. Alternative boundaries may be defined as long as certain important functions are appropriately performed. Similarly, flow chart blocks may also be arbitrarily defined herein to illustrate certain important functionality. Within the scope of use, flow boundaries and sequences may be defined in other ways and still perform certain important functionality. Therefore, such alternative definitions of functional building blocks and flow chart blocks and sequences are within the scope and spirit of the claimed invention. Those skilled in the art will also recognize that the functional building blocks and other illustrative blocks, modules, and components herein may be implemented as described or by discrete components, application specific integrated circuits, processors executing appropriate software, etc., or any combination thereof.

Claims

1. A network interface controller for use with a packet network for transmitting video data to an image processing or display system, the network interface controller comprising: Media Access Controller MAC; a host agent interface configured to transfer the non-video transport packets to a network buffer for further processing in a network stack; and a video transport accelerator configured to receive video transport packets from the media access controller, extract video payloads from the video transport packets, and control the host agent interface to write pixel data directly into a video frame buffer of the image processing or display system, wherein the video transport accelerator is configured to resolve errors by:

1. concealing errors due to packet corruption by saving previously received data in the video frame buffer or by copying adjacent scanline contents into the corrupted data block; or 2. concealing the errors due to packet corruption by interpolating lost pixels from surrounding valid pixels.

2. The network interface controller of claim 1 , wherein the host agent interface transfers non-video transport packets to a network buffer for further processing in the network stack, and the host agent interface directly writes the pixel data without requiring host processor software to recopy or reformat the pixel data.

3. The network interface controller of claim 1 , wherein the packet network is a high-speed Ethernet network, and the network interface controller is configured to transmit the mix of general data and the video transport packets to a network buffer for processing by network stack software. 4 . The network interface controller of claim 1 , wherein the video transport packet comprises a sequence identifier field for indicating a sequence number of the video transport packet.

5. The network interface controller of claim 3, wherein the video delivery accelerator is configured to provide a map of corrupted or lost data blocks in the video frame buffer to facilitate software corrective action on the corrupted or lost data.

6. The network interface controller of claim 1, wherein the video delivery accelerator is configured to correct the errors due to packet damage and loss by implementing automatic generation of a control message to a transmitter side to request retransmission of damaged pixel data blocks.

7. The network interface controller of claim 1, wherein the host agent interface comprises at least one of a PCI Express, AMBA AHB, or AXI interface to the image processing or display system.

8. The network interface controller of claim 1, wherein the video delivery accelerator is configured to provide a map of corrupted or lost data blocks in the video frame buffer to facilitate software corrective action on the corrupted or lost data.

9. A method for transmitting video data to an image processing or display system using a packet network, the method comprising: transferring, via a host agent interface of the network interface controller, the non-video transport packets to a network buffer for further processing in a network stack; and A video transport accelerator of the network interface controller receives video transport packets from a media access controller of the network interface controller, extracts a video payload from the video transport packets, and controls the host agent interface to write pixel data directly into a video frame buffer of the image processing or display system, wherein errors are resolved by the video transport accelerator by:

1. concealing errors due to packet corruption by saving previously received data in the video frame buffer or by copying adjacent scan line contents into a corrupted data block; or 2. concealing the errors due to packet corruption by interpolating lost pixels from surrounding valid pixels, and wherein the packet network is a high-speed Ethernet network and the network interface controller is configured to transmit a mix of general data and the video transport packets to a network buffer for processing by network stack software.

10. The method of claim 9, wherein the host agent interface comprises at least one of a PCI Express, AMBA AHB, or AXI interface to the image processing or display system.

11. The method of claim 9, wherein the video delivery accelerator is configured to correct the errors due to packet damage and loss by implementing automatic generation of a control message to a transmitter side to request retransmission of damaged pixel data blocks.

Images