Serialized video transmission

Abstract

An apparatus for processing a serialized video stream and related aspects are disclosed. For example, the apparatus (302) may comprise: a receiver (304) configured to receive a serialized video stream comprising a plurality of serially transmitted video frames from a video source (102), each video frame comprising a plurality of frames, each frame comprising a plurality of frame lines, each frame line comprising a set of active pixels and blanking pixels; at least one buffer (305) arranged to buffer at least effective pixels of the received serialized video stream until pixels of a complete frame line are received; one or more processors or processing circuitry (308) configured to, for pixels of each buffered complete frame row, assign a frame row position indicator to an effective pixel group comprising effective pixels of the buffered complete frame row, and generate a data structure comprising the effective pixel group and the frame row position indicator assigned to the effective pixel group, each data structure comprises a set of effective pixels from the received frame line, and at least one blanking pixel of the received frame line is omitted; and a transmitter (310) configured to forward transmit an outgoing serialized video stream comprising the generated data structure; and after transmitting a data structure comprising a set of valid pixels and the assigned frame line position indicator, suspending forward transmission of the outgoing serialized video stream between the data structures until a next data structure comprising a different set of valid pixels and a different frame line position indicator is ready for transmission.

Description

Technical Field

[0001] The present disclosure relates to serialized video transmission, apparatus and methods for intermittently forwarding serialized video frame rows received as a continuous stream of pixels from at least one video source to a viewing device, and related aspects.

[0002] Specifically, but not exclusively, the present disclosure relates to a device configured to discard one or more blanking pixels from each received video frame row, the device pausing transmission after transmitting a group of pixels including active pixels from the video frame row and zero or more remaining blanking pixels until the pixels of the next video frame row are ready to be transmitted forward.

[0003] Specifically, but not exclusively, some embodiments of the disclosed technology relate to a system in which video from two or more sources is streamed in real time with minimal latency to one or more remote displays (such as near-eye displays of a head-mounted device) to achieve synchronized low-latency output.

[0004] Some aspects and embodiments of the disclosed technology may be particularly useful, but not limited to, use cases requiring highly accurate and reliable real-time video streams from two or more independent asynchronous sources to be reconstructed on a receiver display with minimal latency. Such images may be used in a variety of applications, particularly those that provide some visual indication of perspective and / or depth in an image, such as a stereoscopic display may provide.

[0005] For example, some embodiments of the disclosed technology can be used in a medical setting, such as for surgical procedures, where a surgeon or similar medical operator is performing or assisting in a surgical procedure while viewing a real-time video clip displayed on a near-eye display or screen near the surgical site. Such a display can advantageously allow for images of a patient undergoing surgery, where one or more image modes allow for the differentiation of healthy tissue from unhealthy tissue, body parts from organs, and a magnified view of a particular area being viewed and undergoing surgery.

[0006] However, this use environment requires that the video feeds from at least two video sources have very low latency, high and stable synchronization, in addition to requiring any head-mounted device type display to be lightweight and comfortable to wear. The synchronization of images from different sources is extremely important, because any loss or jitter in synchronization in the real-time video streams can cause serious problems during surgery. For example, if the left-view near-eye video stream is not synchronized with the video stream including the right-view near-eye image. Even if video is received from a single video source, it is important to manage the data flow to avoid problems such as buffer overflow when processing the video for forward transmission from the source to the display device. Background Art

[0007] Displays using liquid crystal display (LCD) panels can be designed to follow industry standard video standards, such as high definition (HD), ultra high definition (UHD). These standards require strict compliance, they establish a set number of vertical and horizontal pixels in each vertical and horizontal line of the video frame and specify vertical and horizontal blanking intervals (which include "pixels" not intended for display), as well as specifying standards for transmission rates.

[0008] In a raster scan display, the vertical blanking interval (VBI) is the time between the end of the last visible line of a frame or field and the beginning of the next visible line of the next frame. Although modern thin panel displays (such as liquid crystal displays and organic light emitting diode (OLED) displays) do not require a blanking interval, for legacy reasons, they are still configured to receive a video stream including a blanking interval, which includes a plurality of blanked pixels. The terms "blanked pixels" or "blank intervals" are known in the art to include data that is not intended for display. In other words, a "blank interval" or "blank pixel" does not necessarily carry no information.

[0009] Video standards also place strict requirements on transmission and clock synchronization timing, for example, the timing further requires that the LCD panel receiver clock be synchronized with the video source clock to ensure that the image is displayed correctly. When a source video (e.g., a camera output video signal) is delivered using a common standard such as the Serial Digital Interface (SDI) standard, the SDI signal is configured in a way that the video source clock is forwarded from the video source to the LCD panel for synchronization.

[0010] Each SDI cable can only forward the clock of a single video source. Thus, problems arise when using a single SDI cable to transmit video from two or more asynchronous video sources, such as, for example, video clips that may specifically require perspective or depth.

[0011] The current solution is to provide an intermediate video frame buffer that decouples the input video clock from each video source and synchronizes the SDI transmission to a single clock that is independent of any one source. However, adding an intermediate video frame buffer can require hardware reconfiguration and increase the latency between one or two frames in the video output at the LCD panel display. This level of latency is undesirable when the video source is a real-time stream for interactive control (e.g., surgery). When video from a single source is routed for onward transmission via an intermediate hub, similar latency issues may occur when displaying the video due to possible buffer overflow issues at the hub.

[0012] The disclosed techniques seek to mitigate, avoid, alleviate or eliminate various problems known in the art that affect the transmission of one or more video streams to an LCD or LCD-type display, such as, for example, the problems mentioned above. Summary of the invention

[0013] While the invention is defined by the appended claims, various aspects of the disclosed technology (including the claimed technology) are set forth in this abstract section, along with examples of some preferred embodiments and indications of possible technical benefits.

[0014] A first aspect of the disclosed technology includes an apparatus for processing a serialized video data stream for forward transmission, the serialized video data stream comprising a plurality of serially transmitted video frames from a video source, each video frame comprising a plurality of frame rows, at least one frame row comprising a group of valid pixels and one or more blanking pixels, the apparatus comprising: a receiver configured to receive the serialized video data stream; a memory arranged to buffer the received serialized video data until at least all valid pixels of the frame rows of pixels of the received video frame have been buffered; one or more processors or processing circuit systems configured to generate a data structure for each group of valid pixels of the frame rows of pixels, the data structure comprising at least a valid pixel group of the frame row and a frame row position indicator assigned to the valid pixel group, each data structure comprising a group of valid pixels from the received frame row and omitting at least one blanking pixel of the received frame row; and a transmitter configured to intermittently forward transmit the generated data structures by pausing transmission after each data structure is transmitted until the next data structure is ready to begin transmission.

[0015] Advantageously, in some embodiments, latency delays can be reduced because instead of buffering the received data stream at the device until an entire video frame is received, the receiver buffer is configured to clear after each complete video frame row is received without waiting for the entire video frame to be received.

[0016] In some embodiments, more than one blanked pixel is discarded from each received video frame line that includes a group of valid pixels and at least one blanked pixel.

[0017] In some embodiments, any blanked pixels of a received video frame row are not included in the outgoing serialized video stream comprising the data structure for forward transmission.

[0018] Advantageously, by removing one or more or all blanking pixels from each video frame line, the transmission time of each video frame line to the viewing device is reduced due to the reduced number of blanking pixels that need to be transmitted forward.

[0019] In some embodiments, the intermittently transmitted video data stream has a fixed frame line length and a variable transmission pause duration.

[0020] Advantageously, by pausing transmissions between groups of pixels at the device, any transmitter clock used to serialize video frames at the video source can be decoupled from any receiver clock used at the viewing device.

[0021] Advantageously, at the viewing device, the variable transmission pause duration may be compensated for by adding a variable number of blanking pixels so as to be output to the viewing device as a continuous stream of variable line length video data.

[0022] In some embodiments, the variable number of blanked pixels depends on the transmission duration of each forwarded data structure to the viewing device.

[0023] In some embodiments, the frame row indicator assigned to each group of valid pixels is included in a header of a data structure including the group of valid pixels.

[0024] Another, second aspect of the disclosed technology includes a viewing device, the viewing device including a receiver configured to receive a stream of intermittently transmitted serialized video, the stream of serialized video including multiple data structures, each data structure including at least one group of valid pixels from a video frame row and a frame row position indicator assigned to the group of valid pixels; and one or more processors or processing circuit systems, the one or more processors or processing circuit systems configured to generate a complete row of video data for each received data structure by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure including the next pixel group in the serialized video stream is received, and cause each complete row of video data to be output for presentation on a display at a position in the video frame indicated by the frame row position indicator of the valid pixels assigned to the complete row of video data.

[0025] In some embodiments, a serialized video stream is received from an apparatus for processing a serialized video data stream for forward transmission according to the first aspect disclosed herein or any one of its embodiments, wherein the received intermittently transmitted serialized video data stream is processed by one or more processors or processing circuit systems to add a variable number of blanking pixels so as to be output to a display as a continuously serialized variable line length video stream.

[0026] In some embodiments, the viewing device also includes a display.

[0027] In some embodiments, the viewing device may include more than one display and split the received video stream into two or more streams. One or more of these streams may be external to the viewing device. For example, in some embodiments, a large flat panel display may be used to display the same image as that displayed on the near-eye display, for example, if the large screen can be driven in the same manner as the near-eye display.

[0028] In some embodiments, the viewing device comprises means for processing a serialized video stream for onward transmission according to the first aspect disclosed herein or any one of its embodiments.

[0029] Advantageously, reducing the number of transmitted blanked pixels may reduce the likelihood of buffer overflow at the receiving viewing device.

[0030] Another, third aspect of the disclosed technology includes a method for processing a serialized video stream for forward transmission, the method comprising: receiving a serialized video stream comprising a plurality of serially transmitted video frames from a video source, each video frame comprising a plurality of frames, each frame comprising a plurality of frame rows, the frame rows comprising a group of valid pixels and blanking pixels; buffering at least valid pixels of the received serialized video stream until pixels of an entire frame row are received; and processing pixels of each buffered entire frame row by assigning a frame row position indicator to a valid pixel group comprising valid pixels of the buffered entire frame row; and generating a data structure comprising the valid pixel group and comprising a frame row position indicator assigned to the valid pixel group, each data structure comprising a group of valid pixels from the received frame row and omitting at least one blanking pixel of the received frame row; intermittently transmitting forward outgoing serialized video data comprising the generated data structure when each data structure is ready for transmission; and pausing forward transmission of the outgoing serialized video data between the data structures.

[0031] For example, in some embodiments, pausing forward transmission of outgoing serialized video data includes, after transmitting a data structure including a set of valid pixels and assigned frame row position indicators, until a next data structure including a different set of valid pixels with different frame row position indicators is ready for transmission.

[0032] In some embodiments, each data structure that includes a set of valid pixels from a received frame line omits all blanked pixels for that received video frame line in the outgoing serialized video stream that includes the data structure transmitted forward.

[0033] In some embodiments, the forward intermittently transmitted video data stream has a fixed frame line length and has a variable transmission pause duration.

[0034] Another, fourth aspect of the disclosed technology includes a method for processing a received intermittent serialized video stream for display, the method comprising receiving the received serialized video stream comprising a plurality of data structures, each data structure comprising at least one group of valid pixels from a video frame row and a frame row position indicator assigned to the group of valid pixels; generating a complete row of video data by adding one or more blanking pixels to each valid pixel group of the received data structures until a data structure comprising the next pixel group in the serialized video stream is received; and causing each complete row of video data to be output for presentation on a display at a position in the video frame indicated by the frame row position indicator of the valid pixels assigned to the complete row of video data.

[0035] In some embodiments, an intermittent serialized video stream is received by an apparatus for processing a serialized video stream for onward transmission according to the first aspect or any one of the embodiments disclosed herein.

[0036] In some embodiments, the received intermittently transmitted serialized video data is processed by one or more processors or processing circuitry (1002) to add a variable number of blanking pixels for output to a display as a continuously serialized variable line length video stream.

[0037] In some embodiments, the method further includes determining that a second to last frame row of a frame has been received, generating a frame row including blanked pixels as a last frame row of the frame, and outputting the frame row of the frame as a serial data stream of pixels to a display.

[0038] Another, fifth aspect of the disclosed technology includes a computer program product comprising computer code which, when loaded from a memory and executed by one or more processors of an apparatus for processing a serialized video stream for forward transmission of the aspects or any one of the embodiments disclosed herein, causes the apparatus to implement any one of the disclosed aspects or embodiments of the method for processing a serialized video stream for forward transmission.

[0039] Another, sixth aspect of the disclosed technology includes a computer program product, which includes computer code, which, when loaded from a memory and executed by one or more processors of a viewing device of the aspects or any embodiment disclosed herein, causes the viewing device to implement any of the disclosed aspects or embodiments of the method of processing a received video stream.

[0040] Another, seventh aspect of the disclosed technology includes a signal generated by an apparatus for processing a serialized video stream for forward transmission of any of the disclosed aspects or embodiments, wherein the signal includes a serialized video stream of data structures, each data structure including at least one group of valid pixels from a video frame row and a frame row position indicator assigned to the group of valid pixels.

[0041] In some embodiments, the outgoing serialized video stream including the forwarded data structure does not contain any blanked pixels.

[0042] In some embodiments, an outgoing serialized video stream comprising forward transmitted data structures includes in each data structure at least one blanked pixel having a number less than the number of blanked pixels of a received video frame row comprising an active pixel group of the data structure.

[0043] Another, eighth aspect of the disclosed technology includes a video display system, wherein the video display system includes an apparatus for processing a serialized video stream for onward transmission of an aspect or any embodiment disclosed herein and a viewing device of an aspect or any embodiment disclosed herein.

[0044] In some embodiments, the apparatus for processing the serialized video stream for onward transmission further comprises a video source.

[0045] In some embodiments, the viewing device further comprises at least one display configured to present video received via the device from a video source.

[0046] Another, ninth aspect of the disclosed technology includes a computer-readable storage medium, which includes a computer program code, which, when executed by one or more processors or processing circuit systems of an apparatus for processing a serialized video stream for forward transmission of an aspect or embodiment disclosed herein, enables the apparatus to implement a method for processing a serialized video stream for forward transmission of an aspect or embodiment of the method disclosed herein.

[0047] Another, tenth aspect of the disclosed technology includes a computer program carrier, which carries a computer program including a computer program code, which, when loaded from the computer program carrier and executed by one or more processors or processing circuit systems of the device for processing a serialized video stream for forward transmission of the aspects or embodiments disclosed herein, causes the device to implement the method for processing a serialized video stream for forward transmission of the aspects or embodiments disclosed herein, wherein the computer program carrier is one of an electronic signal, an optical signal, a radio signal or a computer-readable storage medium.

[0048] Another aspect of the disclosed technology comprises means for performing a method according to any of the method aspects disclosed above.

[0049] Another, eleventh aspect of the disclosed technology includes an apparatus for combining multiple serialized video streams from different video sources for serialized forward transmission through a channel, the apparatus comprising: a receiver configured to receive each serialized video stream separately through a separate channel; a plurality of buffers, each buffer being arranged to buffer valid pixels of a received frame row of a serialized frame in at least the received video stream; one or more processors or processing circuit systems configured to, for each group of valid pixels in a completely received row of pixels in a received frame of the received serialized video stream, assign a video source indicator to the valid pixel group and generate a data structure including the valid pixel group, each data structure omitting at least one blanking pixel of the completely received frame row, a source indicator assigned to the pixel group, and a frame row position indicator of the valid pixel group; and a transmitter configured to forward the generated data structure of two received serialized video streams as serialized video data through a common channel.

[0050] In some embodiments, the video device comprises a hub that is co-located with the surgical imaging device.

[0051] In some embodiments where the hub comprises a hub co-located with the surgical imaging device, the hub is configured to receive serialized video streams from two or more video sources (eg, left view and right view video sources).

[0052] In some embodiments, each video source comprises a camera and each camera is part of a surgical imaging device.

[0053] In some embodiments, the hub is, or is part of, a surgical imaging device including a surgical microscope.

[0054] In some embodiments, the hub is co-located with the surgical microscope.

[0055] In some embodiments, the hub is part of the control system or provided as another computing component or unit of the surgical microscope.

[0056] In some embodiments, the method further comprises assigning a frame row position indicator to each valid pixel group prior to forwarding it, however, in alternative embodiments, the method may instead comprise retaining the frame row position indicator associated with the valid pixel group.

[0057] In some embodiments, after a data structure from one video source has been transmitted, the sender pauses transmission until the next data structure from another video source or from the same video source is ready to begin transmission.

[0058] In some embodiments, the received video data stream has a fixed line length and the transmission pause length between forward transmitted data structures varies.

[0059] In some embodiments, the transmitter transmits onward the data structures generated from the multiple sources as a continuous stream of combined video data.

[0060] Advantageously, in some embodiments, latency delays can be reduced because, instead of buffering the received data stream at the device until an entire video frame is received from a video source, the receiver buffer is configured to clear after each complete video frame row is received without waiting for the entire video frame to be received.

[0061] Advantageously, by pausing transmissions between groups of pixels at the device, any transmitter clock used at the video source to serialize pixels forming video frame data may be decoupled from any receiver clock used at the viewing device to extract pixel data.

[0062] Advantageously, by decoupling the clock at the video source and the viewing device, the transmission delay for multiple serialized video streams can be reduced. In addition, the receiver circuitry can be simpler because there is no need to use a phase-locked loop to derive the same frequency as the source. This allows multiple serialized video streams to be presented with better synchronization when displayed concurrently.

[0063] In some embodiments, the video source indicator identifies a spatial relationship between at least two of the plurality of video sources.

[0064] In some embodiments, the plurality of serialized video streams include a video stream for a left view of a scene obtained from a video source and a video stream for a right view of the scene obtained from the video source, wherein the video source indicator identifies whether a pixel group belongs to a frame of video for the left view or the right view of the scene obtained from the video source.

[0065] In some embodiments, the allocated video source indicator and frame line position indicator are included in a header of the data structure, wherein the bit size of the header matches the parallel width of a serializer of a transmitter configured to combine data structures from multiple different video sources into one outgoing serialized video stream.

[0066] In some embodiments, data structures from two sources are interleaved by the serializer in an outgoing serialized video stream.

[0067] In some embodiments, the transmitter is configured to suspend forward transmission between data structures after transmitting a data structure including a set of valid pixels having a first video source indicator until a data structure having the same or a different video source indicator is ready for transmission.

[0068] In some embodiments, interleaving alternates data structures from one video source with data structures from another video source in an outgoing serialized video stream, however, this may not always occur and one or more frame rows may be sent continuously from one source if no other frame rows are available for transmission.

[0069] In some embodiments, at least one received serialized video stream is received via a channel having one or more of a data rate and a video resolution that is different than a data rate or a video resolution of at least one other channel receiving another received video stream.

[0070] In some embodiments, any blanked pixels of a received video frame row are not included in the outgoing serialized video stream comprising the data structure for forward transmission.

[0071] However, in some embodiments, only one or more blanked pixels are removed and any remaining blanked pixels for a received video frame line are included in the onward transmitted data structure of the outgoing serialized video stream.

[0072] Advantageously, by removing one or more or all blanking pixels from each video frame line, the transmission time of each video frame line to the viewing device is reduced because the number of blanking pixels to be transmitted forward is reduced. In addition, since it is not necessary to buffer an entire frame of video, but rather only an entire line of a video frame is buffered at the transmitter prior to output, the likelihood of a buffer overflow at the receiver of the viewing device can be reduced. In combination with having no blanking pixels or a reduced number of blanking pixels in the received transmission stream and / or if the data structure is only transmitted intermittently, the likelihood of a buffer overflow at the combiner device and the viewing device is reduced.

[0073] In some embodiments, the video combiner device may decouple the source clock of the received serialized video stream from the outgoing serialized video stream by varying the duration of the pauses between the forward transmitted data structures or by varying the duration of the transmission of the forward transmitted data structures.

[0074] In some embodiments, the transmitter is configured to transmit the outgoing serialized video stream over a data communication channel along a wired link and / or over a wireless data communication channel to a device configured to cause display of the received video data.

[0075] In some embodiments, the weight of the viewing device (e.g., if it is a head mounted device display) can be reduced by using only one wired link with a video combiner device to carry serialized video data from multiple serialized video sources. This can make the head mounted device less cumbersome for the wearer.

[0076] Another, twelfth aspect of the disclosed technology includes a viewing device comprising: a receiver configured to receive a serialized video stream including a plurality of data structures, each data structure including an effective pixel group, a source indicator assigned to the pixel group, and a frame row position indicator assigned to the effective pixel group; and one or more processors or processing circuit systems configured to generate a complete row of video data by adding one or more blanking pixels to each effective pixel group of the received data structures until a data structure including the next pixel group in the serialized video stream is received, and to cause each complete row of video data to be output for presentation on a display associated with the source indicator of the complete row of video at a position in the video frame indicated by the frame row position indicator of the effective pixel assigned to the complete row of video data.

[0077] In some embodiments, the video data output to one or more displays for multiple complete frame lines includes variable line length video data, in other words, the video data output to the display does not have a fixed frame line length of active pixels and blanked pixels because the number of blanked pixels can vary.

[0078] Advantageously, variable frame line length video data allows the data output to one or more displays to be a continuous stream(s) of pixels and blanked pixels.

[0079] In some embodiments, one or more displays include a flat panel display, for example, a display based on LCD or OLED technology configured to receive serialized video data having a raster-type display format.

[0080] In some embodiments, the received serialized video stream is received from an apparatus for combining a plurality of serialized video streams from different video sources for serialized forward transmission through a channel according to the eleventh aspect or any one of the embodiments disclosed herein.

[0081] In some embodiments, the video combiner device also includes a video source.

[0082] In some embodiments, the viewing device further comprises at least one display configured to present video received from a video source via the device.

[0083] Another, thirteenth aspect of the disclosed technology includes a method for combining multiple serialized video streams from different video sources, each serialized video stream is received through a different communication channel for serialized forward transmission through the channel, the method comprising: individually receiving each of the multiple serialized video streams; individually buffering valid pixels of received frame rows of serialized frames of at least each of the multiple received video streams; processing each buffered valid pixel group in a fully received frame row by assigning at least a video source indicator to the valid pixel group and generating a data structure including the valid pixel group, an indicator of the video source of the pixel group, and a frame row position indicator of the valid pixel group; and forwarding the generated data structure through a common channel, the data structure including the buffered valid pixels of the frame row of the video frame from one of the multiple received serialized video streams.

[0084] In some embodiments, the received intermittently transmitted serialized video data is processed by one or more processors or processing circuitry of the viewing device to add a variable number of blanking pixels for output to the display as a continuously serialized variable line length video stream.

[0085] In some embodiments, the method further includes determining, after each data structure has been transmitted, whether another data structure of video frame rows originating from another of the plurality of video sources is ready to begin forward transmission, and if so, then forwarding the data structure from the other of the plurality of video sources over the common channel, otherwise determining whether another data structure from one of the plurality of video sources is ready for transmission, and if so, then forwarding the data structure from the one of the plurality of video sources over the common channel.

[0086] In some embodiments, after the data structure has been transmitted, the method further includes determining that no data structure from any of the plurality of video sources is ready for transmission, pausing transmission until a data structure from one of the plurality of video sources is ready for transmission, and transmitting the data structure forward over the shared channel.

[0087] In some embodiments, the method includes intermittently forwarding the generated data structures from each received serialized video stream as a single serialized video stream and pausing the forward transmission between the data structures after transmitting a data structure including a set of valid pixels and assigned frame row position indicators until a next data structure including a different set of valid pixels with a different frame row position indicator is ready for transmission.

[0088] In some embodiments, the combined intermittently transmitted video data stream has a fixed frame line length and a variable transmission pause duration.

[0089] Another, fourteenth aspect of the disclosed technology includes a method for viewing multiple serialized video streams received as a single serialized video stream, the method comprising: receiving a serialized video stream including multiple data structures, each data structure including an effective pixel group, a source indicator assigned to the pixel group, and a frame row position indicator assigned to the effective pixel group; processing the received serialized video stream by adding one or more blanking pixels to each effective pixel group of the received data structure until a data structure including the next pixel group in the serialized video stream is received to generate a complete row of video data; and causing each complete row of video data to be output for presentation on a display associated with the source indicator of the complete row of video at a position in the video frame indicated by the frame row position indicator of the effective pixel assigned to the complete row of video data.

[0090] In some embodiments, the method further comprises displaying each complete line of video data on the display.

[0091] In some embodiments, the received intermittently transmitted serialized video stream data may be video stream data intermittently transmitted by a combiner device of one of the aspects or embodiments disclosed herein.

[0092] In some embodiments of the method, the received serialized video stream (622) is a video stream output by a combiner device according to the above-described combiner device aspects disclosed herein or any of its embodiments.

[0093] Another, fifteenth aspect of the disclosed technology relates to a video display system, wherein the video display system comprises a video combiner device according to the eleventh aspect disclosed herein or any one of its embodiments and a viewing device according to the twelfth aspect disclosed herein or any one of its embodiments.

[0094] In some embodiments, the video combiner device also includes a video source.

[0095] In some embodiments, the viewing device further comprises at least one display configured to present video received from a video source via the device.

[0096] The sixteenth aspect of the disclosed technology relates to a computer program product comprising computer code, which, when loaded from a memory and executed by one or more processors of an apparatus according to the eleventh aspect disclosed herein or any one of its embodiments, enables the apparatus to implement a method according to the thirteenth aspect.

[0097] The seventeenth aspect of the disclosed technology relates to a computer program product, which includes computer code, which, when loaded from a memory and executed by one or more processors of a viewing device according to the twelfth aspect disclosed herein or any one of its embodiments, enables the viewing device to implement a method according to the fourteenth aspect disclosed herein or any one of its embodiments.

[0098] Another aspect of the disclosed technology includes a signal generated by an apparatus for combining video streams according to the eleventh aspect disclosed herein or any one of its embodiments, wherein the signal includes a serialized video stream of data structures, each data structure including at least one group of valid pixels from a video frame row and a frame row position indicator assigned to the group of valid pixels.

[0099] In some embodiments, the outgoing serialized video stream including the forwarded data structure does not contain any blanked pixels.

[0100] In some embodiments, an outgoing serialized video stream comprising forward transmitted data structures includes in each data structure at least one blanked pixel having a number less than the number of blanked pixels of a received video frame row comprising an active pixel group of the data structure.

[0101] Another aspect of the disclosed technology includes a method for viewing two or more serialized video streams received as a single serialized video stream, the method comprising: receiving a first serialized video stream comprising a plurality of data structures, each data structure comprising an effective pixel group, a source indicator assigned to the pixel group, and a frame row position indicator assigned to the effective pixel group; and processing the received serialized video stream by adding one or more blanking pixels to each effective pixel group of the received data structures until a data structure comprising the next pixel group in the serialized video stream is received to generate a complete row of video data; and causing each complete row of video data to be output for presentation on at least one display at a position in a video frame indicated by the frame row position indicator of the effective pixels assigned to the complete row of video data, each of the at least one display being associated with the source indicator of the complete row of video.

[0102] In some embodiments, the method further comprises displaying each complete line of video data on the display.

[0103] In some embodiments of the method, the received serialized video stream is a discontinuous video stream output by a combiner device according to the above-described combiner device aspect disclosed herein or any one of its embodiments.

[0104] Another aspect of the disclosed technology includes means for performing a method according to any of the disclosed method aspects.

[0105] Another aspect of the disclosed technology includes a computer program product, which includes computer code, which, when loaded from a memory and executed by one or more processors of a video combiner device according to the eleventh aspect disclosed herein or any one of its embodiments, enables the video combiner device to implement a method according to the thirteenth aspect disclosed herein or any one of its embodiments.

[0106] Another aspect of the disclosed technology includes a computer program product, which includes computer code, which, when loaded from a memory and executed by one or more processors of a viewing device according to the twelfth aspect disclosed herein or any one of its embodiments, enables the viewing device to implement the method of processing a received video stream of any one of the disclosed aspects or embodiments.

[0107] Another aspect of the disclosed technology includes a signal generated by an apparatus according to the eleventh aspect disclosed herein or any one of its embodiments, wherein the signal includes a serialized video stream of data structures, each data structure including at least one group of valid pixels from a video frame row and a frame row position indicator assigned to the group of valid pixels.

[0108] In some embodiments, the outgoing serialized video stream including the forwarded data structure does not contain any blanked pixels.

[0109] In some embodiments, an outgoing serialized video stream comprising forward transmitted data structures includes in each data structure at least one blanked pixel having a number less than the number of blanked pixels of a received video frame row comprising an active pixel group of the data structure.

[0110] Another aspect of the disclosed technology includes a video display system, wherein the video display system includes a device according to the eleventh embodiment disclosed herein or any one of the embodiments and a viewing device according to the twelfth aspect disclosed herein or any one of the embodiments.

[0111] In some embodiments, the apparatus for processing the serialized video stream for onward transmission further comprises a video source.

[0112] In some embodiments, the viewing device further comprises at least one display configured to present video received from a video source via the device.

[0113] In some embodiments, the video display system further comprises a plurality of video sources, each video source being arranged to provide serialized video data to the apparatus according to the eleventh embodiment.

[0114] Another aspect of the disclosed technology includes a computer-readable storage medium, which includes a computer program code, which, when executed by one or more processors or processing circuit systems of the video combiner of the aspects or embodiments disclosed herein, causes the video combiner device to implement the method according to the thirteenth aspect or embodiment of the method disclosed herein.

[0115] Another, tenth aspect of the disclosed technology includes a computer program carrier, which carries a computer program including computer program code, which, when loaded from the computer program carrier and executed by one or more processors or processing circuit systems of the video combiner device of the aspects or embodiments disclosed herein, causes the video combiner device to implement the method according to the thirteenth aspect or embodiment disclosed herein, wherein the computer program carrier is one of an electronic signal, an optical signal, a radio signal or a computer-readable storage medium.

[0116] It will be apparent to one of ordinary skill in the art that the disclosed aspects and embodiments may be combined with each other in any suitable manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Some embodiments of the disclosed technology will now be described with reference to the accompanying drawings, which are provided by way of example only and in which:

[0118] Figure 1A schematically illustrates how a time delay may be established according to the prior art when there are two video sources, each of which outputs a serialized video stream to an intermediate device, the intermediate device being configured to buffer the received video streams line by line before forwarding the combined video stream to two remote displays;

[0119] Figure 1B to Figure 1C Schematically illustrates how a time delay may be established according to the disclosed technology when there are two video sources, each of which outputs a serialized video stream to an intermediate device, the intermediate device being configured to buffer the received video stream line by line before forwarding the combined video stream to two remote displays;

[0120] Figure 2A schematically illustrates an example of serial video frame transmission;

[0121] Figure 2B schematically illustrates an example of how serially transmitted video frames may be displayed by an LCD display panel;

[0122] Figure 3A schematically illustrates an example of an apparatus according to some embodiments of the disclosed technology;

[0123] Figure 3BSchematically illustrates an example of a method according to some embodiments of the disclosed technology;

[0124] Figure 4 schematically illustrates an example of serialized transmission of variable video frame lengths from a single video source according to some embodiments of the disclosed technology;

[0125] Figure 5 Schematically illustrates some embodiments of the disclosed technology. Figure 3A How does the device generate the data structure header?

[0126] Figure 6 schematically illustrates an example of a viewing device according to some embodiments of the disclosed technology;

[0127] Figure 7 schematically illustrates a method performed by a viewing device according to some embodiments of the disclosed technology;

[0128] Figure 8 Schematically illustrates a system for displaying video from two video sources according to some embodiments of the disclosed technology;

[0129] Fig. 9 schematically illustrates an example of serialized transmission of variable video frame lengths from multiple video sources to multiple displays according to some embodiments of the disclosed technology;

[0130] Fig.10 schematically illustrates an example of another method according to some embodiments of the disclosed technology;

[0131] Fig.11 schematically illustrates an example of how a device according to some embodiments of the disclosed technology generates a data structure header; and

[0132] Fig.12 Another example of processing received video data by a viewing device according to some embodiments of the disclosed technology is schematically shown. DETAILED DESCRIPTION

[0133] Various aspects of the present disclosure will be described more fully below with reference to the accompanying drawings. However, the devices and methods disclosed herein may be implemented in a variety of different forms and should not be construed as limited to the aspects set forth herein. Whether explicitly mentioned or implicitly, if a step is not necessary for some of the disclosed embodiments, the step may be reordered or omitted. Like numbers in the figures always represent like elements, however, the same or different embodiments may have the same or different numbers, depending on which sheet they are first shown on in the figure.

[0134] The terms used herein are only used for the purpose of describing specific aspects of the present disclosure and are not intended to limit the disclosed technical embodiments described herein. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0135] Figure 1A Schematically illustrates how a time delay may be established when there are two video sources, each of which outputs a serialized video stream to an intermediate device, which is configured to buffer the received video stream frame by frame before forwarding the combined video stream to two remote displays while complying with the Serial Data Interface (SDI) video standard transmission requirements. Figure 1A In the present invention, two asynchronous video sources #1 and #2 of serialized video data streams each output data to a combiner device via a separate 3G serialized video link, and the combiner device is configured to merge the two streams into one outgoing serialized video stream for forward transmission to a viewing device including two displays via a 6Gbit / s link according to the SDI video standard.

[0136] exist Figure 1A In FIG. 1 , two video sources #1 and #2, labeled 102a and 102b, are both configured to generate a video of a scene, in this example, they are both configured to capture a view of a swinging pendulum. Figure 1A As shown in , video source #1, 102a outputs an image of a pendulum, which is transmitted toward a display 106a via a serialized data interface (SDI) link 104 via a video combiner device or hub 108, which combines the video from video source #1 with the video of another pendulum from video source #2.

[0137] like Figure 1A As shown in FIG. 1 , the video frames received from video source #1 are shown as being stored in a buffer at hub 108 ( Figure 1A However, because video source #2 102b is running at a slightly lower clock speed to transmit video, or because link 104b is slower, its transmission of the frame for the other pendulum arrives slightly later at hub 108. Since the hub is configured to wait until both frames are fully received from both sources before any forward transmission of either frame, Figure 1A The result shown in is that one and a half frames from video source #1 are buffered in the top rows of buffers Buf 1 and Buf 2 at hub 108 before the lower buffer Buf 1 holds the entire image of the same pendulum from video source #2.

[0138] Since the combiner box cannot lock to the clock of either source, both image frames must be completely received at the hub 108 before being forwarded line by line to the respective displays, and therefore the buffers at the hub 108 will accumulate at different rates to compensate for the difference in asynchronous transmission between the two video sources before the video can be forwarded. This increases latency at the hub 108.

[0139] One way to reduce latency may be to configure the hub 108 to only buffer incoming video line by line. Figure 1B to Figure 1C It is schematically shown how this approach can lead to buffer underflow problems at the hub 108 if it is used to try to reduce the latency accumulated at the hub 108, in other words, if the hub 108 outputs video line by line from two asynchronous video sources.

[0140] Figures 1B to 1D Schematically illustrates a series of examples of what would happen at the hub 108 if the hub buffer was configured to forward frames line by line to the displays 106a, 106b without waiting for complete frames to be received from both sources at the hub device. Figures 1B to 1D , the hub 108 instead forwards the video received from the two sources #1, #2 to the displays 106a, 106b line by line.

[0141] First, if Figure 1B As shown in , the video streams received from two asynchronous video sources can be kept sufficiently synchronized so that when the video frame from source #1A is received by the top buffer Buf 1 at hub 108, the lower buffer Buf 1 at the hub already has the first complete row of the frame from source #B. Figure 1C 1 shows how video frames are transmitted line by line as they are received at the hub 108, thereby reducing the delay for the displays 106a, 106b to receive the first line of their respective video frames. However, since the two sources are asynchronous, a situation eventually arises where only one line can be sent from one video source (here source #1) because a buffer underflow occurs at the hub 108 for the video from the other video source (source #2), as shown in FIG. Figure 1D In other words, Figure 1D The lower buffers Buf 1 and Buf 2 in are both empty.

[0142] Thus, to avoid the risk of buffer underflow, it is desirable to buffer the complete frame at the hub 108 before sending any frame rows to the remote display.

[0143] Some embodiments of the disclosed technology seek to provide another solution that can reduce latency and reduce the risk of buffer underflow even when the hub 108 is configured to forward the received video data line by line. In some embodiments, similar latency reduction can be achieved even if only one video stream is received from a single video source.

[0144] Each line of a video frame at a video source includes a plurality of active pixels, where the active pixels represent pixels to be reproduced on a remote display; and a plurality of blanking pixels, which are added to generate a continuous stream of data. The blanking pixels form a blanking interval and are a historical convention to allow a raster scan or cathode ray tube (CRT) type display to resume to the beginning of the physical display to start the next line of the image after the end of the previous line, and at the end of the active pixels of the last line of the frame, a line of blanking pixels is added to allow the CRT light beam to physically move to the beginning of the top line of the next frame. This convention still applies to other types of displays, including flat panel displays, including, for example, organic light emitting diodes, OLEDs and LCDs, liquid crystal displays, or displays based on optical coupling device technology. References to displays herein refer to displays configured to receive data from a video source including blanking pixels. The viewing device may include one or more displays, and may include one or more processors for processing the received serialized video data stream for output to the display.

[0145] The term "blank" as used herein only means "not intended to be displayed as part of a video image frame line", and does not necessarily mean that the "blank interval" pixels do not contain any data or information. In the prior art, each pixel of the received serialized video data stream, whether it is an active pixel intended to be displayed or a blanking pixel not intended to be displayed, is transmitted along the data link 104 at the clock frequency of the video image source 102 according to standard technology.

[0146] When there is only one video source, buffer underflow or overflow at the hub 108 or at any viewing device having one or more displays 106 is generally eliminated by design. However, at high data transfer speeds, especially if there are two sources, the likelihood of buffer underflow or overflow increases, as described above when the two sources are asynchronous. Thus, when the latency delay caused by frame-by-frame buffering at the hub 108 is unacceptable, it is advantageous if the hub 108 can buffer line by line and forward the video line by line for display.

[0147] For example, a serialized video data transmission rate of about tens of milliseconds of frame buffering for transmitting video via the hub 108 may encounter a latency of about 16 milliseconds from the time the image is transmitted from the source 102 to the time the transmitted image is displayed on the remote display 108. This latency is too high for use cases such as, for example, when a medical procedure is performed using a camera to capture images of a body area where the medical procedure is being performed and the surgeon or one or more assistants performing or assisting with the medical procedure wear a near-eye display or head-mounted device. Thus, it is desirable to improve how a hub device or the like can process the received serialized video stream data to reduce the occurrence of any additional latency or delay, thereby further reducing and preferably minimizing the time from capturing an image to displaying the image on the display 106.

[0148] In the attached figure Figure 2A Schematically illustrated in greater detail is how fixed line length video frames are transmitted from a video source 12 (such as a camera) to a remote display 106 via a frame-by-frame transmission standard link 104 (eg, a serial data interface (SDI) transport link 104, such as a wired link). Figure 2A Active pixels of the video frame 100 are schematically shown as boxes with numbers, and blank or blanked pixels are shown as empty boxes. Figure 2A It is also schematically shown that, in addition to including multiple rows (where Figure 2A In addition to the four lines 202a-202d shown by way of example only, each frame may also end with a blank or blanked video line 202e.

[0149] Each video frame 204 includes a fixed number y of horizontal pixel rows and a fixed number x of vertical pixel columns, which are usually set according to the specifications of a specific video standard. Figure 2A As shown in , y = 5 and x = 6. This means that Figure 2A In the example shown in , each source video frame 204 includes 30 pixels, including 5 rows, each row has 6 pixels, 4 of which are valid pixels and 2 are blanking pixels, and each vertical line includes 5 pixels, 4 of which are valid pixels.

[0150] like Figure 2A As shown in , each video frame in the serialized video stream from video source 102 over link 104 has a vertical blanking interval of 1 line and a horizontal blanking interval of 2 pixels. Figure 2AThe lower half of the diagram shows how each group of pixels, including active pixels and blanked pixels, forms a horizontal line of the video frame 100, and illustrates how the video frame 100 is transmitted serially as a continuous stream of data, horizontal frame line by horizontal frame line, over the SDI transport link 104 to the remote display 106. The time to serialize each video frame including 30 pixels is determined by the clock period to transmit each pixel multiplied by the number of pixels in each video frame, so in this example it is 30×clock period.

[0151] Figure 2A and Figure 2B In more detail, it is shown how the top row 202a is transmitted first, where the valid pixels are transmitted before any blanking pixels, followed by the next row 202b, the third row 202c and the last row 202d and the last row 202e, which only includes the last transmitted blanking pixels. After the transmission of the blanking pixel row 202e, the transmission of the next image frame begins with the first horizontal row of the frame, followed by the second, third, fourth rows, and finally the blanking rows of the next video frame, and the process is iterated while the video source 202 generates video data.

[0152] As mentioned above, each pixel takes a certain amount of time to transmit, which is determined by the clock cycle used by the video source 102 to transmit the video. If the video source clock frequency is 1 pixel per clock cycle, as an example only, the clock cycle is 1 μs, then each pixel will take 1 μs to be ready for transmission. The horizontal lines 202a-202e each include 6 pixels, so each line takes 6 μs to be ready for transmission. Since the video frame 204 has five lines 202a-202e, the total time to serialize the entire video frame 204 for transmission along the SDI link 104 is 30 μs.

[0153] When the video stream from the video source 102 is delivered to a display 106 (such as an LCD panel display) via the SDI delivery link 104, the LCD panel at the remote display 106 will need to synchronize the start of each received line 202a-202d in order to reconstruct the frame 204 for proper display. Metadata is provided for a group of pixels for each video frame line to provide information that enables the display 106 to properly render the image carried by the serialized video frame data.

[0154] Figure 2B Schematically shows Figure 2A An example of how a serially transmitted video frame may be processed for display in more detail at display 106 is shown in FIG. Figure 2BIn the example shown in , a video source 102 of a serialized video data stream provides a series of fixed length lines. The received video stream is processed at a receiver 202 of a display 106, which forwards the data as each line of data is received for display on a screen 204, such as an LCD panel 204. The clock period of the video source 102 is used to synchronize the columns of pixels as they are displayed. This requires that the clock frequency at the display 106 is locked to the frequency of the video source clock used to send the pixel data over the SDI link 104. For example, if the source clock period is 1 μs, then the display video clock frequency must have a clock period that is based on the source clock period. By locking the display clock frequency to the source clock frequency, as Figure 2B As schematically shown in , the first column of pixels in each frame, such as frame 204, may be vertically aligned when reconstructed on LCD panel 204. If the timing of the two clocks is incorrect, a buffer overflow or underflow may occur depending on how out of sync the two clocks are with each other.

[0155] Therefore, in order to Figure 2A and Figure 2B In order to correctly display the captured image on the display 106 in the example display system shown in , the source clock cannot be decoupled from the display clock. This imposes various restrictions on the design of a display system including at least one video source and at least one video display. First, only one source clock frequency can be physically forwarded using the serialized video data stream transmitted through each SDI link 14, thereby limiting each video data stream to provide video captured by a single video source 102.

[0156] One example where this can cause problems is when it is desirable to reduce the number of cables, for example if the hub 108 is attached to a head mounted device viewing apparatus including one or more displays 106 to reduce the bulkiness of the viewing apparatus. Second, any processing of the video stream received at the intermediate device or hub to add video overlays or similar effects, such as image enhancement, for example, results in the generation of two video streams that cannot be transmitted from the hub 108 to the displays 106 using the same video link. Furthermore, transmission latency can mean that even with line-by-line transmission, the delay in displaying the live streaming video can cause problems in some embodiments where there is a dependency on the live streaming video for performing actions. An example of such a use environment is video surgery, where a surgeon or similar operator uses a head mounted device with a near eye display to receive an augmented view of a person or animal being operated on.

[0157] Moreover, such usage environments may benefit from stereoscopic displays, where the presented images provide a sense of depth. In this example embodiment, images may be reconstructed on one or two near-eye displays of a head-mounted device viewing device. Each display may receive a video stream from a separate source, and one or both video streams may be additionally processed at an intermediate device to add image enhancements, such as in the form of color overlays. Even if the images were originally captured by two or more synchronized video sources, unwanted delays, jitter, and different frame rates may cause the displayed images to lose synchronization.

[0158] In general, if two video sources require their images to be synchronized when displayed remotely, then two separate feeds may need to be provided, and even then, additional mechanisms are required to ensure that the reconstructed video frames are properly aligned, and this adds additional latency. For example, if two or more video sources are to be provided to a viewing device that includes a display (such as the left and right eye displays in a near-eye display head-mounted device), then the video sources cannot share a single SDI cable because their source clock frequencies will never be exactly the same, which means that the images displayed on each display will be out of sync. However, having two separate cables on such devices is undesirable due to the added bulk and additional weight of the resulting head-mounted device. As described in more detail below Figure 6 An example embodiment is shown in which video is provided from two or more video sources.

[0159] Figure 3A An example of an apparatus 302 for processing a serialized video stream is schematically shown. In some embodiments, the apparatus 302 may include a hub co-located with a surgical imaging device. In some embodiments in which the apparatus 302 includes a hub co-located with the surgical imaging device and is configured to receive a serialized video stream from one or more video sources, the video source may include one or more cameras as part of the surgical imaging device. In some embodiments, the apparatus 302 is or is part of a surgical imaging device including a surgical microscope. In some embodiments, the hub may be co-located with the surgical microscope and / or may be part of a control system or provided as other computing components or units of the microscope.

[0160] The device 302 includes a receiver 304 configured to receive a serialized video stream from a single video source 101 via a serialized video data link 103. However, according to the disclosed technology, the device 302 decouples the source clock from the display clock by generating a discontinuous stream of serialized video data, rather than forwarding a continuous stream of video data that can be used to align the source clock with the display clock. For each horizontal line of a video frame, the discontinuous stream of serialized video data includes at least an effective group of pixels, followed by a pause in transmission. In some embodiments, for each received video frame line including at least one group of effective pixels, at least one blanking pixel is omitted when the group of effective pixels is forwarded. This reduces the time to transmit a video frame because there are fewer blanking pixels to transmit. Alternatively, the transmitter can pause transmission until the next complete row of pixels including effective pixels and blanking pixels is buffered, at which time the buffer is cleared and the transmitter transmits the group of effective pixels and omits at least one blanking pixel. In some embodiments, one or more blanking pixels are omitted. In some embodiments, the number of blanking pixels forwarded or omitted is fixed for all video lines. After transmitting a data structure including a valid group of pixels from a received video frame line and any blanking pixels from the received frame line that were not omitted from forward transmission, the transmitter is configured to wait until the next line of video frame data is ready for transmission. This results in a pause in transmission and results in a discontinuous stream of intermittently transmitted serialized video frame data being transmitted over the serialized video transmission link to the viewing device 410. Removing one or more blanking pixels from the end of each horizontal line of the video frame and / or by removing or reducing the blanking pixels that form the last line of blanking pixels in each video frame results in intermittent transmission of serialized video data and allows the clock of the remote viewing device to be decoupled from the clock of the video source 101.

[0161] exist Figure 3A , the received serialized video frame data is buffered line by line by buffer 305, which is configured to discard one or more blanking pixels in each variable line length of the video frame and forward at least the valid pixels of each fully received video line as a group of pixels with a regular line length to transmitter 310. By removing at least one blanking pixel from each line received by device 302 from source 101, the transmitter of device 302 decouples the forwarded data from the video source clock, which allows the receiver clock to be reset accordingly.

[0162] Some embodiments of the device 302 include means for processing a serialized stream of video data for onward transmission (e.g., to a viewing device 312). The serialized stream of video data may include a plurality of serially transmitted video frames from a video source, each video frame including a plurality of frame lines, at least one frame line including a set of active pixels and one or more blanking pixels.

[0163] The apparatus may include: a receiver 304 configured to receive a serialized video data stream; a memory 306, which may include a memory 305 arranged to buffer the received serialized video data until at least all valid pixels of a frame row of pixels of a received video frame have been buffered; one or more processors or processing circuit systems 308 configured to generate a data structure for each group of valid pixels of a frame row of pixels, the data structure comprising at least the group of valid pixels of the frame row and a frame row position indicator assigned to the group of valid pixels; and a transmitter 310 configured to intermittently forward the generated data structures by pausing transmission after each data structure has been transmitted until the next data structure is ready to begin transmission.

[0164] In some embodiments of the apparatus 302 for processing a serialized video stream, the receiver 304 is configured to receive a serialized video stream 104, for example, a video stream sent from a video source 102 via an SDI link. The serialized video stream 104 includes a plurality of serially transmitted video frames from the video source 102, each video frame including a plurality of video frame rows. The first video frame row to the second to last video frame row each include a set of valid pixels and blanking pixels. The last frame row of each video frame includes blanking pixels. The received video frame data can be transmitted forward to a viewing device 312 using a video transmission standard transmission link, such as a serialized data interface (SDI) transmission link 103, but using a different transmission link 311.

[0165] The memory 306 may include various types of memory, for example, memory for storing data (including computer program code) and memory (such as at least one buffer 305) arranged to buffer at least valid pixels of the received serialized video stream 103 until pixels of a complete frame row are received from the video source 101. One or more processors or processing circuit systems 308 of the device 302 are configured to include a frame row position indicator for each buffered complete frame row of pixels in the forward transmitted data. In some embodiments, the valid pixels and the frame row position indicator form a data structure. In other words, the forward transmitted data includes intermittently transmitted data structures, each data structure including the valid pixel group and the frame row position indicator assigned to the valid pixel group. In some embodiments, each data structure includes a group of valid pixels from the received frame row and omits at least one blanking pixel of the received frame row, but, in some embodiments, any blanking pixels of the received video frame row are not included in the outgoing serialized video stream including the forward transmitted data structure.

[0166] In some embodiments, the frame row position indicator may be configured to indicate whether the frame row is the first frame row of the frame or a subsequent frame row in the form of a binary "is first row" or "not first row" indicator, since the number of rows in the frame is known. However, in other embodiments, a specific row position in the frame may be indicated. In some embodiments, the frame row indicator may be received in an incoming video data stream from a source and retained in outgoing forward transmitted data, however, in other embodiments, the frame row position indicator may be assigned by a (one or more) processors or processing circuit systems. Thus, the display of valid pixels for the first row in the received video frame allows the entire image to be correctly reconstructed on the display of the viewing device 312.

[0167] In some embodiments, the frame row indicator assigned to each group of valid pixels is included in a header or a footer of a data structure including the group of valid pixels, wherein a bit size of the header or the footer matches a parallel width of a serializer of a transmitter (310) configured to serialize the data structure in the outgoing video stream data.

[0168] The transmitter 310 is configured to intermittently forward transmit serialized video data comprising a series of data structures, wherein there is a pause between the data structures if the next data structure is not ready for transmission after the last data structure is transmitted. In other words, the transmitter is further configured to pause the forward transmission of the outgoing serialized video stream between the data structures after transmitting a data structure comprising a group of valid pixels and their assigned frame line position indicators until the next data structure comprising a different group of valid pixels with the next frame line position indicator is ready for transmission.

[0169] The transmitter 310 may forward the data structure as a serialized video stream 311 to a viewing device 312 that includes a display, for example, a suitable raster-type display such as the display shown in FIG. 2 that includes one or more LCD panels 204 .

[0170] References herein to video frame lines, or simply frame lines, refer to horizontal video frame lines, such as Figures 1A-1D , Figure 2A and Figure 2B , when they are stacked vertically and displayed in sequence, they form a video frame. In other words, in some embodiments, when displayed in the usual orientation of a video screen, each video frame row comprises a horizontal row of video frames, however, it is also possible to display the frame rows in different orientations.

[0171] Now go to Figure 4 , Figure 4 In more detail, it is schematically shown how the device 302 can pause the transmission between the forwarded data structures. Figure 44, multiple serialized data structures 402a-402d form a video frame when displayed on display 414 of viewing device 312, and serialized data structures 406a-406b are associated with subsequent image frames. Data structures 404a-404d and 406a-406b are shown as being transmitted serially from device 302 to viewing device 312.

[0172] like Figure 4 As shown in FIG. , each data structure includes a set of valid pixel groups and related header information (for clarity, Figure 4 402a, 402b, 402c, 402d are transmitted in sequence, and the video frame row position indicator is provided as metadata, for example, in the header or footer of the data structure. The video frame row indicator indicates to the viewing device 312 and / or the display 414 which data structure should be displayed first when reconstructing the serialized image on the display 414. Additional metadata may be provided so that when reconstructing the video frame image 416 on the display, the group of valid pixels of the data structure can be displayed on the display 414 in the correct row and the correct pixel order. Figure 4 Each video frame in the example of comprises only four horizontal lines, each line having four active pixels.

[0173] Figure 4 It is also schematically illustrated how the device 310 can pause transmission 404a-404d, 408a-408b between consecutive data structures 402a-402d, 406a-406d. Since there will usually be a video frame line including only blanked pixels after the second to last frame line including valid pixels, the pause 404d before transmitting the first line of the next video frame in the data structure 406a can be longer than the other pauses 404a-404c.

[0174] In some embodiments, the intermittently transmitted video data stream has a fixed frame line length, and the duration of the transmission pause is variable. Based on the number of blanking pixels received at the device 302 but not forwarded because they were omitted in the data structure transmitted onward from the device 302 to the viewing device 312, the pause between transmissions can vary accordingly.

[0175] In some embodiments, the received video data stream has a variable line length and the duration of the pauses between the forwarded data structures varies. It is also possible to implement the invention wherein the video data stream has a fixed number of pixels in each frame line.

[0176] The outgoing serialized video stream transmission pauses between the data structures of the first video frame are marked 404a, 404b, 404c, and after the last data structure 404d, the first data structure of the next video frame is then sent, followed by a transmission pause 408a, followed by the next data structure 406b of the next video frame, after which the subsequent transmission pause is marked 408b, and so on.

[0177] Now return to the attached figure Figure 3B , Figure 3B An example of a method 300 performed by a device 312 for processing a serialized video stream 104 for onward transmission according to some embodiments of the disclosed technology is schematically illustrated.

[0178] like Figure 3B As shown in FIG. 3 , a method 300 for processing a serialized video stream for onward transmission includes receiving a serialized video stream 314. The serialized video stream may include a video stream from a video source (e.g., Figure 3B The method 300 further includes buffering at least the valid pixels of the received serialized video stream in 316 until the pixels of the complete frame row are received in 318. The method 300 further includes processing the pixels of each buffered complete frame row in 320 to associate the group of valid pixels of the frame row with the frame row position indicator, which may include allocating the frame row position indicator or reusing or retaining the existing frame row position indicator from the received incoming video data. The method 300 also includes generating a data structure 322, which includes the valid pixel group and the frame row position indicator assigned to the valid pixel group. Once the data structure is ready for transmission 324, the method 300 also includes forward transmission 326 as the outgoing serialized video data. The method then includes checking if the next data structure is ready for transmission at 324 and, if not, pausing forward transmission 328 of the outgoing serialized video stream until the next data structure is ready for transmission at 324, at which point the data structure is transmitted at 326.

[0179] The ongoing transmission is intermittent by pausing transmission between data structures 328 after transmitting a data structure including a set of valid pixels and assigned frame row position indicators until a next data structure including a different set of valid pixels with a different frame row position indicator is ready for transmission 324. However, in some embodiments, if the next data structure is always ready for transmission after the previous data structure is transmitted, then the ongoing transmission may be continuous. In this case, it can be considered that the data transmitted forward is transmitted more efficiently with lower latency because the device 302 will not wait until each video frame has been received, but the latency will depend on the delay caused by waiting for each video frame row to be received, and by removing one or more or all blanking pixels from each video frame row, each frame row has a smaller number of pixels to transmit, in other words, there is less data to transmit in order to transmit the frame row to the destination.

[0180] The term "data structure" as used herein refers to structured data in a serialized video stream that is transmitted forward, which includes a group of valid pixels and one or more indicators associated with the group of valid pixels, which indicators may also be referred to as tags. Metadata, for example, one or more indicators or tags, may be provided as a header preceding the group of associated valid pixels in some embodiments, or as a tail following the group of pixels with which it is associated in some embodiments. In some embodiments, one or more indicators or tags may be located before the associated group of pixels, and one or more indicators or tags may also follow the associated group of pixels. References to headers herein are also references to indicators or tags provided at other locations (such as tails) relative to their associated groups of valid pixels, unless the context clearly indicates otherwise.

[0181] Accordingly, reference to means 302 for generating a data structure refers to means 302 for preparing the structured data for output in a serialized video stream for onward transmission.

[0182] In some embodiments of the method 300 performed by the device 302, any blanked pixels in the received video frame lines are not included in the outgoing serialized video stream including the forwarded data structure.

[0183] In some embodiments, the method 300 performed by the device 302 also includes omitting or discarding at least one blanking pixel for each received video frame row that includes a group of valid pixels and at least one blanking pixel, and including the group of valid pixels and any remaining blanking pixels for the received video frame row in a forward transmitted data structure of the outgoing serialized video stream.

[0184] In some embodiments, the duration of the forward transmission pause varies depending on the number of blanking pixels present in each frame line. For example, since the last frame line consists only of blanking pixels, there will be an extra long pause in the forward transmitted video data stream between the transmission of the data structure corresponding to the second to last frame line of one frame and the transmission of the data structure corresponding to the first line of the next frame.

[0185] In some embodiments, the method 300 performed by the device 302 results in or also includes decoupling a source clock of the received serialized video stream 104 in the outgoing serialized video stream 311 .

[0186] In some embodiments, method 300 allows viewing device components to process a received serialized video stream using a clock that is not based on a clock of a device containing a serializer used to generate the serialized video stream.

[0187] In some embodiments of the method 300, transmitting the outgoing serialized video stream 311 onward includes transmitting the outgoing serialized video stream over a data communication channel along a wired link to a viewing device (312) configured to display 416 the received video data. This provides a secure link to the display 414, however, it may not be the fastest link. In some embodiments, instead or in addition, transmitting the outgoing serialized video stream 311 onward includes transmitting the outgoing serialized video stream 311 over a wireless data communication channel to a viewing device (312) configured to display 416 the received video data.

[0188] In Figure 3 and Figure 4 In some embodiments of the apparatus 302 shown in , blanked pixels of received video frame rows are not included in the outgoing serialized video stream including the forward transmitted data structure. However, in some embodiments, it is also possible to discard at least one blanked pixel in each row and retain one or more blanked pixels so that each received variable row length is regularized to a fixed number of valid pixels and blanked pixels. In other words, in some embodiments, at least one blanked pixel of each received video frame row including a set of valid pixels and at least one blanked pixel is discarded, and the set of valid pixels and any remaining blanked pixels of the received video frame row are included in the forward transmitted data structure of the outgoing serialized video stream.

[0189] In some embodiments, Figure 4, although the average frame period matches the frame period of video source 102, the buffering and removal of blanking pixels in the outgoing serialized video transmission 311 enables the video stream received by viewing device 312 to no longer be locked to the same clock period used by video source 104. In other words, in some embodiments, device 312 decouples the source clock of the received serialized video stream 104 in the outgoing serialized video stream 311 by varying the pause duration between forward transmitted data structures.

[0190] In some embodiments, the transmitter 310 of the device is configured to transmit the outgoing serialized video stream as a discontinuous serialized video stream via a data communication channel along a wired link with the viewing device 312. However, in some embodiments, instead or in addition, the onward transmitted serialized video stream data is intermittently transmitted via a wireless data communication channel to the viewing device 312 configured to cause display of the received video data.

[0191] In some embodiments, Figure 4 As shown in , the decoupling is achieved by the device 302 using a slightly different clock period X+δ, where δ is a small percentage difference in the clock period of each transmitted pixel in the serialized video stream 311 transmitted onward.

[0192] The disclosed techniques also allow the clock of the receiver 410 at the viewing device 302 to use a clock that is not locked to the clock cycle used by the device 302 or the clock cycle used by the video source 102. For example, Figure 4 As shown in , the clock at the receiver 410 may run at X-δ, where δ may be the same or different than the amount by which the clock at the receiver drifts from the source video clock X.

[0193] The viewing device 312 generates variable length video lines by adding blanking pixels after receiving valid pixels of the received data structure until the next data structure is received in the serialized video transmission 311. In other words, during pauses when no data structure is received, the receiver 410 adds one or more blanking pixels. This allows the receiver 410 to generate a continuous stream of video data as output 412 to the display 414 for display in some embodiments by adding a sufficient number of blanking pixels at the end of the last group of valid pixels received.

[0194] Figure 5 Another schematic diagram of an apparatus 302 for serializing data for onward transmission according to some of the disclosed embodiments of the present technology is shown. Figure 5, the apparatus 302 is shown to include a memory 306, one or more processors or processing circuit systems 308, a receiver 304, and a transmitter 310. The memory 306 may include a buffer memory 305, but also includes a suitable memory for storing computer code 500, which is also schematically shown in Figure 5 When the computer program code 500 is loaded from the memory 306 and executed by one or more processors or processing circuit systems 308 of the apparatus 302 for processing the serialized video data stream for forward transmission, the apparatus 302 is caused to implement Figure 3B An embodiment of the method 300 shown in , or a method for processing a received serialized video stream for onward transmission according to one of any suitable method aspects disclosed herein.

[0195] exist Figure 5 In the embodiment shown in , computer code 500, when executed by one or more processors or processing devices, includes one or more software modules or circuit systems that cause device 302 to generate 502 a data structure and transmit a data structure 504. Figure 5 As shown in , the data structure generation module or circuitry includes a module or circuitry 503 that associates metadata (e.g., data provided in a header or footer of a data structure) with a valid pixel group of the data structure. Although references below may refer only to the header of a data structure, as one of ordinary skill in the art will find apparent, the footer or other designated location in a data structure may be used in place of the header, and references to the term "header" should be interpreted accordingly herein unless explicitly referenced to the contrary.

[0196] The computer code 500 also includes a module or circuitry 504 for populating the header with valid pixels and associated metadata to form structured data, also referred to herein as a data structure. The metadata may include metadata provided from the video source 102 or generated by the device 302.

[0197] In some embodiments, the header filling and header association modules or circuit systems 503, 504 may include the same modules. The header filling module or circuit system 504 fills the header with one or more indicators or tags. For example, the header may be filled with a first indicator or tag to indicate in 506 whether the pixel group of the data structure includes a group of valid pixels forming the first row of the displayed video frame. Other indicators 508 (in Figure 5The header may include a trailer (shown as adding a tag 508 in FIG. 5 ) to capture other information required to properly display the valid pixel groups on the display. In some embodiments, the header may alternatively or additionally include a trailer, and references to the header should be interpreted accordingly. The data structure transmission module or circuit system 510 outputs the data structure for onward transmission to a viewing device (such as viewing device 312). If the next data structure is ready for transmission after transmitting one data structure, the data structure can be output as a continuous stream 311 of video data. If it is not ready for transmission, the data structure transmission module 510 pauses the transmission until the next data structure is ready for transmission.

[0198] Figure 6 An example embodiment of a viewing device 312 is schematically shown, where the viewing device 312 includes a receiver 410, one or more processors or processing circuitry 418 (e.g., a graphics processing unit), and a memory 420. The receiver 410 is configured to receive a video from a computer such as a graphics processing unit. Figure 3A and Figure 4 A device such as device 302 receives serialized video data 311. In some embodiments, display device 312 includes display 414, which includes one or more flat panel display panels, such as LCD panels or similar panels configured to receive serialized video frame data. The flat panel display can be a curved display.

[0199] Only active pixels are presented at display 414, while blanked pixels are ignored or discarded. Alternatively, display 414 may be provided on a separate device connected to viewing device 312, which receives serialized data from device 302. In some embodiments, display 414 may include a near-eye display screen or a screen on a head-mounted device.

[0200] The receiver 410 is configured to receive and process the discontinuous or intermittent stream 311 of video data, and when a pause occurs in the received data, the receiver 410 is configured to add one or more blanking pixels to the end of each received data structure. The receiver 410 then outputs each frame row including a group of valid pixels and header or footer information indicating the row position of the group of pixels in the video frame and one or more blanking pixels as a continuous data stream to the display 414, which is configured to render the valid pixels correctly so as to reconfigure the complete video frame to be rendered in the correct order on the display 414. The display 414 discards the added blanking pixels and reconstructs each video frame 416 using the frame row position indicator to correctly order the received valid pixel groups on the display.

[0201] exist Figure 6In the example embodiment shown in FIG. 4 , viewing device 312 includes display 416, but, as noted above, it will be apparent to one of ordinary skill in the art that in some embodiments, viewing device 312 may be connected to an external display and configured to output received video stream data to separate display 414, rather than providing received video stream data to an internal display (such as a 4K display). Figure 6 414 shown in FIG.

[0202] Figure 6 The memory 420 shown in FIG. 4 may store computer code 600 that, when loaded from the memory 420 and executed by one or more processors 418 of an embodiment of a viewing device 312 according to the disclosed technology, causes the viewing device 312 to implement an embodiment of a method 600 for receiving serialized video data (e.g., a continuous or intermittent sequence of data structures) for presentation on a display 414. Figure 6 As shown in FIG. 6 , the computer code 600 includes a module or circuit system 602 configured to receive a serialized video stream 311 including a plurality of data structures, each data structure including a set of at least valid pixels and a header or footer indicating the row position of the set of valid pixels in a video frame when displayed, for example, on a display 414. The computer program code also includes a module or circuit system 604 configured to add one or more blanking pixels to the end of the data structure, in other words, if the data structure includes valid pixels, then add additional blanking pixels until the next data structure is received. This can result in blanking pixels for an entire video frame row being generated after receiving a data structure for the second-to-last video frame row of a video frame before receiving the first frame row of the next video frame.

[0203] The computer code 600 also includes a module or circuitry 604 configured to output the valid pixels of the received data structure with the line position indicator information and any added blanked pixels as a continuous stream of variable line length video data for display on the display 414 .

[0204] In some embodiments of the disclosed technology, the computer code 600 may also be provided in the form of a computer program product including the computer code 600 .

[0205] In some embodiments, the computer code 600 is encoded in software as one or more modules, however, in some embodiments, the computer code may instead be partially or completely hard-coded in the circuitry. In some embodiments, the processor or processing circuitry may be configured to control the execution of the hard-coded or soft-coded computer code. In some embodiments, one or more of the processors or processing circuitry may include a graphics processor or dedicated graphics processing circuitry.

[0206] Some embodiments of the viewing device 312 are accordingly configured to cause the computer code 600 to be loaded from the memory 420 and executed by one or more processors or processing circuit systems 418 to implement the method 600 of processing a received serialized video stream for display. The received serialized video stream may be received intermittently in some embodiments, or may be received continuously in some other embodiments. Some embodiments of the method include receiving a received serialized video stream including a plurality of data structures, each data structure including at least one group of valid pixels from a video frame row and a frame row position indicator assigned to the group of valid pixels, generating a complete row of video data by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure including the next pixel group in the serialized video stream is received, and causing each complete row of video data to be output for presentation on the display at a position in the video frame indicated by the frame row position indicator of the valid pixels assigned to the complete row of video data.

[0207] In some embodiments, a serialized video stream is received from device 302 in accordance with any of the disclosed embodiments.

[0208] In some embodiments, the method also includes determining 605a that the penultimate frame row of a frame has been received, generating 605b a frame row including blanked pixels as the last frame row of the frame, and outputting 606 the frame row of the frame as a serial data stream of pixels for presentation on the display 414.

[0209] Now go to Figure 7 , Figure 7 Shown in greater detail is how viewing device 312 may process incoming intermittent video data (e.g., a discontinuous stream of serialized video data) and add blanking pixels so as to output a continuous stream of serialized video data including active pixels and blanking pixels to display 414 in the presence of a single video source.

[0210] like Figure 7 As shown in FIG. 6 , module or circuitry 602 examines header or trailer information of a discontinuous data stream received at viewing device 312 that indicates whether the associated pixel group is the first horizontal row pixel group of a video frame from single video source 104 .

[0211] In some embodiments, the frame row position indicator is a binary tag having a first value indicating whether the pixel group belongs to the first row of the video frame, and a different value for all other rows in the video frame. This allows the receiving device to output the received pixel groups in the correct order, because the valid pixel groups are received in order in the received serialized video stream 311 based on the preconfigured number of rows in the frame.

[0212] In some embodiments of the disclosed technology, a frame row indicator assigned to each group of valid pixels is included in a header of the data structure.

[0213] In some embodiments, the viewing device is configured to sample the received video data in regular bit size intervals or chunks. If the header bit size can be matched to the bit size of the parallel width of the serializer of the transmitter at the video source 102 so that it has the same bit size as the transmitted pixel data, then the viewing device can process the received data stream more efficiently because it only has to check one set bit size interval at a time to extract the header information.

[0214] In some embodiments, if the header is too small, then, for example, the transmitter 310 of the device may pad the header with additional bits to increase it so as to allow the header information to be extracted from the pixel data stream in a more efficient manner at the viewing device 312. In this case, the viewing device 312 is configured to only check the data at set bit intervals, rather than being configured to look for smaller or larger bit intervals, which would increase processing and potential latency at the viewing device 312.

[0215] These row indicators are used at the interface of the display 414, where the continuous stream of video data received from the receiver 410 of the viewing device 412 is presented on the display 414. The viewing device may buffer or discard the received data until it recognizes that a group of pixels is preceded by the first row indicator or label, at which time it outputs the pixels to the display 414. The pixel header information may also include information about how the rows are presented on the display 414 to correctly present the image video frame 416 (e.g., which pixel should be presented first) as well as the row position.

[0216] As mentioned above, in some embodiments, instead of or in addition to having a header, each data structure may also provide equivalent information in a tail associated with a valid pixel group. References to the header herein should be considered to also refer to when the header is provided as a tail after a group of valid pixels in the data structure. The indicator or tag that fills the header field (whether the header is provided as a header before a group of valid pixels or as a tail after a group of valid pixels) is used to identify which row is the first row, and in some embodiments, this information can be provided in the form of, for example, a 64-bit header tag using techniques defined by the SDI standard. In some embodiments, the data structure of the disclosed technology can use different header or tail sizes to identify whether the pixel group is the first row of pixel groups.

[0217] In some embodiments, the size of the header or trailer is the same as the bit size of the pixel. This means that if the pixel is 20 bits, then the header or trailer can also be 20 bits. This makes it easier to configure the receiver of the viewing device to extract the header or trailer in an efficient manner, because the received serialized data stream can then be captured every 20 bits, 20 bits, etc. If the header or trailer has a different bit size than the bit size of the pixel, for example, if the header has a size of 2 bits, then this means that the receiver must capture 2 bits, then 20 bits, 20 bits, 20 bits, etc. In order to make the bit size of the header the same as the bit size of the pixel, one or more dummy bits can be added to the header by the serializer of the transmitter.

[0218] Figure 6 The modules or circuitry 602 shown in FIG. 6 accordingly configures the receiver 410 of the viewing device 312 to buffer data until the first frame line of the video frame is received. Figure 7 As shown in , in some embodiments, at the viewing device 312, the receiver waits for the first frame row of the video frame in the received serialized video data by checking the received data structure to find the first frame row position indicator in the header. Once all valid pixels of the first row of the video frame are received, the modules or circuit systems 604, 605 configure the receiver 410 to add zero, one or more blanking pixels to follow the valid pixels. In some embodiments, these added blanking pixels cause the viewing device to generate variable line length video data, which allows a continuous stream of serialized frame rows to be output to the display 414. The module or circuit system 606 configures the receiver 410 to provide the valid pixels and subsequent blanking pixels as a continuous stream of pixels output to the display 414 until the next group of valid pixels is received and streamed to the display, etc. For example, in an example where the display 414 is a raster type display, where video rows of pixels are displayed from right to left, the first pixel sent is displayed on the right, and the subsequent pixels of the valid pixel group are displayed from right to left along the horizontal line.

[0219] In some embodiments, for each received pixel group corresponding to a frame row of a video frame, once all valid pixels sent to the display have been displayed, the receiver sends blanking pixels until the next frame row of the video frame can be output. Alternatively, if the last frame row output is the second to last frame row in the frame, then the receiver 410 or generates the end of the frame row including blanking pixels. In each case, the frame row data is then provided to the display 414 for presentation. After the valid pixels of each frame row are presented on the display, all subsequent pixels will be considered blanking pixels and not displayed until the next row indicator label is received.

[0220] Which pixel in the pixel group is the first pixel in the video frame row is identified at the interface of the display, and this conforms to standard techniques for raster type displays (e.g., LCD or OLED type displays). In some embodiments, this can also be encoded within the header structure for the pixel or pixel group. In other words, it is known in the art to provide another signal to indicate to the display 414 which pixel is the first pixel of a subsequent row (thus, by counting pixels, beyond a certain number of pixels, the display 414 will discard any additional pixels as blanking pixels). Such a signal sent for each row is used to indicate the first pixel and can conform to, for example, a standard VGA or HD signal format.

[0221] The form of display 414 may vary in various embodiments, and may include a near-eye display in some embodiments.

[0222] Although the above-described embodiments relate to a single video source, in some embodiments, additional data (such as graphics overlay data) may be provided from one or more other sources and merged with the video image data at video source 102 .

[0223] In some of the above embodiments, as well as in some of the embodiments described below, additional control modules or devices may be provided at the viewing device and / or device to control the receiver and transmitter in response to executing computer code so as to implement the relevant methods disclosed herein at the transmitter device and the receiver device.

[0224] In the attached figure Figure 8 An example embodiment of a video display system 800 according to the disclosed technology is shown. The video display system 800 includes a video hub, also referred to herein as a video combiner device 810 according to any of the embodiments of such devices 810 disclosed herein, and a viewing device 824 according to any of the disclosed embodiments of such devices disclosed herein. The video combiner device 810 may include one or more video sources 812a, 812b, or these video sources may be provided separately, such as Figure 8 . The video sources 812a, 812b are each configured to simultaneously provide a stream of serialized video data intended for synchronized display on two displays 834a, 834b at the viewing device 824. The viewing device 824 may be connected to or include at least one of the two displays 834a, 834b. In some embodiments, each display 834a, 834b is configured to present video received from one of the video sources 812a, 812b. The viewing device may be connected to the combiner device via a serialized data link, such as a serialized data interface (SDI) data link. The viewing device 824 may be provided as part of the video combiner device 810, in which case, Figure 8The transmission channel 822 shown in can be provided by an internal wired or wireless data communication channel, which may or may not conform to the SDI data standard. The display interfaces are configured to each receive a serialized video data stream and present the received data on a display without or with minimal buffering.

[0225] exist Figure 8 In one embodiment of the system shown in FIG. 8 , a video display system 800 includes a video combiner device 810 configured to process two serialized video data streams 814 a , 814 b .

[0226] Each video stream includes video frame data captured by a video source 812a, 812b. The device 810 includes a transmitter 820, which includes a serializer 819, which is configured to combine the two parallel streams of video data 814a, 814b into a single serialized data stream at the transmitter 820 on a frame-by-frame basis for onward transmission. The transmitter 820 outputs the combined serialized data for onward transmission to a viewing device 824 via a serialized data transmission channel 822. The output may include a continuous stream or a discontinuous data stream that is transmitted intermittently, depending on the timing when a complete frame line from either source is ready for transmission.

[0227] The combiner device 810 includes a memory 815, which may include one or more buffers 816a, 816b for buffering received video data streams 814a, 814b at a receiver 818 of the device 810. The device 810 also includes one or more processors or processing circuit systems 817, which are configured to process the valid pixels of the buffered data streams 819a, 818b to associate them with information such as their video source and, if not already provided, with the row position in the video frame of each pixel group. In some embodiments, the one or more processors or processing circuit systems may form part of a serializer 819. The serializer 819 may be provided as part of a transmitter 820, which is arranged to output video data from the two buffers 816a, 816b in a serialized combined data stream 822. The buffers 816a, 816b are each arranged to process the received data streams 814a, 814b in parallel to forward the data to the serializer 819 concurrently.

[0228] The viewing device 824 includes a receiver / display driver 826 configured to receive the serialized combined video stream 822 from the video combiner device 810. The receiver 826 is configured to check the header information of each received valid pixel group, and includes a deserializer 827 configured to deserialize the valid pixels into two parallel channels 828a, 828b based on the header information, which indicates that they have different video sources 812a, 812b. References to "headers" herein should also be interpreted as references to trailers and / or other specified locations that may be used instead in some embodiments.

[0229] The receiver / LCD driver 826 also adds blanking pixels to each group of active pixels and can buffer the output until the first frame row of video data is provided so that the outputs 830a, 830b, shown as the left output (LEFT OUT) 830a and the right output (RIGHT OUT) 830b to each display 834a, 834b, respectively, will be displayed correctly.

[0230] Each output data stream 832a, 832b includes a continuous stream of active pixels and blanked pixels. At each of the displays 834a, 834b, the blanked pixels of each output data stream 832a, 832b are discarded, and only active pixels are presented on each video frame line in the proper order and each video frame line is correctly positioned in the frame.

[0231] In some embodiments, sometimes or always, Figure 8 The disclosed viewing device may accordingly receive a combined data stream 822 that has undergone row-by-row interleaving of groups of pixels from the different video sources 812a, 812b to the combiner device 810. Furthermore, by omitting at least one blanking pixel, the transmission time to send video from both sources to the viewing device 824 is reduced because there are fewer pixels to transmit over any shared link used for onward transmission.

[0232] In some embodiments, the video combiner device 810 may include two or more video sources 812a, 812b, but in some embodiments, they may be provided separately. In some embodiments, the video combiner device 810 and the viewing device may also be combined together.

[0233] In some embodiments, the viewing device 824 includes two or more displays 834a, 834b, wherein each different display is configured to present video received from a different video source 812a, 812b, the video data being forwarded by the combiner device 810 over the same video data channel 822. However, in some embodiments, the displays may be provided separately.

[0234] In some embodiments, the displays 834a, 834b are provided as left and right near-eye displays, such as left and right displays of a head-mounted device. In some embodiments, the displays 834a, 834b may be connected to the viewing device 824, but separate from the viewing device 824, to make the head-mounted device less bulky or to reduce the weight of the head-mounted device that accommodates the displays 834a, 834b. In some embodiments, the viewing device 824 may provide a dedicated device, a portion of the display, implemented using a tablet or similar computer type device, and may provide output to an actual screen or virtual screen partitions to replicate the left and right displays. However, in other use cases, the head-mounted device or near-eye display may include both left and right displays, for example, if the entire viewing device is sufficiently compact / lightweight, etc. In some embodiments, a combination of different displays may be provided, for example, if a suitable driver is configured, the viewing device 824 may output video to a near-eye display and simultaneously or nearly simultaneously to a large format display screen.

[0235] In some embodiments, the video combiner device (also referred to herein as a hub) 810 may include a hub co-located with a surgical imaging device. In some embodiments where the hub 810 includes a hub co-located with the surgical imaging device, the hub 810 is configured to receive serialized video streams from two or more video sources (e.g., left view and right view video sources). In some embodiments, each video source includes a camera and each camera is part of the surgical imaging device. In some embodiments, the hub 810 is or is part of a surgical imaging device including a surgical microscope. In some embodiments, the hub 810 may be co-located with the surgical microscope and / or may be part of a control system or provided as other computing components or units of the surgical microscope.

[0236] exist Figure 8 In the embodiment of the system shown in , there are two separate video sources 812a, 812b, each configured to provide a separate serialized variable line length video stream 814a, 814b to the combiner device 810. In some embodiments, more than two video sources may be configured to provide serialized video data to the video combiner 810. In some embodiments, the images from each video source are synchronously presented on a different display at the viewing device 824. However, in some embodiments, additional information may be added to one or more of the video data streams, such as superimposed image data or information, which may come from another source and be combined with the video image data from one of the image video sources 812a, 812b.

[0237] Overlay type information intended to be presented on the same display as the video image from source 812a, 812b can be fused or otherwise integrated with the video information at source 812a, 812b or at combiner device 810. Such overlay information can, for example, modify how the video is presented on the display, for example, it can highlight or adjust the color or contrast of one of the images presented on one or both displays. Alternatively or additionally, the overlay information can provide text or symbols in the video presentation. In some embodiments, the overlay information is synchronized with two or more video sources of video data so that in some embodiments, the same overlay information source can be split and combined with two serialized video streams 814a, 814b.

[0238] In some embodiments, if the number of video sources does not match the number of separate displays, the video data can be fused so that the effective pixels presented on one of the displays can include data from more than one source. In this case, in some embodiments, the data structure header information can include an identifier that indicates in some embodiments that the data structure includes pixels for superposition and image data from two sources (or other forms of combined data).

[0239] In some embodiments, the hub can process the video data received from the source to add overlays or image enhancements. This can be embedded in the video stream output by the hub, or in some embodiments can be provided as a separate stream.

[0240] In some embodiments of the disclosed technology, a video combiner device 810 includes a device for combining video from at least two different video sources 812a, 812b (e.g., Figure 8As shown in the embodiment of the present invention, a device 810 for combining multiple serialized video streams 814a, 614b from two different video sources 812a, 812b). The video combiner device 810 includes a receiver 818, which is configured to receive each serialized video stream 814a, 814b from different video sources 812a, 812b through a separate channel. Each received serialized video stream includes a plurality of video frames, and the video frames include rows of valid pixels and blanked pixels. The combiner device 810 also includes one or more processors or processing circuit systems 817 and a memory 815. The memory 815 may include a plurality of buffers 816a, 816b, each buffer being arranged to buffer at least valid pixels of the received serialized video streams 814a, 814b. The one or more processors or processing circuit systems 817 are configured to, for each group of valid pixels in a row of pixels completely received in a received frame of the received serialized video stream 814a, 814b, assign a video source indicator to the valid pixel group, include or assign a frame row position indicator to the valid pixel group, and generate a data structure including the valid pixel group, the source indicator of the pixel group, and the frame row position indicator assigned to the valid pixel group. The combiner device 810 also includes a transmitter 820, which is configured to transmit the generated data structure from each received serialized video stream 814a, 814b onward as a single serialized video stream 822.

[0241] In some embodiments, one or more processors or processing circuit systems 817 may be provided as part of the transmitter 820 of the combined device 810. In some embodiments, the one or more processors or processing circuit systems may be configured to perform operations such as by loading and executing computer program code (such as Fig.11 The computer program code 1106 shown in FIG. 1 is configured to store the computer program code 1106 in a suitable form of memory 815 .

[0242] At the viewing device 824, in some embodiments, one or more processors or processing circuitry 836 may also be provided, for example, as part of the receiver 826 or deserializer 827. In some embodiments, the one or more processors or processing circuitry 836 may be configured, for example, by loading and executing computer program code (such as a memory 838) from a suitable form of memory 838. Fig.121200) to cause a viewing device to implement a method for viewing multiple serialized video streams received as a single serialized combined video stream. For example, in some embodiments, the method includes receiving a serialized video stream including a plurality of data structures, each data structure including an active pixel group, a source indicator assigned to the pixel group, and a frame line position indicator assigned to the active pixel group; and generating a complete line of video data by adding one or more blanking pixels to each active pixel group of the received data structures until a data structure including the next pixel group in the serialized video stream is received, and causing each complete line of video data to be output for presentation on a display associated with the source indicator of the complete line of video at a position in the video frame indicated by the frame line position indicator of the active pixel assigned to the complete line of video data.

[0243] In some embodiments, the viewing device 824 includes a receiver 826 configured to receive a serialized video stream 822 including data from a plurality of different sources 812a, 812b, such as a combined serialized video stream 822 from the combiner device 810. The serialized video stream 822 includes a plurality of data structures, each data structure including an effective pixel group, a source indicator assigned to the pixel group, and a frame line position indicator assigned to the effective pixel group. The viewing device 824 also includes one or more processors or processing circuit systems 836 configured to generate a complete row of video data by adding one or more blanking pixels to each effective pixel group of the received data structure until a data structure including the next pixel group in the serialized video stream is received. Each complete row of video data is then provided for presentation on a display 834a, 834b associated with the source indicator of the complete row of video at a position in the video frame indicated by the frame line position indicator of the effective pixel assigned to the complete row of video data.

[0244] In the attached figure Fig. 9 Some embodiments of the operation of the combiner device 810 are schematically illustrated (eg, Figure 8 In an embodiment of the apparatus 810 schematically shown in FIG. 8 , how the sender 820 can occasionally pause the forward transmission between data structures.

[0245] exist Fig. 9 In the example, serializer 819 is not shown for clarity. The pauses are labeled TX PAUSE #1 to #5 and may occur from time to time between transmissions of data structures, or may always occur between data structures. Fig. 9, the data structures are labeled A#1, B#1, A#2, B#2, A#3, where A# indicates that the data structure includes valid pixels originating from video data source 812a and B# indicates that the data structure includes valid pixels originating from 812b.

[0246] Transmission pauses may occur at regular or irregular intervals depending on the timing of transmitting data structures and the time it takes for the next data structure to be ready for transmission after the transmission of the previous data structure is completed.

[0247] Each data structure including a group of valid pixels is assigned a source indicator at the combiner device before being transmitted onward, the source indicator being included in the header of the pixel group. Additional header information may include a frame line position indicator for the pixel group and any other header information that may be retained in the serialized data received from the video source. In some embodiments, the transmission of a frame line from one source 812a may be completed before the next data structure including a different valid pixel group with a different source indicator #B is ready for transmission, in which case, if another data structure from the first source is ready for transmission, that data structure is transmitted instead of pausing, rather than waiting for another source frame line to be ready.

[0248] like Fig. 9 As shown in FIG. 1 , the first frame line (at Fig. 9 1). Then, since no subsequent data structure is ready for transmission, a transmission pause marked as TX#1 occurs. However, once a subsequent data structure B#1 is ready for transmission, it is output along the serialized video data link 822. In the example shown, the next data structure ready for transmission is marked as B#1, and after the transmission occurs another pause, pause #2, data structure #A2 is ready for transmission, then that data structure is transmitted, and another pause #3 occurs before B#2 is ready for transmission, then transmitted, after which pause #4 occurs, then data structure A#3 is transmitted, followed by another pause.

[0249] However, the transmitter 820 may transmit a continuous stream of pixels by alternating which source pixel groups are transmitted, so that after transmitting a data structure including valid pixels of a video frame from a first video source 812a, if the data structure is ready for transmission when the data structure including pixels from the first video source 812a is completed, the transmitter next transmits a data structure including valid pixels of a video frame from another video source 812b. In other words, in some examples of the disclosed technology, some or none of the pauses shown may occur, and the sequence of frame lines may not strictly alternate from one source to another.

[0250] In some embodiments, each frame line comprises a horizontal line of a video frame when displayed.

[0251] In some embodiments, the video source indicator identifies a spatial relationship between at least two of the plurality of video sources. Figure 8 As shown in , in some embodiments, two video sources 812a, 812b provide left and right views of a scene, and a receiver 826 of a viewing device 824 is configured to examine the received video stream 822 for an indicator for each video source and to allocate the active pixel group of a data structure associated with the source indicator to one of the two displays 834a, 834b accordingly.

[0252] In some embodiments, the multiple serialized video streams 814a, 814b include a video stream for a left view of the scene obtained from a video source and a video stream for a right view of the scene obtained from a video source, and different video source indicators are used to identify whether a pixel group belongs to a video frame for the left or right view of the scene obtained from the video source.

[0253] In some embodiments, the assigned video source indicator and frame row position indicator are included in a header of the data structure, and wherein the bit size of the header matches the parallel width of a serializer of a transmitter 820 configured to combine data structures from multiple different video sources 812a, 812b into an outgoing serialized video stream 822.

[0254] In some embodiments, the data structures from the two sources 812a, 812b are interleaved by the serializer in the outgoing serialized video stream 822.

[0255] In some embodiments, interleaving alternates data structures from one video source with data structures from another video source in the outgoing serialized video stream. In some embodiments, if the data structure from the other video source is not yet ready for transmission, the transmitter 820 will pause transmission until it is ready for transmission. Alternatively, in some embodiments, the transmitter may transmit a data structure of a frame row from the same source as the previously transmitted frame row.

[0256] The apparatus 810 may be configured to receive at least one serialized video stream 814a, 814b via a channel having one or more of a different data rate and a different video resolution than at least one other channel via which another received video stream 814a, 814b is received.

[0257] Although latency at displays 834a, 834b is better reduced by removing all blanking pixels in the video frame rows received at device 810, thereby preferably not including any blanking pixels in the received video frame rows in the outgoing serialized video stream 822 including the forward transmitted data structure, a similar benefit of reduced latency can be achieved by instead removing and discarding at least one blanking pixel in each received video frame row that includes a set of valid pixels and at least one blanking pixel and including the set of valid pixels and any remaining blanking pixels in the received video frame row in the forward transmitted data structure of the outgoing serialized video stream.

[0258] In some embodiments, the video combiner device 810 decouples the source clock of the received serialized video stream in the outgoing serialized video stream by switching between different video sources when combining serialized video data for forward transmission and / or pausing the transmission of a data structure when the data structure is not ready for transmission after the transmission of the previous data structure. In some embodiments, the receiver is decoupled from the source clock accordingly, and can be further decoupled by varying the duration of the pause between the forward transmitted data structures. The transmitter clock of the serializer 819 can also be decoupled from the clock at the receiver 826 of the viewing device 824 accordingly.

[0259] In some embodiments, transmitter 810 is configured to transmit the outgoing serialized video stream over data communications channel 822 along a wired link and / or over a wireless data communications channel to viewing device 824, which is configured to cause display of the received video data.

[0260] In the attached figure Fig.10 1 shows an example embodiment of a method 1000 performed by a combiner device 810 configured to combine two parallel streams of serialized video data. Fig.10 As shown in , method 1000 includes receiving 1002a from a video source (e.g., Figure 8 or Fig. 9 812a) and concurrently receives 1002b a video stream from another video source (e.g., Figure 8 or Fig. 9812b). Each stream is then buffered separately 804a, 804b until a complete video frame row is received in 1006a, 1006b, at which point, in some embodiments, a header associated with the valid pixels of each video frame row is populated 1008a, 1008b, where an indicator or tag provides an indication of the video source of the video frame for each group of valid pixels. The header may already include or be assigned a video frame row indication and any other relevant information (such as which pixel in the group of valid pixels is the first pixel that should ultimately be presented on the display of the video frame at the viewing device 824). A data structure 1010a, 1010b including the valid pixels and header information is then generated, and which of the two resulting data structures is then ready for transmission 1012, is transmitted forward in 1014, for example, towards Figure 8 and Fig. 9 824 shown in . In some embodiments, another data structure may be buffered until it can be transmitted, in which case the forward transmission may be continuous from one data structure to another, and in some embodiments the data structures may alternate sources. If no data structure from either source is ready for transmission after the previous data structure is transmitted, then the sender may pause transmission 1016 until the next data structure is ready for transmission.

[0261] Since any pause in transmission will disrupt the timing of the pixel groups being transmitted from their original clock timing, the clock at the viewing device can be decoupled from the transmission clock at the original video source 812a, 812b.

[0262] In some embodiments, the method 1000 includes combining multiple serialized video streams 814a, 814b from different video sources 812a, 812b, each serialized video stream received through a different communication channel for serialized forward transmission through the channel. In some embodiments, the method includes: receiving 802a, 802b each of the multiple serialized video streams separately; buffering 804a, 804b at least valid pixels of the received frame row of the serialized frame of each of the multiple received video streams separately; processing each buffered valid pixel group in the fully received frame row by at least the following steps: assigning 808a, 808b a video source indicator to the valid pixel group; and generating 810a, 810b a data structure including a valid pixel group, an indicator of the video source of the pixel group, and a frame row position indicator of the valid pixel group; and forwarding the generated data structure through a common channel, the data structure including the buffered valid pixels of the frame row of the video frame from one of the multiple received serialized video streams.

[0263] In some embodiments of method 1000, the method further includes, after each data structure has been transmitted, determining whether another data structure in a video frame row of a video frame originating from another video source among the plurality of video sources is ready to begin forward transmission, and if so, then forwarding the data structure from the other video source among the plurality of video sources over the common channel, otherwise determining whether another data structure from one of the plurality of video sources is ready for transmission, and if so, then forwarding the data structure from the one of the plurality of video sources over the common channel.

[0264] In some embodiments of the method, after the data structure has been transmitted, the method further includes determining that no data structure from any of the plurality of video sources is ready for transmission, pausing transmission until the data structure from a video source in the plurality of video sources is ready for transmission, and transmitting the data structure forward over the shared channel.

[0265] Fig.11 The schematic diagram shows how the device 812 may include computer code 1106, for example, in the form of a computer program stored in a memory or configured in hardware, which can be executed by one or more processors or processing circuit systems 817. In some embodiments, the computer program code includes controlling various components of the combiner device 810 to implement functions such as Fig.10 One or more modules or circuitry of a method such as method 1000 shown in .

[0266] Fig.11 The computer code 1106 is illustrated as how it may include one or more modules or circuitry 1108 that, when executed, causes the generation of a data structure having a header, and one or more modules or circuitry 1110 that, when executed, causes the filling of the header. In some embodiments, the computer code 1110 includes one or more modules or circuitry 1202 that, when executed, causes the combiner device 810 to determine or check if there is an indicator of a video line position in a frame of a valid pixel group in the received video data stream. The frame line indicator is included in the forward transmitted data structure and may indicate, for example, whether the valid pixels of a video frame line belong to the first line of the video frame.

[0267] If there is not already a frame row position indicator associated with a valid pixel group, then the header is filled with the frame row indicator.In some embodiments, the frame row indicator may include a binary indicator indicating whether the frame row is the first frame row.

[0268] The computer code 1106 also includes one or more modules or circuitry 1204 configured to, when executed, cause the header to be populated with any other relevant information. For example, the computer code 1106 may include one or more modules or circuitry configured to, when executed, cause an indicator or tag to be added to the header to identify the video source of the valid pixel set or indicator in 1116, wherein the viewing device is configured to send the valid pixels associated with the particular video source identifier to a particular one of its displays 834a, b. Alternatively or additionally, in some embodiments, the indicator or tag may identify the display on which the valid pixel set is to be displayed, in which case the viewing device, upon detecting a particular display indicator, is configured to forward the received valid pixels (and any added blanked pixels) with the particular display indicator to the particular display.

[0269] In some embodiments, other relevant information may include indicators or labels added to indicate which pixel in each row is displayed first and / or the direction in which the row is presented on the display (eg, from left to right, etc.).

[0270] As will be apparent to one of ordinary skill in the art, in the above-described embodiments, the header may alternatively or additionally be provided as a tail, or located at another predetermined position relative to the valid pixel group in the forward transmitted data structure.

[0271] Fig.12 Schematically illustrates how the viewing device 824 includes one or more processors or processing circuitry 838, memory 836, and a receiver 828, such as a receiver / driver for multiple displays 834a, 834. Fig.12 As shown in , the displays are included in the viewing device, but in some embodiments, they can be separate displays. In some embodiments, the viewing device 824 can be configured to receive video routed from the combined device 810. In some embodiments, the displays 834a, 834b include left and right near-eye displays in a head-mounted device or similar near-eye display viewing device.

[0272] like Fig.12 As shown in , the memory 836 includes memory storing computer code 1200. The computer code 1200 may include one or more modules or circuit systems 1202, 1204, 1206, which, when executed by one or more processors or processing circuit systems 838, are configured to cause the viewing device 824 to perform a method for viewing multiple serialized video streams received as a single serialized video stream.

[0273] In some embodiments, for example, the computer code includes one or more receiver modules or circuit systems 1202 that, when executed, cause the viewing device 824 to receive a serialized video stream 822 comprising a plurality of data structures, each data structure comprising an active pixel group, a source indicator assigned to the pixel group, and a frame line position indicator assigned to the active pixel group. The computer program code may also include one or more deserialization modules or circuit systems 1204 that, when executed by the viewing device 824 or by a deserializer 827 of the device 824, are configured to process the received serialized video stream 822 to generate a complete line of video data by adding one or more blanking pixels to each active pixel group of the received data structure until a data structure comprising the next pixel group in the serialized video stream is received.

[0274] The computer program code may also include one or more data output modules or circuit systems 1206 that are configured, when executed, to cause output of each complete line of video data for presentation on a display 834a, 834b associated with the source indicator of the valid pixel at a position in the video frame indicated by the frame row position indicator of the valid pixel assigned to the complete line of video data.

[0275] In the above embodiments, the video received by the device 302, 802 from one or more video sources 102, 812a, 812b can be received from the video source via the SDI link 14. The memory 306 may include a suitable read-only memory and / or random access memory that stores computer code for execution by one or more processors or processing circuit systems 308. The memory 306 may also include at least one line buffer 305, 816a, 816b for storing video data for each horizontal line received from the video source 102, 812a, 812b via the SDI link. When the buffer 306, 816a, 816b is full, in other words, when a complete horizontal line of valid pixels and blanking pixels have been received, the blanking pixels are discarded or otherwise ignored or removed, and the valid pixel group is processed for forward transmission. In some embodiments, the forward transmission may not include any pauses and may be continuous. In some embodiments, if some data structures are buffered and ready for transmission when the device 310, 810 completes transmitting the previous data structure, then they can be transmitted without a pause, however, sometimes there can be a pause if no data structures are ready.

[0276] The display 414, 834a, 834 may include any suitable type of display, such as a raster scan type display, for example, a display that uses liquid crystal display (LCD) or light emitting diode (LED) or organic light emitting diode (OLED) technology to display video.

[0277] Video data is transmitted in a serial manner, that is, pixel by pixel, over a serialized data interface (SDI) link or channel. The order in which the bits of each pixel (usually 24 bits) are transmitted can be determined in different ways by different communication protocols. However, if it is serialized, the most significant bit of each pixel is sent first, and the remaining pixels are sent one by one until the last pixel is sent. Depending on the required video resolution, the serialized video stream can run at speeds up to 6Gbit / s or 12Gbit / s.

[0278] In some embodiments, the transmission time of each frame is different, and the device 310, 810 simply matches the average frame period of the forward transmission to the frame period of the video received from the video source 102, 812a, 812b. In other words, the source clock used by the device 310, 810 to transmit each pixel does not need to be the same as the display clock used at the display 314, 834a, 834b to receive each transmitted pixel.

[0279] exist Figure 3A and Figure 8 In the embodiment described herein, the transmitters 312 and 820 are respectively configured to ensure that the average transmitted frame period matches the transmission time of each video source. In other words, in some embodiments, the disclosed transmitters 312 and 820 are configured to allow the transmission period of each pixel in the video frame to be different. In some embodiments, the idle period during the transmission is used to compensate for the extra / fewer valid pixels from (one or more) video sources due to the clock difference between the device 310, 810 and the source 102.

[0280] For example, now return to Figure 4, while the following is also relevant to embodiments combining video from two sources, in pixel group 402a, as an example embodiment, the clock period for transmitting pixels may be X=0.1000 μs plus δ=0.0001 μs, in other words, the clock is X+δ=0.10001 μs, and the clock period of the receiver of display 312 for receiving pixels of blanking lines may be X=0.1000 μs minus δ=0.0001 μs, in other words, X-δ=0.999 μs. In other words, in some embodiments of the disclosed technology, the receiver 410, 824 of the viewing device 410, 824 uses a local clock to drive the output to the display 414, 834a, 834b, in other words, the viewing device may use a local clock to drive the rate at which pixels are presented on the display (e.g., (one or more) LCD panels), and does not need to derive a clock from the transmitter 310, 820. This allows the transmission time for individual frames to vary within a certain tolerance level, in other words as long as on average the frame transmission duration does not vary excessively.

[0281] Variations in the arrival time of each horizontal line of serialized video data are compensated for by varying the blanking period that is added to the end of each received active pixel group 402a-402d of the video frame by the display's driver 412, 824. As mentioned above, these blanked pixels are not displayed, but can compensate for differences between the receiver clock of the viewing device 410, 824 and the source clock at the transmitter 310, 820 and / or the source clock of the video source 102, 812a, 812b.

[0282] Because display added blanking pixels are used to compensate for different video line arrival times at the display, the pixels forming the pixel columns output on the display panel(s) 414, 834a, 834b of the viewing device 410, 824 can be synchronized at the display even if two or more separate video sources are present.

[0283] Some embodiments of the disclosed technology are configured to provide video output to a viewing device 410, 824 that is configured to drive a display 414, 834a, 834b (e.g., an LCD panel display) having variable length horizontal video frame lines. In some embodiments, the variable length horizontal frame lines are set to catch up with the source video horizontal timing, but this is not necessary to lock to the video source clock timing. Thus, some embodiments may use a proprietary serial data interface type of video delivery method that does not require locking to the video source clock at the display.

[0284] In some embodiments, the viewing device 410, 824 is configured to automatically add a variable number of video blanking pixels for each pixel group comprising a horizontal line of valid pixels received from the device 310, 810 until the last valid pixel of the next complete video line is received. Advantageously, this can allow error correction to be performed on the next received video frame line.

[0285] In some embodiments of the disclosed technology, device 302 receives two serialized video streams from different sources. Figure 5 An example embodiment of apparatus 302 is shown, which includes apparatus 510 .

[0286] In some embodiments, the video data being transmitted forward is compressed, and one or more communication links are configured to have a total effective bandwidth that is less than the sum of the effective bandwidths of each source.

[0287] In some embodiments, the video source 12, 512a, 512b may be an RGB camera source.In some embodiments, the display 312 or display system 518 comprises a near-eye display, such as a stereoscopic near-eye display.

[0288] The display may include a microdisplay or liquid crystal display that specifies a fixed number of blanking pixels following a video line of active pixels, for example, 1920 active pixels followed by 40 blanking pixels, but it has been found that the disclosed viewing device can provide 1920 active pixels followed by 39 or 41 blanking pixels, etc., without significantly affecting how the video is displayed. Thus, in some embodiments, the viewing device does not need to add a fixed number of blanking pixels at the end of each active pixel group. Advantageously, the disclosed embodiments, the combiner device 810 that receives video from two or more sources allows the two or more video sources 812a, 812b to have independent clocks and still share the same transmission cable. In some embodiments, an additional video frame buffer may be used, but in the absence of such a frame buffer, the video received by the display 16, 518 can have a low or very low latency, such as a latency of about tens of μs.

[0289] The embodiments disclosed above may allow displays such as LCD panels to no longer need to obtain their clock from the source of the video they receive for display. Compared to using a conventional SDI transmitter that requires locking the source video clock, the disclosed technology allows video from two or more different video sources to be transmitted over the same or different physical channels, even if the clocks of the video sources are not synchronized. In addition, by transmitting data line by line in the manner disclosed herein, the long delays required when using conventional SDI transmitters can be avoided because there is no need to have a video frame buffer.

[0290] In some embodiments, the displays 834a, 834b comprise near-eye stereoscopic displays based on diode technology, such as LCD or OLED displays.

[0291] The program code mentioned above can also be provided as a computer program product, for example, in the form of a data carrier carrying a computer program code or code means, which is used to perform the embodiments of this article when loaded into the processing circuit system in the control unit. The data carrier or computer readable medium can be one of an electronic signal, an optical signal, a radio signal or a computer readable storage medium. The computer program code can be provided in the control unit or on a server and downloaded to the control unit as pure program code, for example. Therefore, it should be noted that in some embodiments, the function of the control unit can be implemented as a computer program stored in a memory (e.g., a computer readable storage unit) for execution by a processor or a processing module (e.g., a processing circuit system in a control unit).

[0292] Those skilled in the art will also recognize that the above-mentioned processing circuit system and memory or computer-readable storage unit may refer to a combination of analog and digital circuits, and / or one or more processors configured with software and / or firmware (e.g., stored in memory), which, when executed by one or more processors (such as processing circuit system), is executed as described above. One or more of these processors and other digital hardware may be included in a single application-specific integrated circuit (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether packaged independently or assembled into a system on a chip (SoC).

[0293] In some embodiments, the control unit may also include or be able to control how the signal is sent along the communication link 103, 822 (which may be a wired or wireless communication link). If the signal is sent wirelessly, the control unit may also appropriately control the operation of the antenna.

[0294] The communication channel may be point-to-point, or networked, for example, via a cellular network or satellite network supporting wireless communications. The wireless communications may conform to one or more public or proprietary communications standards, protocols, and / or technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), and / or Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS)), or any other suitable communications protocol, including communications protocols that have not yet been developed as of the filing date of this document.

[0295] The operating system of the disclosed technology may require various other software components and / or drivers to control and manage general system tasks (e.g., memory management, storage device control, power management, etc.) and, in some embodiments, facilitate communication between various hardware and software components, which are not shown in the figures for clarity when their inclusion is obvious.

[0296] When the disclosed technology is described with reference to the accompanying drawings in the form of block diagrams and / or flowcharts, it should be understood that several entities in the drawings, such as the blocks in the block diagrams, and combinations of entities in the drawings, can be implemented by computer program instructions, which can be stored in a computer-readable memory and can also be loaded onto a computer or other programmable data processing device. Such computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, and / or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer and / or other programmable data processing device create components for implementing the functions / actions specified in the block diagram and / or one or more flowchart blocks.

[0297] In some embodiments, according to some aspects of the present disclosure, the functions or steps indicated in the square blocks may not occur in the order indicated in the operating instructions. For example, two square blocks shown in succession may actually be executed substantially concurrently, or the square blocks may sometimes be executed in reverse order, depending on the functionality / actions involved. Moreover, according to some aspects of the present disclosure, the functions or steps indicated in the square blocks may be executed continuously in a cycle.

[0298] The description of the example embodiments provided herein is presented for illustrative purposes. This description is not intended to be exhaustive or to limit the example embodiments to the precise form disclosed, and modifications and variations are possible in light of the above teachings, or may be obtained by practicing various alternatives to the embodiments provided. The examples discussed herein are selected and described in order to explain the principles and properties of the various example embodiments and their practical applications, so that those skilled in the art can utilize the example embodiments in various ways and with various modifications suitable for the intended specific use. The features of the embodiments described herein may be combined in all possible combinations of methods, devices, modules, systems, and computer program products. It should be appreciated that the example embodiments presented herein may be practiced in any combination with one another.

[0299] It should be noted that the word "comprising" does not necessarily exclude the presence of other elements, features, functions or steps than those listed, and the word "a" or "an" before an element does not exclude the presence of a plurality of such elements, features, functions or steps. It should also be noted that any reference signs do not limit the scope of the claims, that the exemplary embodiments may be implemented at least in part by means of both hardware and software, and that several "components", "units" or "devices" may be represented by the same hardware item.

[0300] The various example embodiments described herein are described in the general context of methods and may refer to elements, functions, steps, or processes, one or more or all of which may, in one aspect, be implemented by a computer program product embodied in a computer-readable medium, the computer program product including computer-executable instructions, such as program codes, executed by computers in a network environment.

[0301] Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), which may be static RAM (SRAM) or dynamic RAM (DRAM). ROM may be a programmable ROM (PROM) or an erasable programmable ROM (EPROM) or an electrically erasable programmable ROM (EEPROM). Suitable storage components for memory may be integrated as chips into a printed circuit board or other substrate connected to one or more processors or processing modules, or provided as removable components, for example, by flash memory (also known as USB stick), compact disk (CD), digital versatile disk (DVD) and any other suitable form of memory. Unless not suitable for the application at hand, memory may also be distributed on various forms of memory and storage components, and may be provided remotely on one or more servers, such as may be provided by a cloud-based storage solution. In general, program modules may include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of program codes for executing the steps of the methods disclosed herein. A specific sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in such steps or processes.

[0302] Thus, regardless of the form of the electronic device described herein, the memory used by any device includes any suitable device readable and / or writable medium, examples of which include, but are not limited to: any form of volatile or non-volatile computer readable memory, including, but not limited to, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (e.g., hard disk), removable storage media (e.g., flash drive, compact disk (CD) or digital video disk (DVD)), and / or any other volatile or non-volatile, non-transient device readable and / or computer executable memory device that stores information, data and / or instructions that can be used by the processing circuit system. The memory can store any suitable instructions, data or information, including computer programs, software, applications including one or more of logic, rules, codes, tables, etc., and / or other instructions that can be executed by the processing circuit system and used by the device in any form of electronic device. The memory can be used to store any calculations performed by the processing circuit system and / or any data received via a user or communication or other type of data interface. In some embodiments, the processing circuit system and the memory are integrated. The memory may also be distributed among one or more system or device components. For example, in some embodiments, the memory may include multiple different memory modules, including modules located on other network nodes.

[0303] In the drawings and the specification, exemplary aspects of the present disclosure have been disclosed. However, many variations and modifications may be made to these aspects, all of which are within the scope of the appended claims. Therefore, the present disclosure should be considered illustrative rather than restrictive in supporting the scope of the claims, which are not limited to the specific examples of the above-mentioned aspects and embodiments. The invention illustrated herein by the various aspects and embodiments described above has a scope defined by the following claims.

Claims

1. An apparatus (302) for processing a serialized stream of video data for forward transmission, the serialized stream of video data comprising a plurality of serially transmitted video frames from a video source, each video frame comprising a plurality of frame lines (202a-202e), at least one frame line (202a-202d) comprising a group of valid pixels and one or more blanking pixels, the apparatus (302) comprising: a receiver (304) configured to receive the serialized video data stream; a memory (306) arranged to buffer the received serialized video data until at least all valid pixels of a frame row of pixels of a received video frame have been buffered; one or more processors or processing circuitry (308) configured to generate a data structure for each group of active pixels of a frame row of pixels, the data structure comprising at least the group of active pixels for the frame row and a frame row position indicator assigned to the group of active pixels, each data structure comprising a group of active pixels from a received frame row and omitting at least one blanked pixel of the received frame row; as well as A transmitter (310) is configured to intermittently forward the generated data structures by pausing transmission after each data structure is transmitted until the next data structure is ready to begin transmission.

2. The apparatus (302) of claim 1, wherein more than one blanked pixel is discarded from each received video frame line that includes a group of valid pixels and at least one blanked pixel.

3. An apparatus (302) as claimed in claim 1 or 2, wherein none of the blanked pixels of the received video frame lines are included in an outgoing serialized video stream including a data structure for forward transmission.

4. The apparatus (302) of any one of the preceding claims, wherein the intermittently transmitted video data stream has a fixed frame line length and a variable transmission pause duration.

5. The apparatus (302) of any one of the preceding claims, wherein the frame line indicator assigned to each group of valid pixels is included in a data structure comprising the group of valid pixels.

6. A viewing device (312), comprising: A receiver (410) configured to receive an intermittently transmitted serialized video data stream, the serialized video data stream comprising a plurality of data structures (402), each data structure comprising at least one group of valid pixels from a video frame line and a frame line position indicator assigned to the group of valid pixels; as well as One or more processors or processing circuitry (1002) configured to: for each received data structure, generating a complete line of video data by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure including a next pixel group in the serialized video stream is received; as well as Each complete line of video data is caused to be output for presentation on a display at a position in the video frame indicated by the frame line position indicator of the valid pixels assigned to the complete line of video data.

7. A viewing device as claimed in claim 6, wherein the serialized video stream is received from an apparatus (302) as claimed in any one of claims 1 to 5, wherein the received intermittently transmitted serialized video data stream is processed by the one or more processors or processing circuit systems (1002) to add a variable number of blanking pixels for output to the display as a continuous serialized variable line length video stream.

8. The viewing device (312) of claim 6 or 7, wherein the viewing device further comprises the display.

9. A method of processing a serialized video stream for onward transmission, the method comprising: receiving a serialized video stream (314), the serialized video stream comprising a plurality of serially transmitted video frames from a video source, each video frame comprising a plurality of frames, each frame comprising a plurality of frame lines, the frame lines comprising a set of active pixels and blanking pixels; buffering at least the valid pixels (316) of the received serialized video stream until a complete frame row of pixels (318) is received; and The pixels of each buffered full frame row are processed by the following operations: assigning a frame line position indicator to a valid pixel group (320) comprising the valid pixels of the buffered complete frame line; and generating data structures including the active pixel groups and including frame line position indicators assigned to the active pixel groups (322), each data structure including a set of active pixels from a received frame line and omitting at least one blanking pixel of the received frame line; When each data structure is ready for transmission (324), intermittently transmitting forward the outgoing serialized video data (326) including the generated data structure; as well as Onward transmission of the outgoing serialized video data is paused between data structures (328).

10. The method of claim 9, wherein each data structure comprising a set of valid pixels from a received frame line omits all blanking pixels for that received video frame line in the outgoing serialized video data comprising the forwarded data structure.

11. The method of claim 10, wherein the forward intermittently transmitted video data stream has a fixed frame line length and a variable transmission pause duration.

12. A method of processing a received serialized video stream for display, the method comprising: receiving a received intermittent serialized video stream comprising a plurality of data structures, each data structure comprising at least one group of valid pixels from a video frame line and a frame line position indicator assigned to the group of valid pixels; generating a complete line of video data by adding one or more blanking pixels to each valid pixel group in the received data structure until a data structure including a next pixel group in the serialized video stream is received; as well as Each complete line of video data is caused to be output for presentation on a display at a position in the video frame indicated by the frame line position indicator of the valid pixels assigned to the complete line of video data.

13. A method (600) as claimed in claim 12, wherein the intermittent serialized video stream is received from an apparatus (302) as described in any one of claims 1 to 5, wherein the received intermittently transmitted serialized video data is processed by the one or more processors or processing circuit systems (1002) to add a variable number of blanking pixels for output to the display as a continuous serialized variable line length video stream.

14. The method (600) of claim 12 or 13, wherein the method further comprises: determining (605a) that the second to last frame row of the frame has been received; generating (605b) an end of a frame line including blanked pixels as a last frame line of the frame; and The received frame row including the end of the frame row of the frame is output (606) to the display as a serial data stream of pixels.

15. A computer program product comprising computer code (500) which, when loaded from a memory (306) and executed by one or more processors (308) of an apparatus (302) according to any one of claims 1 to 5, causes the apparatus (302) to implement a method according to any one of claims 9 or 10.

Images