Serialized video transfer

Abstract

An apparatus and related aspects are disclosed for combining multiple serialized video streams from different video sources for serialized forward transmission over a channel. For example, an apparatus may include a receiver configured to receive each serialized video stream of separate channels, respectively; a plurality of buffers, each buffer arranged to buffer at least effective pixels of a frame row of the received serialized frame in the received video stream; the one or more processors or processing circuits are configured to: for each effective pixel group in a fully received pixel row in a received frame of a received serialized video stream, receive the serialized video stream; assigning a video source indicator to the effective pixel group and generating a data structure including the effective pixel group, a source indicator assigned to the pixel group, and a frame line position indicator for the effective pixel group, the effective pixel data structure omitting at least one blanking pixel effective pixel from the same video line as the effective pixel group; and a transmitter configured to forward transmit the generated data structure of the two received serialized video streams as serialized video data on a common channel.

Description

Technical Field

[0001] The present disclosure relates to serialized video transmission, to apparatus and methods for intermittently forwarding rows of serialized video frames received as a continuous stream of pixels from at least one video source to a viewing device, and related aspects.

[0002] In particular, but not exclusively, the present disclosure relates to an apparatus configured to discard one or more blanked pixels from each received video frame row, the apparatus pausing transmission after transmitting a group of pixels from a video frame row comprising valid pixels and zero or more remaining blanked pixels until pixels from the next video frame row are ready to be transmitted forward.

[0003] In particular, but not exclusively, some embodiments of the disclosed technology relate to a system in which video from two or more sources is streamed in real time with minimal latency to one or more remote displays, such as a near-eye display of a head-mounted device, for synchronized low-latency output.

[0004] Some aspects and embodiments of the disclosed technology may be particularly useful, but not limited to, use cases where highly accurate and reliable real-time video streams from two or more separate asynchronous sources are reconstructed at a receiver display with minimal latency. Such images may be used in a variety of applications, particularly those that provide some visual indication of perspective and / or depth in the image, such as may be provided by a stereoscopic display.

[0005] For example, some embodiments of the disclosed technology may be used in a medical context, such as a surgical procedure, where a surgeon or similar medical operator is performing or assisting in a surgical procedure while viewing a real-time video clip displayed on a near-eye display or screen proximal to the surgical site. Such a display may be advantageous for obtaining an image of a patient undergoing surgery, with one or more image modes on the image allowing for differentiation of healthy tissue from unhealthy tissue, magnification of body parts and organs, and magnified views of specific areas being viewed and operated on.

[0006] However, this usage context requires very low latency, high and stable synchronization of video feeds from at least two video sources, in addition to the need for any head-mounted type display to be lightweight and comfortable to wear. The synchronization of images from different sources is very important, because any synchronization loss or jitter in the real-time video streams can cause serious problems during surgery. For example, if the left view near-eye video stream is not synchronized with the video stream including the right view near-eye image. Even if video is received from a single video source, it is important to manage the data flow to avoid problems such as buffer overflow when processing the video for onward transmission from the source to the display device. Background Art

[0007] Displays using Liquid Crystal Display (LCD) panels can be designed to meet industry standard video standards such as High Definition (HD) and Ultra High Definition (UHD). These standards require strict compliance, they establish a set number of vertical and horizontal pixels on each vertical and horizontal line of the video frame, specify vertical and horizontal blanking intervals that include "pixels" that are not intended to be displayed, and specify standards for transmission rates.

[0008] In a raster scan display, the vertical blanking interval (VBI) refers to the time from the end of the last visible line of a frame or field of view to the beginning of the next visible line of the next frame. Although modern thin panel displays, such as liquid crystal displays and organic light emitting diode (OLED) displays do not require blanking intervals, for traditional reasons, they are still configured to receive video streams including blanking intervals containing many blanking pixels. The terms "blank pixels" or "blank intervals" are known in the art to include data that is not intended to be displayed. In other words, a "blank interval" or "blank pixel" does not necessarily carry no information.

[0009] Video standards also set strict requirements for transmission and clock synchronization timing, for example, the timing requires that the LCD panel receiver clock be synchronized with the video source clock to ensure that the image is displayed correctly. When the video source (for example, the video signal output by a camera) is transmitted using a common standard such as the Serialized Digital Interface (SDI) standard, the SDI signal is configured to forward the video source clock from the video source to the LCD panel for synchronization.

[0010] Each SDI cable can only forward the clock of a single video source. Therefore, problems arise when using a single SDI cable to transmit video from two or more asynchronous video sources, for example, there may be a special need to provide video clips that provide perspective views or depth.

[0011] The current solution is to provide an intermediate video frame buffer that decouples the input video clock from each video source and synchronizes the SDI transmission to a single clock independent of either source. However, adding an intermediate video frame buffer may require hardware reconfiguration and increase latency by an order of magnitude of one to two frame delays in the video output at the LCD panel display. This level of latency is undesirable when the video source is a real-time stream for interactive control (e.g., surgery). When displaying video from a single source, similar latency issues may occur when the video from a single source is routed via an intermediate hub for onward transmission due to possible buffer overflow issues at the hub.

[0012] The disclosed techniques are intended to mitigate, obviate, alleviate or eliminate various problems known in the art that may affect the transmission of one or more video streams to an LCD or LCD-type display, such as those mentioned above. Summary of the invention

[0013] Although the present invention is defined by the appended claims, various aspects of the disclosed technology are listed in this summary, including the claimed technology, and some examples of preferred embodiments and indications of possible technical effects are provided.

[0014] A first aspect of the disclosed technology includes an apparatus for processing a serialized video data stream for forward transmission, the serialized video data stream comprising a plurality of serially transmitted video frames from a video source, each video frame comprising a plurality of frame rows, at least one frame row comprising a valid pixel group and one or more blanking pixels, the apparatus comprising: a receiver configured to receive the serialized video data stream; a memory arranged to buffer the received serialized video data until at least all valid pixels in a frame row of pixels of the received video frame have been buffered; one or more processors or processing circuits configured to, for each valid pixel group of a frame row of pixels, generate a data structure comprising at least the valid pixel group of the frame row and a frame row position indicator assigned to the valid pixel group, each data structure comprising a valid pixel group from the received frame row and omitting at least one blanking pixel of the received frame row; and a transmitter configured to intermittently forward transmit the generated data structures by pausing transmission after each data structure has been transmitted until the next data structure is ready to begin transmission.

[0015] Advantageously, in some embodiments, latency delays can be reduced because, instead of buffering the received data stream at the device until an entire video frame has been received, the receiver buffer is configured to clear after each complete video frame row has been received, without waiting for the entire video frame to be received.

[0016] In some embodiments, more than one blanked pixel is discarded from each received video frame line that includes a group of active pixels and at least one blanked pixel.

[0017] In some embodiments, all blanked pixels of a received video frame row are excluded from the outgoing serialized video stream comprising the data structure transmitted forward.

[0018] Advantageously, by removing one or more or all blanking pixels from each video frame line, the transmission time of each video frame line to a viewing device is reduced as the number of blanking pixels subject to onward transmission is reduced.

[0019] In some embodiments, the intermittently transmitted video data stream has a fixed frame line length, and a variable transmission pause duration.

[0020] Advantageously, by pausing transmissions between groups of pixels at the device, any transmitter clock used to sequence video frames at the video source can be decoupled from any receiver clock used at the viewing device.

[0021] Advantageously, at the viewing device, variable transmission pause lengths may be compensated for by adding a variable number of blanking pixels for output to the viewing device as a continuous stream of variable line length video data.

[0022] In some embodiments, the variable number of blanked pixels depends on the transmission duration of each forwarded data structure to the viewing device.

[0023] In some embodiments, the frame line indicator assigned to each active pixel group is included in a header of a data structure that includes the active pixel group.

[0024] Another second aspect of the disclosed technology includes a viewing device, comprising: a receiver configured to receive an intermittently transmitted serialized video stream comprising a plurality of data structures, each data structure comprising at least a valid pixel group from a video frame row and a frame row position indicator assigned to the valid pixel group; and one or more processors or processing circuits configured to, for each received data structure, generate a complete row of video data by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure comprising the next pixel group in the serialized video stream is received, and cause each complete row of video data to be output for presentation on a display at a position in the video frame indicated by the frame row position indicator of the valid pixels assigned to the complete row of video data.

[0025] In some embodiments, a serialized video stream is received from an apparatus for processing a serialized video data stream for forward transmission according to the first aspect disclosed herein or any one of its embodiments, wherein the received intermittently transmitted serialized video data stream is processed by one or more processors or processing circuits to add a variable number of blanking pixels for output to a display as a continuous serialized variable line length video stream.

[0026] In some embodiments, the viewing device further comprises a display.

[0027] In some embodiments, the viewing device may include more than one display and split the received video stream into two or more streams. One or more of the streams may be external to the viewing device. For example, in some embodiments, a large flat panel display may be used to display the same image as is being displayed on the near eye display if the large screen can be driven in the same manner as the near eye display.

[0028] In some embodiments, the viewing device comprises means for processing a serialized video stream for onward transmission according to the first aspect disclosed herein or any one of its embodiments.

[0029] Advantageously, reducing the number of transmitted blanked pixels can reduce the likelihood of buffer overflow at the receiving viewing device.

[0030] Another third aspect of the disclosed technology includes a method for processing a serialized video stream for forward transmission, the method comprising: receiving a serialized video stream, the serialized video stream comprising a plurality of serially transmitted video frames from a video source, each video frame comprising a plurality of frames, each frame comprising a plurality of frame rows comprising valid pixel groups and blanking pixels; buffering at least valid pixels of the received serialized video stream until a complete pixel frame row is received; and processing each buffered complete pixel frame row by: assigning a frame row position indicator to a valid pixel group comprising valid pixels of the buffered complete frame row, and generating a data structure comprising the valid pixel group, the valid pixel group having a frame row position indicator assigned to the valid pixel group, each data structure comprising a valid pixel group from the received frame row and omitting at least one blanking pixel of the received frame row, and when each data structure is ready for transmission, intermittently transmitting forward outgoing serialized video data comprising the generated data structure, and pausing the forward transmission of the outgoing serialized video data between the data structures.

[0031] For example, in some embodiments, a pause in the forward transmission of outgoing serialized video data includes following transmission of a data structure including valid pixel groups and assigned frame row position indicators until a next data structure including a different valid pixel group with a different frame row position indicator is ready for transmission.

[0032] In some embodiments, each data structure comprising a set of valid pixels from a received frame row omits all blanked pixels for that received video frame row in the outgoing serialized video stream comprising the data structure transmitted forward.

[0033] In some embodiments, the video data stream that is intermittently transmitted forward has a fixed frame line length and a variable transmission pause duration.

[0034] Another fourth aspect of the disclosed technology includes a method for processing a received intermittent serialized video stream for display, the method comprising: receiving a received serialized video stream comprising a plurality of data structures, each data structure comprising at least a valid pixel group from a video frame row, and a frame row position indicator assigned to the valid pixel group; generating a complete row of video data by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure comprising the next pixel group in the serialized video stream is received, and causing each complete row of video data to be output for presentation on a display at a position in the video frame indicated by the frame row position indicator of the valid pixels assigned to the complete row of video data.

[0035] In some embodiments, an intermittent serialized video stream is received from an apparatus for processing a serialized video stream for onward transmission according to the first aspect disclosed herein or any embodiment thereof.

[0036] In some embodiments, the received intermittently transmitted serialized video data is processed by one or more processors or processing circuits (1002) to add a variable number of blanking pixels for output to a display as a continuous serialized variable line length video stream.

[0037] In some embodiments, the method further includes determining a penultimate frame row of the received frame, generating a frame row including blanked pixels as a last frame row of the frame, and outputting the frame row of the frame as a serialized data stream of pixels to a display.

[0038] Another fifth aspect of the disclosed technology includes a computer program product, which includes computer code, which, when loaded from a memory and executed by one or more processors of the aspects or any one embodiment disclosed in this article of an apparatus for processing a serialized video stream for forward transmission, enables the apparatus to implement any disclosed aspect or embodiment of the method for processing a serialized video stream for forward transmission.

[0039] Another sixth aspect of the disclosed technology includes a computer program product, which includes computer code, which, when the computer code is loaded from a memory and executed by one or more processors of the disclosed aspects or any embodiment of the viewing device, enables the viewing device to implement any disclosed aspect or embodiment of the method for processing a received video stream.

[0040] Another seventh aspect of the disclosed technology includes a signal generated by any disclosed aspect or embodiment of an apparatus for processing a serialized video stream for forward transmission, wherein the signal includes a serialized video stream of data structures, each data structure including at least a valid pixel group from a video frame row and a frame row position indicator assigned to the valid pixel group.

[0041] In some embodiments, the outgoing serialized video stream including the forwarded data structure does not contain any blanked pixels.

[0042] In some embodiments, an outgoing serialized video stream comprising forward transmitted data structures comprises in each data structure at least one fewer blanked pixels than the number of blanked pixels of a received video frame row comprising a valid pixel group of the data structure.

[0043] Another eighth aspect of the disclosed technology includes a video display system, wherein the video display system includes an aspect or any embodiment disclosed herein of an apparatus for processing a serialized video stream for onward transmission, and an aspect or any embodiment disclosed herein of a viewing apparatus.

[0044] In some embodiments, the apparatus for processing the serialized video stream for onward transmission further comprises a video source.

[0045] In some embodiments, the viewing device also includes at least one display configured to present video received from the video source via the device.

[0046] Another ninth aspect of the disclosed technology includes a computer-readable storage medium including a computer program code, which, when executed by one or more processors or processing circuits of an aspect or embodiment disclosed herein of an apparatus for processing a serialized video stream for forward transmission, causes the apparatus to implement an aspect or embodiment disclosed herein of a method for processing a serialized video stream for forward transmission.

[0047] Another tenth aspect of the disclosed technology includes a computer program carrier carrying a computer program, the computer program including computer program code, which, when loaded from the computer program carrier and executed by one or more processors or processing circuits of the aspects or embodiments disclosed herein of the apparatus for processing a serialized video stream for forward transmission, enables the apparatus to implement the aspects or embodiments disclosed herein of the method for processing a serialized video stream for forward transmission, wherein the computer program carrier is one of an electronic signal, an optical signal, a radio signal or a computer-readable storage medium.

[0048] Another aspect of the disclosed technology includes means for performing a method according to any of the above disclosed method aspects.

[0049] Another eleventh aspect of the disclosed technology includes an apparatus for combining multiple serialized video streams from different video sources for serialized forward transmission on a channel, the apparatus comprising: a receiver configured to receive each serialized video stream of a separate channel respectively; a plurality of buffers, each buffer being arranged to buffer at least valid pixels of a frame row of a received serialized frame in the received video stream; one or more processors or processing circuits configured to, for each valid pixel group in a fully received pixel row in a received frame of the received serialized video stream, assign a video source indicator to the valid pixel group and generate a data structure including a valid pixel group, a source indicator assigned to the pixel group, and a frame row position indicator for the valid pixel group, each data structure omitting at least one blanking pixel in the fully received frame row; and a transmitter configured to forward the generated data structure of two received serialized video streams as serialized video data on a common channel.

[0050] In some embodiments, the video device comprises a hub co-located with the surgical imaging equipment.

[0051] In some embodiments, the hub comprises a hub co-located with the surgical imaging device, the hub being configured to receive serialized video streams from two or more video sources, for example, left view and right view video sources.

[0052] In some embodiments, each video source comprises a camera, and each camera is part of a surgical imaging device.

[0053] In some embodiments, the hub is a surgical imaging device, or a portion thereof, including a surgical microscope.

[0054] In some embodiments, the hub is co-located with the surgical microscope.

[0055] In some embodiments, the hub is part of the control system or provided as another computing component or unit of the surgical microscope.

[0056] In some embodiments, the method further comprises assigning a frame row position indicator to each valid pixel group prior to onward transmission thereof, however, in alternative embodiments, the method may instead comprise retaining the frame row position indicators associated with the valid pixel groups.

[0057] In some embodiments, the transmitter pauses transmission after having transmitted a data structure from one video source until the next data structure from another video source or from the same video source is ready to begin transmission.

[0058] In some embodiments, the received video data stream has a fixed line length and the transmission pause lengths between forward transmitted data structures vary.

[0059] In some embodiments, the transmitter transmits onward the generated data structures from the multiple sources as a continuous combined video data stream.

[0060] Advantageously, in some embodiments, latency delays can be reduced because, instead of buffering the received data stream at the device until an entire video frame has been received from a video source, the receiver buffer is configured to clear after each complete video frame row has been received without waiting for the entire video frame to be received.

[0061] Advantageously, by pausing transmissions between groups of pixels at the device, any transmitter clock at the video source used to serialize pixels forming video frame data can be decoupled from any receiver clock at the viewing device used to extract pixel data.

[0062] Advantageously, by decoupling the clocks at the video source and viewing device, the transmission latency of multiple serialized video streams can be reduced. Additionally, the receiver circuitry can be simpler because a phase-locked loop does not need to be used to derive the same frequency as the source. This allows multiple serialized video streams to have better synchronization when displayed concurrently.

[0063] In some embodiments, the video source indicator identifies a spatial relationship between at least two of the plurality of video sources.

[0064] In some embodiments, the multiple serialized video streams include a video stream for a left view of the scene obtained from a video source and a video stream for a right view of the scene obtained from the video source, wherein a video source indicator identifies whether a pixel group belongs to a video frame for the left view of the scene obtained from the video source or a video frame for the right view of the scene.

[0065] In some embodiments, the allocated video source indicator and frame line position indicator are included in a header of the data structure, where the bit size of the header matches the parallelism width of a serializer configured to combine data structures from multiple different video sources into one outgoing serialized video stream.

[0066] In some embodiments, the serializer interleaves the data structures from the two sources in the outgoing serialized video stream.

[0067] In some embodiments, the transmitter is configured to, after transmitting a data structure including a valid pixel group with a first video source indicator, suspend forward transmission between data structures until a data structure with the same or a different video source indicator is ready for transmission.

[0068] In some embodiments, interleaving alternates data structures from one video source with data structures from another video source in an outgoing serialized video stream, however, this may not always occur and one or more frame rows may be sent continuously from one source if there are no other frame rows available for transmission.

[0069] In some embodiments, at least one received serialized video stream is received on a channel having one or more of a different data rate and a different video resolution relative to a data rate or video resolution of at least one other channel, and another received video stream is received via at least one other channel.

[0070] In some embodiments, all blanked pixels of a received video frame row are excluded from the outgoing serialized video stream comprising the data structure transmitted forward.

[0071] However, in some embodiments, only one or more blanked pixels are removed and any remaining blanked pixels for the received video frame line are included in the forwarded data structure of the outgoing serialized video stream.

[0072] Advantageously, by removing one or more or all blanking pixels from each video frame line, the transmission time of each video frame line to the viewing device is reduced as the number of blanking pixels subject to forward transmission is reduced. In addition, since the entire video frame does not need to be buffered, but only the entire line of the video frame is buffered at the transmitter before it is output, the possibility of buffer overflow at the receiver of the viewing device can be reduced. This, coupled with the absence or reduction of the number of blanking pixels in the received transmission stream and / or if there is only intermittent transmission of the data structure, reduces the possibility of buffer overflow at the combiner device and at the viewing device.

[0073] In some embodiments, the video combiner device may decouple the source clock of the received serialized video stream in the outgoing serialized video stream by changing the pause duration between the forward transmitted data structures or by changing the transmission duration of the forward transmitted data structures.

[0074] In some embodiments, the transmitter is configured to transmit the outgoing serialized video stream over a data communications channel along a wired link and / or over a wireless data communications channel to a device configured to cause display of the received video data.

[0075] In some embodiments, the weight of the viewing device, for example, if it is a head mounted display, can be reduced by using only one wired link to the video combiner device to carry serialized video data from multiple serialized video sources. This can make the head mounted device less cumbersome for the wearer.

[0076] Another twelfth aspect of the disclosed technology includes a viewing device, the viewing device comprising: a receiver configured to receive a serialized video stream including a plurality of data structures, each data structure including an effective pixel group, a source indicator assigned to the pixel group, and a frame row position indicator assigned to the effective pixel group; and one or more processors or processing circuits configured to generate a complete row of video data by adding one or more blanking pixels to each effective pixel group of the received data structure until a data structure including the next pixel group in the serialized video stream is received, and to output each complete row of video data so as to be presented on a display associated with the source indicator of the complete row of video at a position in the video frame indicated by the frame row position indicator of the effective pixel assigned to the complete row of video data.

[0077] In some embodiments, multiple full frame lines of video data output to one or more displays include variable line length video data. In other words, the video data output to the display does not have a fixed frame line length of active pixels and blanked pixels because the number of blanked pixels can vary.

[0078] Advantageously, variable frame line length video data allows the data to be output to one or more displays as a continuous stream of pixels and blanked pixels.

[0079] In some embodiments, one or more displays include a flat panel display, such as a display based on LCD or OLED technology configured to receive serialized video data having a raster-type display format.

[0080] In some embodiments, the received serialized video stream is received from an apparatus for combining a plurality of serialized video streams from different video sources for serialized onward transmission on a channel according to the eleventh aspect disclosed herein or any embodiment thereof.

[0081] In some embodiments, the video combiner device further comprises a video source.

[0082] In some embodiments, the viewing device also includes at least one display configured to present video received from the video source via the device.

[0083] Another thirteenth aspect of the disclosed technology includes a method for combining multiple serialized video streams from different video sources for serialized forward transmission on a channel, each serialized video stream being received on a different communication channel, the method comprising: receiving each of the multiple serialized video streams respectively, buffering at least valid pixels of a frame row of a received serialized frame of each of the multiple received video streams respectively, processing each buffered valid pixel group in a fully received frame row at least by: assigning a video source indicator to the valid pixel group and generating a data structure including the valid pixel group, an indicator of the video source for the pixel group and a frame row position indicator for the valid pixel group, and forwarding on a common channel the generated data structure including the buffered valid pixels of a frame row of a video frame from one of the multiple received serialized video streams.

[0084] In some embodiments, the received intermittently transmitted serialized video data is processed by one or more processors or processing circuits of the viewing device to add a variable number of blanking pixels for output to a display as a continuous serialized variable line length video stream.

[0085] In some embodiments, the method further includes: after each data structure has been transmitted, determining whether another data structure of video frame rows of video frames from another video source of the plurality of video sources is ready to begin forward transmission, and if so, forward transmitting the data structure from the another video source of the plurality of video sources on the common channel, otherwise determining whether another data structure from one of the plurality of video sources is ready to be transmitted, and if so, forward transmitting the data structure from the one of the plurality of video sources on the common channel.

[0086] In some embodiments, after the data structure has been transmitted, the method further includes: determining that no data structure from any of the multiple video sources is ready for transmission, pausing transmission until the data structure from the multiple video sources is ready for transmission, and forwarding the data structure on the common channel.

[0087] In some embodiments, the method includes intermittently forwarding the generated data structures from each received serialized video stream as a single serialized video stream, and after transmitting a data structure including valid pixel groups and assigned frame row position indicators, pausing the forward transmission between the data structures until a next data structure including a different valid pixel group with a different frame row position indicator is ready for transmission.

[0088] In some embodiments, the combined intermittently transmitted video data stream has a fixed frame line length, and a variable transmission pause duration.

[0089] Another fourteenth aspect of the disclosed technology includes a method for viewing multiple serialized video streams received as a single serialized video stream, the method comprising: receiving a serialized video stream comprising multiple data structures, each data structure comprising a valid pixel group, a source indicator assigned to the pixel group, and a frame row position indicator assigned to the valid pixel group; processing the received serialized video stream in the following manner: generating a complete row of video data by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure comprising the next pixel group in the serialized video stream is received, and causing each complete row of video data to be output so as to be presented on a display associated with the source indicator of the complete row of video at a position in the video frame indicated by the frame row position indicator of the valid pixel assigned to the complete row of video data.

[0090] In some embodiments, the method further comprises displaying each complete line of the video data on the display.

[0091] In some embodiments, the received intermittently transmitted serialized video stream data may be video stream data intermittently transmitted by an aspect or an embodiment of the combiner apparatus disclosed herein.

[0092] In some embodiments of the method, the received serialized video stream (622) is a video stream output by a combiner device according to the above-described combiner device aspects or any one of its embodiments disclosed herein.

[0093] Another fifteenth aspect of the disclosed technology relates to a video display system, wherein the video display system comprises a video combiner device according to the eleventh aspect disclosed herein or any embodiment thereof and a viewing device according to the twelfth aspect disclosed herein or any embodiment thereof.

[0094] In some embodiments, the video combiner device further comprises a video source.

[0095] In some embodiments, the viewing device also includes at least one display configured to present video received from the video source via the device.

[0096] The sixteenth aspect of the disclosed technology relates to a computer program product comprising computer code, which, when loaded from a memory and executed by one or more processors of a device according to the eleventh aspect disclosed herein or any embodiment thereof, enables the device to implement a method according to the thirteenth aspect.

[0097] The seventeenth aspect of the disclosed technology relates to a computer program product comprising computer code, which, when loaded from a memory and executed by one or more processors of a viewing device according to the twelfth aspect disclosed herein or any embodiment thereof, enables the viewing device to implement a method according to the fourteenth aspect disclosed herein or any embodiment thereof.

[0098] Another aspect of the disclosed technology includes a signal generated by an apparatus for combining video streams according to the eleventh aspect disclosed herein or any embodiment thereof, wherein the signal includes a serialized video stream of data structures, each data structure including at least a valid pixel group from a video frame row, and a frame row position indicator assigned to the valid pixel group.

[0099] In some embodiments, the outgoing serialized video stream including the forwarded data structure does not contain any blanked pixels.

[0100] In some embodiments, an outgoing serialized video stream comprising forward transmitted data structures includes in each data structure at least one fewer number of blanked pixels than the number of blanked pixels of a received video frame row comprising a valid pixel group of the data structure.

[0101] Another aspect of the disclosed technology includes a method for viewing two or more serialized video streams received as a single serialized video stream, the method comprising: receiving a first serialized video stream comprising a plurality of data structures, each data structure comprising a valid pixel group, a source indicator assigned to the pixel group, and a frame row position indicator assigned to the valid pixel group; and processing the received serialized video stream by: generating a complete row of video data by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure comprising the next pixel group in the serialized video stream is received, and causing each complete row of video data to be output for presentation on at least one display at a position in a video frame indicated by the frame row position indicator of the valid pixels assigned to the complete row of video data, each of the at least one display being associated with a source indicator for the complete row of video data.

[0102] In some embodiments, the method further comprises displaying each complete line of the video data on the display.

[0103] In some embodiments of the method, the received serialized video stream is a discontinuous video stream output by a combiner device according to the above-mentioned combiner device aspect or any one of its embodiments disclosed herein.

[0104] Another aspect of the disclosed technology includes means for performing the method according to any of the disclosed method aspects.

[0105] Another aspect of the disclosed technology includes a computer program product, which includes computer code. When the computer code is loaded from a memory and executed by one or more processors of a video combiner device according to the eleventh aspect disclosed herein or any embodiment thereof, the video combiner device implements a method according to the thirteenth aspect disclosed herein or any embodiment thereof.

[0106] Another aspect of the disclosed technology includes a computer program product comprising computer code, which, when the computer code is loaded from a memory and executed by one or more processors of a viewing device according to the twelfth aspect disclosed herein or any one of its embodiments, enables the viewing device to implement any disclosed aspect or embodiment of the method for processing a received video stream.

[0107] Another aspect of the disclosed technology includes a signal generated by an apparatus according to the eleventh aspect disclosed herein or any embodiment thereof, wherein the signal includes a serialized video stream of data structures, each data structure including at least a valid pixel group from a video frame row and a frame row position indicator assigned to the valid pixel group.

[0108] In some embodiments, the outgoing serialized video stream including the forwarded data structure does not contain any blanked pixels.

[0109] In some embodiments, an outgoing serialized video stream comprising forward transmitted data structures includes in each data structure at least one fewer number of blanked pixels than the number of blanked pixels of a received video frame row comprising a valid pixel group of the data structure.

[0110] Another aspect of the disclosed technology includes a video display system, wherein the video display system includes a device according to the eleventh embodiment disclosed herein or any one of the embodiments and a viewing device according to the twelfth aspect disclosed herein or any one of the embodiments.

[0111] In some embodiments, the apparatus for processing the serialized video stream for onward transmission further comprises a video source.

[0112] In some embodiments, the viewing device also includes at least one display configured to present video received from the video source via the device.

[0113] In some embodiments, the video display system further comprises a plurality of video sources, each video source being arranged to provide serialized video data to the apparatus according to the eleventh embodiment.

[0114] Another aspect of the disclosed technology includes a computer-readable storage medium including a computer program code, which, when executed by one or more processors or processing circuits of the aspects or embodiments disclosed herein of the video combiner, causes the video combiner device to implement a method according to the thirteenth aspect or embodiment of the method disclosed herein.

[0115] Another tenth aspect of the disclosed technology includes a computer program carrier carrying a computer program including computer program code, which, when the computer program code is loaded from the computer program carrier and executed by one or more processors or processing circuits of the aspects or embodiments disclosed herein of the video combiner device, causes the video combiner device to implement the method according to the thirteenth aspect or embodiment disclosed herein, wherein the computer program carrier is one of an electronic signal, an optical signal, a radio signal or a computer-readable storage medium.

[0116] It will be apparent to a person skilled in the art that the aspects and embodiments of the present disclosure may be combined with each other in any suitable manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Some embodiments of the disclosed technology are described below with reference to the accompanying drawings, which are provided as examples only, in which:

[0118] Figure 1A Schematically illustrates how latency is generated when there are two video sources, each of which outputs a serialized video stream to an intermediate device according to the prior art, the intermediate device being configured to buffer the received video stream frame by frame before forwarding the combined video stream to two remote displays;

[0119] Figures 1B to 1C Schematically illustrates how latency is generated when there are two video sources, each of which outputs a sequenced video stream to an intermediate device according to the disclosed technique, the intermediate device being configured to buffer the received video stream line by line before forwarding the combined video stream to two remote displays;

[0120] Figure 2A An example of serialized video frame transmission is schematically shown;

[0121] Figure 2B An example of how to display serially transmitted video frames using an LCD display panel is schematically shown;

[0122] Figure 3A Schematically illustrates an example of an apparatus according to some embodiments of the disclosed technology;

[0123] Figure 3B Schematically illustrates an example of a method according to some embodiments of the disclosed technology;

[0124] Figure 4 An example of serially transmitting a variable video frame length from a single video source according to some embodiments of the disclosed technology is schematically shown;

[0125] Figure 5 Schematically shows the Figure 3A How does the device generate a data structure header according to some embodiments of the disclosed technology;

[0126] Figure 6 Schematically illustrates an example of a viewing device according to some embodiments of the disclosed technology;

[0127] Figure 7 Schematically illustrating a method performed by a viewing device according to some embodiments of the disclosed technology;

[0128] Figure 8 Schematically illustrates a system for displaying videos from two video sources according to some embodiments of the disclosed technology;

[0129] Fig. 9 Schematically illustrates an example of serially transmitting variable video frame lengths from multiple video sources to multiple displays according to some embodiments of the disclosed technology;

[0130] Fig.10 Schematically illustrates an example of another method according to some embodiments of the disclosed technology;

[0131] Fig.11 schematically illustrates an example of how a device generates a data structure header according to some embodiments of the disclosed technology; and

[0132] Fig.12 Another example of a viewing device for processing received video data according to some embodiments of the disclosed technology is schematically shown. DETAILED DESCRIPTION

[0133] Various aspects of the present disclosure will be described more fully below with reference to the accompanying drawings. However, the apparatus and methods disclosed herein can be implemented in many different forms and should not be construed as limited to the aspects listed herein. Steps, whether explicitly referenced or implicitly referenced, can be reordered or omitted if not necessary for some disclosed embodiments. Like numbers in the figures refer to like elements throughout the figures, however, the same or different embodiments may have the same or different numbers, depending on which page they are first shown in the figure.

[0134] The terms used herein are only used to describe certain aspects of the present disclosure and are not used to limit the disclosed technical embodiments disclosed herein.Herein, the singular forms "a", "an" and "the" also include plural forms, unless the context clearly indicates otherwise.

[0135] Figure 1A Schematic diagram showing how latency is created when there are two video sources, each of which outputs a serialized video stream to an intermediate device, where the intermediate device is configured to buffer the received video streams frame by frame before forwarding the combined video stream to two remote displays in accordance with the Serialized Data Interface (SDI) video standard transmission requirements. Figure 1A In the present invention, two asynchronous video sources #1 and #2 of serialized video data streams respectively output data to a combiner device through separate 3G serialized video links, and the combiner device is configured to merge the two streams into one outgoing serialized video stream for forward transmission to a viewing device including two displays through a 6Gbit / s link according to the SDI video standard.

[0136] exist Figure 1A In FIG. 1 , two video sources #1, #2, labeled 102a, 102b, are both configured to generate a video of a scene, in this example, they are both configured to capture a view of a swinging pendulum. Figure 1A As shown, video source #1, 102a outputs an image of a pendulum, which is transmitted via a serialized data interface (SDI) link 104 toward a display 106a via a video combiner device or hub 108, which combines the video from video source #1 with the video of another pendulum from video source #2.

[0137] like Figure 1A As shown, the video frames received from video source #1 are shown as being stored in a buffer at hub 108 ( Figure 1A However, because video source #2 102b is either running at a slightly lower clock speed for transmitting video, or because link 104b is slower, the frame of the other pendulum it transmits arrives slightly later at hub 108. Since the hub is configured to wait until both frames are fully received from both sources before any forward transmission of any frame, Figure 1A The result shown is that one half frame from video source #1 is buffered into the top rows of buffers Buf1 and Buf2 at hub 108 before the lower buffer Buf1 holds the entire image of the same pendulum from video source #2.

[0138] Since the combiner box cannot lock to the clock of any source, both image frames must be completely received at the hub 108 before progressive forward transmission to the respective displays can occur, so the buffers at the hub 108 will accumulate at different rates to compensate for the difference in asynchronous transmission between the two video sources before the video can be forwarded. This increases the latency at the hub 108.

[0139] One way to reduce latency is to configure the hub 108 to only buffer incoming video line by line. Figures 1B to 1C The schematic diagram shows how this approach, if used to try to reduce the latency incurred at the hub 108, in other words, how a buffer underflow problem may result at the hub 108 if the hub 108 outputs video from two asynchronous video sources line by line.

[0140] Figures 1B to 1D Schematically illustrates a series of examples of what occurs at the hub 108 if the hub buffer is configured to not wait at the hub device to receive a complete frame from both sources before forwarding the frame line by line to the displays 106a, 106b. Figures 1B to 1D , the hub 108 instead forwards the received video from the two sources #1, #2 to the displays 106a, 106b line by line.

[0141] First, if Figure 1B As shown, the received video streams from the two asynchronous video sources can be kept sufficiently synchronized so that when the top buffer Buf1 of hub 108 has received a video frame from source #1A, the first complete row of the frame from source #B is already in the lower buffer Buf1 of the hub. Figure 1C 1 shows how progressive transmission reduces the delay in the display 106a, 106b receiving the first line of the corresponding video frame as it is received at the hub 108. However, since the two sources are asynchronous, there will eventually be a situation where only one line can be sent from one video source, here source #1, because there is a buffer underflow at the hub 108 for the video from the other video source (source #2), as shown in FIG. Figure 1D In other words, Figure 1D The lower and middle buffers Buf1 and Buf2 are both empty.

[0142] Therefore, to avoid the risk of buffer underflow, it is advisable to buffer the complete frame at the hub 108 before sending any frame lines to the remote display.

[0143] Some embodiments of the disclosed technology attempt to provide another solution that can reduce latency and reduce the risk of buffer underflow even if the hub 108 is configured to forward the received video data line by line. In some embodiments, similar latency reduction can be achieved even if only one video stream is received from a single video source.

[0144] At the video source, each line of a video frame includes: a number of active pixels, where the active pixels represent the pixels to be reproduced on the remote display; and a number of blanking pixels, which are added to generate a continuous data stream. The blanking pixels form a blanking interval and are a historical convention to allow a raster scan or cathode ray tube (CRT) type display to have time to recover to the beginning of the physical display so that the next line of the image can begin after the previous line ends, and at the end of the last line of active pixels in the frame, a line of blanking pixels is added to allow the CRT light beam to physically move to the beginning of the top line of the next frame. This convention still applies to other types of displays, including flat panel displays, including organic light emitting diodes (OLED) and liquid crystal displays (LCD) or displays based on optically coupled device technology. References to displays herein refer to displays configured to receive data from a video source including blanking pixels. A viewing device may include one or more displays and may include one or more processors for processing the received serialized video data stream for output to a display.

[0145] The term "blank" as used herein only means "not intended to be displayed as part of a video image frame line", and does not necessarily mean that the "blank interval" pixels do not contain any data or information at all. In the prior art, each pixel of the received serialized video data stream is transmitted along the data link 104 at the clock frequency of the video image source 102 according to standard technology, whether it is a valid pixel intended to be displayed or a blanking pixel not intended to be displayed.

[0146] When there is only one video source, buffer underflow or overflow at the hub 108 or any viewing device having one or more displays 106 is typically eliminated by design. However, at high data transfer speeds, especially when there are two sources, as described above, the likelihood of buffer underflow or overflow may increase if the two sources are asynchronous. If the latency delay caused by frame-by-frame buffering at the hub 108 is unacceptable, it would be advantageous if the hub 108 could buffer and forward the video line by line for display.

[0147] For example, a serialized video data transmission rate of a frame buffer on the order of 10 milliseconds for transmitting video via the hub 108 may experience a latency on the order of 16 milliseconds from the time the image is transmitted from the source 102 to the time the transmitted image is displayed on the remote display 108. This latency is too high for use cases such as, for example, when a medical procedure is being performed, where a camera is used to capture images of a body area where the medical procedure is being performed, and the surgeon or one or more assistants performing or assisting in the medical procedure are wearing a near-eye display or head mounted device. Therefore, it is desirable to improve how a hub device or similar device can process the received serialized video stream data to reduce any additional latency or delay that occurs, thereby further reducing and preferably minimizing the time from capturing an image to displaying the image on the display 106.

[0148] The attached drawings Figure 2A The schematic shows in more detail how fixed line length video frames are transmitted over a frame-by-frame transmission standard link 104 (e.g., a serialized data interface (SDI) transmission link 104, such as a wired link from a video source 12, such as a camera, to a remote display 106). Figure 2A The video frame 100 is schematically shown with active pixels as boxes with numbers and blank or blank pixels as blank boxes. Figure 2A It is also schematically shown that each frame includes a plurality of lines (in Figure 2A In addition to the 4 lines 202a-202d shown by way of example in FIG. 2 , a blank or blanked video line 202e may also be provided.

[0149] Each video frame 204 includes a fixed number of horizontal pixel rows y and a fixed number of vertical pixel columns x, which are usually set according to the specifications of a specific video standard. Figure 2A As shown, y = 5, x = 6. This means that Figure 2A In the example shown, each source video frame 204 includes 30 pixels, consisting of 5 rows, each row has 6 pixels, of which 4 are valid pixels and 2 are blanking pixels, and each vertical line includes 5 pixels, of which 4 are valid pixels.

[0150] like Figure 2A As shown, each video frame in the serialized video stream on link 104 from video source 102 has a vertical blanking interval of 1 line and a horizontal blanking interval of 2 pixels. Figure 2AThe lower portion of the diagram shows how each pixel group, including active and blanked pixels, forms a horizontal line of a video frame 100, and shows how the video frame 100 is serialized as a continuous data stream, horizontal frame line by horizontal frame line, over the SDI transmission link 104 to the remote display 106. The time to serialize each video frame including 30 pixels is determined by the clock period to transmit each pixel multiplied by the number of pixels in each video frame, so in this example it is 30 times the clock period.

[0151] Figure 2A and Figure 2B In more detail, it is shown how the top row 202a is transmitted, with the top row first transmitting valid pixels and then any blanking pixels, followed by the next row 202b, the third row 202c, and the last row 202d, with the last row 202e including only the last transmitted blanking pixels. After the transmission of the blanking pixel row 202e, the transmission of the next image frame begins with the first horizontal row of the frame, followed by the second, third, fourth, and finally the blanking rows of the next video frame, and the process iterates while the video source 202 generates video data.

[0152] As mentioned above, each pixel takes a certain amount of time to transmit, which is determined by the clock cycle used by the video source 102 to transmit the video. If the video source clock frequency is 1 pixel per clock cycle, as an example only, the clock cycle is 1 μs, then each pixel will take 1 μs to prepare for transmission. Each horizontal line 202a-202e includes 6 pixels, so each line takes 6 μs to prepare for transmission. Since the video frame 204 has 5 lines 202a-202e, the total time to serialize the entire video frame 204 for transmission along the SDI link 104 is 30 μs.

[0153] When a video stream is transmitted from a video source 102 to a display 106, such as an LCD panel display, via an SDI transmission link 104, the LCD panel at the remote display 106 will need to synchronize the start of each received line 202a-202d in order to reconstruct the frame 204 so that it is properly displayed. Metadata is provided for each group of pixels of a video frame line to provide information that enables the display 106 to properly render the image carried by the serialized video frame data.

[0154] Figure 2B This diagram shows how to Figure 2A The serially transmitted video frames shown are processed for display in greater detail at display 106. Figure 2BIn the example shown, a video source 102 serializing a video data stream provides a sequence of lines of fixed length. The received video stream is processed at a receiver 202 of a display 106, which forwards each data line as it is received for display on a screen 204, such as an LCD panel 204. The clock cycle of the video source 102 is used to synchronize the columns of pixels as they are displayed. This requires that the frequency of the clock at the display 106 be locked to the frequency used by the video source clock to send pixel data over the SDI link 104. For example, if the source clock period is 1 μs, then the display video clock frequency must have a clock period based on the source clock period. By locking the display clock frequency to the source clock frequency, as Figure 2B As shown, the first column of pixels in each frame, such as frame 204, is vertically aligned when reconstructed on LCD panel 204. If the timing of the two clocks is incorrect, buffer overflow or underflow may occur, depending on the degree to which the two clocks are out of sync.

[0155] Therefore, in order to Figure 2A and 2B In order for the captured image to be correctly displayed on the display 106 of the example display system shown, the source clock and the display clock cannot be decoupled. This imposes various restrictions on the design of a display system including at least one video source and at least one video display. First, only one source clock frequency can be physically forwarded on each SDI link 14 using the transmitted serialized video data stream, limiting each video data stream to provide video 102 captured by a single video source.

[0156] One example of how this can cause a problem is if the hub 108 is connected to a head mounted viewing device that includes one or more displays 106 to reduce the bulkiness of the viewing device, when it is desired to reduce the number of cables. Secondly, any processing performed on the video stream received at the intermediate device or hub to add video overlays or similar effects, such image enhancement, for example, causes the two resulting video streams to not be transmitted from the hub 108 to the displays 106 using the same video link. In addition, transmission delays may mean that even with line-by-line transmission, the delay in displaying real-time streaming video may cause problems in some embodiments that rely on real-time streaming video of performing actions. An example of such a use environment is video surgery, where a surgeon or similar operator uses a head mounted device with a near eye display to receive an augmented view of a person or animal undergoing surgery.

[0157] In addition, such usage environments may benefit from stereoscopic displays, in which the presented images provide a sense of depth. In this example embodiment, the images may be reconstructed on one or two near-eye displays of the head-mounted viewing device. Each display may receive a video stream from a separate source, and one or both video streams may undergo additional processing at an intermediate device to add image enhancements, such as in the form of color overlays. Even if the images are captured by two or more synchronized video sources, unexpected delays, jitter, and different frame rates may cause the displayed images to lose synchronization.

[0158] In general, if two video sources need their images to be synchronized when displayed remotely, it may be necessary to provide two separate feeds, and even then, additional mechanisms may be required to ensure that the reconstructed video frames are properly aligned, which may add additional latency. For example, if two or more video sources are to be provided to a viewing device that includes displays (such as left and right eye displays in a near-eye display headset), the video sources cannot share a single SDI cable because their source clock frequencies will never be exactly the same, which means that the images displayed on each display will not be synchronized. However, it is undesirable to have two separate cables on such a device because this would make the resulting headset bulky and add additional weight. As described in more detail below, Figure 6 An example embodiment is shown in which video is provided from two or more video sources.

[0159] Figure 3A An example of an apparatus 302 for processing a serialized video stream is schematically shown. In some embodiments, the apparatus 302 may include a hub co-located with a surgical imaging device. In some embodiments, the apparatus 302 includes a hub co-located with the surgical imaging device, the hub being configured to receive a serialized video stream from one or more video sources, which may include one or more cameras as part of the surgical imaging device. In some embodiments, the apparatus 302 is a surgical imaging device or a portion thereof that includes a surgical microscope. In some embodiments, the hub may be co-located with the surgical microscope and / or may be provided as part of a control system or as other computing components or units of the microscope.

[0160] The device 302 includes a receiver 304 configured to receive a serialized video stream of a single video source 101 on a serialized video data link 103. However, according to the disclosed technology, instead of forwarding a continuous video data stream that can be used to align the source clock with the display clock, the device 302 decouples the source clock and the display clock by generating a discontinuous serialized video data stream. For each horizontal line of a video frame, the discontinuous serialized video data stream includes at least a valid pixel group, followed by a transmission pause. In some embodiments, for each received video frame line including at least a valid pixel group, at least one blanking pixel is omitted when the valid pixel group is forwarded. This reduces the time to transmit a video frame because there are fewer blanking pixels to transmit. Conversely, the transmitter can pause transmission until the next complete pixel row including valid pixels and blanking pixels is buffered, at which time the buffer is cleared and the transmitter transmits the valid pixel group and omits at least one blanking pixel. In some embodiments, one or more blanking pixels are omitted. In some embodiments, the number of blanking pixels forwarded or omitted is fixed for all video lines. After transmitting the data structure including the valid pixel group from the received video frame line and any blanking pixels from the received frame line that were not omitted for onward transmission, the transmitter is configured to wait until the next line of video frame data is ready for transmission. This causes a pause in the transmission and results in a discontinuous, intermittently transmitted stream of serialized video frame data being transmitted over the serialized video transmission link to the viewing device 410. Removing one or more blanking pixels from the end of each horizontal line of the video frame, and / or by removing or reducing the blanking pixels that form the last line of blanking pixels in each video frame, results in intermittent transmission of the serialized video data and allows the clock of the remote viewing device to be decoupled from the clock of the video source 101.

[0161] exist Figure 3A , the received serialized video frame data is buffered line by line by buffer 305, which is configured to discard one or more blanking pixels in each variable line length of the video frame and forward at least the valid pixels of each fully received video line to transmitter 310 as a pixel group with a regular line length. By removing at least one blanking pixel from each line received by device 302 from source 101, the transmitter of device 302 decouples the forwarded data from the video source clock, allowing the receiver clock to be reset accordingly.

[0162] Some embodiments of the device 302 include means for processing a serialized video data stream for onward transmission, for example, to a viewing device 312. The serialized video data stream may include a plurality of serially transmitted video frames from a video source, each video frame including a plurality of frame lines, at least one frame line including a group of active pixels and one or more blanking pixels.

[0163] The apparatus may include: a receiver 304 configured to receive a serialized video data stream; a memory 306, which may include a memory 305, arranged to buffer the received serialized video data until at least all valid pixels of a pixel frame row of the received video frame have been buffered; one or more processors or processing circuits 308 configured to, for each valid pixel group of the pixel frame row, generate a data structure including at least the valid pixel group of the frame row and a frame row position indicator assigned to the valid pixel group; and a transmitter 310 configured to intermittently forward the generated data structure by pausing transmission after each data structure has been transmitted until the next data structure is ready to begin transmission.

[0164] In some embodiments of the apparatus 302 for processing a serialized video stream, the receiver 304 is configured to receive a serialized video stream 104, for example, a video stream sent via an SDI link from a video source 102. The serialized video stream 104 includes a plurality of serially transmitted video frames from the video source 102, each video frame including a plurality of video frame rows. The first video frame row to the second-to-last video frame row each include a valid pixel group and a blanking pixel. The last frame row of each video frame includes a blanking pixel. The received video frame data can use a video transmission standard transmission link, such as a serialized data interface (SDI) transmission link 103, but can use a different transmission link 311 for forward transmission to a viewing device 312.

[0165] The memory 306 may include various types of memory, for example, memory for storing data including computer program code and memory such as at least one buffer 305, the buffer being arranged to buffer at least valid pixels of the received serialized video stream 103 until a complete pixel frame row has been received from the video source 101. One or more processors or processing circuits 308 of the device 302 are configured to include a frame row position indicator for each buffered complete pixel frame row in the forward transmitted data. In some embodiments, the valid pixels and the frame row position indicator form a data structure. In other words, the forward transmitted data includes intermittently transmitted data structures, each data structure including the valid pixel group and the frame row position indicator assigned to the valid pixel group. In some embodiments, each data structure includes the valid pixel group from the received frame row and omits at least one blanking pixel of the received frame row, however, in some embodiments, none of the blanking pixels of the received video frame row are included in the outgoing serialized video stream including the forward transmitted data structure.

[0166] In some embodiments, the frame row position indicator may be configured to indicate whether the frame row is the first frame row of the frame or a subsequent frame row in the form of a binary "is first row" or "not first row" indicator, since the number of rows in the frame is known. However, in other embodiments, a specific row position in the frame may be indicated. In some embodiments, the frame row indicator may be received in an incoming video data stream from a source and retained in outgoing forward transmitted data, however, in other embodiments, the frame row position indicator may be assigned by a processor or processing circuit. Display of the first row of valid pixels in the received video frame accordingly allows the entire image to be correctly reconstructed on the display of the viewing device 312.

[0167] In some embodiments, the frame row indicator assigned to each valid pixel group is included in a header or a trailer of a data structure including the valid pixel group, wherein a bit size of the header or the trailer matches a parallel width of a serializer of a transmitter (310) configured to serialize the data structure in outgoing video stream data.

[0168] The transmitter 310 is configured to intermittently forward transmit the outgoing serialized video data including a series of data structures with a pause between the data structures if the next data structure is not ready for transmission when the previous data structure has completed transmission. In other words, the transmitter is further configured to, after transmitting a data structure including a valid pixel group and its assigned frame line position indicator, pause the forward transmission of the outgoing serialized video stream between the data structures until the next data structure including a different valid pixel group and the next frame line position indicator is ready for transmission.

[0169] The transmitter 310 may forward the data structure as a serialized video stream 311 to a viewing device 312 including a display, for example, a suitable raster-type display such as the display including one or more LCD panels 204 shown in FIG. 2 .

[0170] Here, video frame lines or simply frame lines refer to horizontal video frame lines, such as Figures 1A-1D , 2A and 2B, when they are stacked vertically and displayed in sequence to form a video frame. In other words, in some embodiments, when displayed in the usual direction of the video screen, each video frame row includes a horizontal row of video frames, but the frame rows can also be displayed in different directions.

[0171] Now turn to Figure 4 , which schematically illustrates in more detail how the device 302 pauses transmission between forwardly transmitted data structures. Figure 4, a plurality of serialized data structures 402a-402d form a video frame and serialized data structures 406a-406b are associated with subsequent image frames when displayed on display 414 of viewing device 312. Data structures 404a-404d and 406a-406b are shown as being serialized by device 302 for transmission to viewing device 312.

[0172] like Figure 4 As shown, each data structure includes a set of valid pixel groups and related header information (for clarity, Figure 4 Header data structures are not shown). Each pixel group of the data structures 402a, 404b, 402c, 402d is transmitted sequentially, with a video frame row position indicator provided as metadata, such as in a header or trailer of the data structure. The video frame row indicator indicates to the viewing device 312 and / or the display 414 which data structure should be displayed first when reconstructing the serialized image on the display 414. Additional metadata may be provided so that when reconstructing the video frame image 416 on the display, the valid pixel groups of the data structures can be displayed on the display 414 in the correct rows and in the correct pixel order. Figure 4 In the example shown in FIG. 4 , each video frame includes only 4 horizontal lines, each line having 4 valid pixels.

[0173] Figure 4 It is also schematically shown how the device 310 pauses the transmission 404a-404d, 408a-408b between consecutive data structures 402a-402d, 406a-406d. Since there is usually a video frame line that compromises only blanked pixels after the second to last frame line that includes valid pixels, the pause 404d may be longer than the other pauses 404a-404c before transmitting the first line of the next video frame in the data structure 406a.

[0174] In some embodiments, the intermittently transmitted video data stream has a fixed frame line length and a variable transmission pause duration. The pause between transmissions can vary accordingly based on the number of blanking pixels received at device 302 but not forwarded because they are omitted from the data structure transmitted forward by device 302 to viewing device 312.

[0175] In some embodiments, the received video data stream has variable line lengths and the pauses between forwarded data structures also vary in length.The invention may also be implemented where the video data stream has a fixed number of pixels in each frame line.

[0176] The outgoing serialized video stream transmission pauses between the data structures of the first video frame are marked 404a, 404b, 404c, and after the last data structure 404d, the first data structure of the next video frame is sent, followed by a transmission pause 408a, and then the next data structure 406b of the next video frame is sent, followed by a subsequent transmission pause marked 408b, and so on.

[0177] Now back to the attached picture Figure 3B , schematically illustrates an example of a method 300 performed by a device 312 according to some embodiments of the disclosed technology, the method 300 being used to process a serialized video stream 104 for forward transmission.

[0178] like Figure 3B As shown, the method 300 for processing a serialized video stream for forwarding includes receiving a serialized video stream 314. The serialized video stream may include a video stream from a video source, such as Figure 3B The method 300 further includes buffering at least valid pixels of the received serialized video stream in 316 until a complete pixel frame row has been received in 318. The method 300 further includes processing each buffered complete pixel frame row in 320 to associate a valid pixel group of the frame row with a frame row position indicator, and the processing may include assigning a frame row position indicator or reusing or retaining an existing frame row position indicator from the received incoming video data. The method 300 further includes generating a data structure 322 including a valid pixel group, the valid pixel group including a frame row position indicator assigned to the valid pixel group. Once the data structure is ready for transmission 324, the method 300 further includes forward transmission 326 as outgoing serialized video data. The method then includes checking if the next data structure is ready for transmission in 324 , and if not, pausing transmission of the outgoing serialized video stream 328 until the next data structure is ready for transmission in 324 , at which point the data structure is transmitted in 326 .

[0179] After transmitting a data structure including a valid pixel group and an assigned frame line position indicator, the ongoing transmission is intermittent by pausing the transmission between data structures 328 until the next data structure including a different valid pixel group and a different frame line position indicator is ready for transmission 324. However, in some embodiments, the ongoing transmission may be continuous if the next data structure is always ready for transmission after the previous data structure is transmitted. In this case, the data being transmitted forward may be considered to be more efficiently transmitted with lower latency because the device 302 does not wait to receive each video frame, which would depend on the delay caused by waiting to receive each video frame line, and by removing one or more or all blanked pixels from each video frame line, each frame line has a smaller number of pixels to be transmitted, in other words, less data needs to be transmitted to pass the frame line to the destination.

[0180] The term "data structure" as used herein refers to structured data in a serialized video stream that is transmitted forward, which includes a valid pixel group and one or more indicators associated with the valid pixel group, which may also be referred to as a tag. Metadata, such as one or more indicators or tags, may be provided as a header in some embodiments, which is located before the associated valid pixel group, or may be provided as a trailer in some embodiments, which is located after its associated pixel group. In some embodiments, one or more indicators or tags may be before the associated pixel group, and one or more indicators or tags may also be after the associated pixel group. Herein, unless the context clearly indicates otherwise, references to messages are also references to indicators or tags provided at other locations (such as trailers) relative to the valid pixel groups to which they are associated.

[0181] References to means 302 for generating a data structure accordingly refer to means 302 for preparing the structured data for output in a serialized video stream for onward transmission.

[0182] In some embodiments of the method 300 performed by the device 302, any blanked pixels from received video frame rows are not included in the outgoing serialized video stream comprising the forwarded data structure.

[0183] In some embodiments, the method 300 performed by the device 302 also includes omitting or discarding at least one blanking pixel for each received video frame row that includes a valid pixel group and at least one blanking pixel, and including the valid pixel group and any remaining blanking pixels for the received video frame row in a forward transmitted data structure of the outgoing serialized video stream.

[0184] In some embodiments, the length of the forward transmission pause varies depending on the number of blanking pixels present in each frame line. For example, since the last frame line consists only of blanking pixels, there will be an extra long pause between transmitting the data structure corresponding to the second to last frame line of the frame and transmitting the data structure corresponding to the first line of the next frame in the forwarded video data stream.

[0185] In some embodiments, the method 300 performed by the apparatus 302 results in or further includes decoupling a source clock of the received serialized video stream 104 in the outgoing serialized video stream 311 .

[0186] In some embodiments, method 300 allows a viewing device component to process a received serialized video stream using a clock that is not based on a clock of a device that includes a serializer used to generate the serialized video stream.

[0187] In some embodiments of method 300, transmitting the outgoing serialized video stream 311 onward includes transmitting the outgoing serialized video stream over a data communication channel along a wired link to a viewing device (312) configured to cause a display 416 of the received video data. This provides a secure link to the display 414, however, it may not be the fastest link. In some embodiments, alternatively or additionally, transmitting the outgoing serialized video stream 311 onward includes transmitting the outgoing serialized video stream 311 onward over a wireless data communication channel to a viewing device (312) configured to cause a display 416 of the received video data.

[0188] In some embodiments of the apparatus 302 shown in FIGS. 3 and 4 , none of the blanked pixels of a received video frame row are included in an outgoing serialized video stream including a forward transmitted data structure. However, in some embodiments, at least one blanked pixel may be discarded from each row and one or more blanked pixels may be retained so as to normalize each received variable row length to a fixed number of valid and blank pixels. In other words, in some embodiments, at least one blanked pixel of each received video frame row that includes a valid pixel group and at least one blanked pixel is discarded, and the valid pixel group and any remaining blanked pixels of the received video frame row are included in the forward transmitted data structure of the outgoing serialized video stream.

[0189] In some embodiments, Figure 4 As shown, although the average frame period matches the frame period of video source 102, the buffering and removal of blanking pixels in the outgoing serialized video transmission 311 causes the video stream received by the viewing device 312 to no longer be locked to the same clock period used by the video source 104. In other words, in some embodiments, the device 312 decouples the source clock of the received serialized video stream 104 in the outgoing serialized video stream 311 by changing the pause length between the forward transmitted data structures.

[0190] In some embodiments, the transmitter 310 of the device is configured to transmit the outgoing serialized video stream as a discontinuous serialized video stream over a data communication channel along a wired link with the viewing device 312. However, in some embodiments, instead or in addition, the onwardly transmitted serialized video stream data is intermittently transmitted over a wireless data communication channel to the viewing device 312 configured to cause display of the received video data.

[0191] In some embodiments, Figure 4 As shown, decoupling is achieved by device 302 using a slightly different clock period X+δ, where δ is a small percentage difference in the clock period for each transmitted pixel in the serialized video stream 311 being transmitted forward.

[0192] The disclosed techniques also allow the clock of the receiver 410 at the viewing device 302 to use a clock that is not locked to the clock cycle used by the device 302 or the clock cycle used by the video source 102. For example, Figure 4 As shown, the clock at the receiver 410 may run at X-δ, where δ may be the same or a different amount of drift of the clock at the receiver from the source video clock X.

[0193] The viewing device 312 generates variable length video lines by adding blanking pixels after receiving valid pixels of the received data structure until the next data structure in the serialized video transmission 311 is received. In other words, during the pauses when no data structure is received, the receiver 410 adds one or more blanking pixels. This allows the receiver 410 to generate a continuous video data stream as output 412 to the display 414 in some embodiments by adding a sufficient number of blanking pixels to the end of the last valid pixel group received for display.

[0194] Figure 5 Another schematic diagram of an apparatus 302 is shown for serializing data for onward transmission according to some disclosed embodiments of the technology. Figure 5 , the device 302 shown includes a memory 306, one or more processors or processing circuits 308, a receiver 304, and a transmitter 310. The memory 306 may include a buffer memory 305, but also includes a suitable memory for storing the computer code 500, also schematically shown as Figure 5 The computer program code 500, when loaded from the memory 306 and executed by one or more processors or processing circuits 308 of the apparatus 302 for processing a serialized video data stream for onward transmission, causes the apparatus 302 to implement Figure 3BThe illustrated embodiment of method 300, or a method according to any suitable one of the method aspects disclosed herein, is for processing a received serialized video stream for onward transmission.

[0195] exist Figure 5 In the illustrated embodiment, when executed by one or more processors or processing devices, computer code 500 includes one or more software modules or circuits that cause device 302 to generate 502a data structure and transmit data structure 504. Figure 5 As shown, the data structure generation module or circuit includes a module or circuit 503, which associates metadata, such as data provided in a header or footer of a data structure, with a valid pixel group of the data structure. Although the following references may only refer to the header of the data structure, anyone of ordinary skill in the art will find that instead of the header, the footer or other designated location in the data structure can be used, and references to the term header should be interpreted accordingly in this article unless explicitly referenced to the contrary.

[0196] The computer code 500 also includes a module or circuit 504 for filling the header in 504 to form structured data, also referred to herein as a data structure, with valid pixels and associated metadata. The metadata may include metadata provided from the video source 102 or generated by the device 302.

[0197] In some embodiments, the header filling and header association modules or circuits 503, 504 may include the same modules. The header filling module or circuit 504 fills the header with one or more indicators or tags. For example, a first indicator or tag may be used to fill the header to indicate in 506 whether the pixel group of the data structure includes the first row of valid pixel groups forming a display video frame. By adding other indicators 508 (such as Figure 5 Other information required to properly display the valid pixel groups on the display may also be captured in the header (as shown by adding tag 508 in the header). In some embodiments, the header may replace or additionally include the trailer, and references to the header should be understood accordingly. The data structure transmission module or circuit 510 outputs the data structure for onward transmission to a viewing device such as viewing device 312. If after transmitting one data structure, the next data structure is ready for transmission, the data structure can be output as a continuous video data stream 311. If it is not ready for transmission, the data structure transmission module 510 pauses the transmission until the next data structure is ready for transmission.

[0198] Figure 6An example embodiment of a viewing device 312 is schematically shown, where the viewing device 312 includes a receiver 410, one or more processors or processing circuits 418, such as a graphics processing unit, and a memory 420. The receiver 410 is configured to receive a video from a device such as Figure 3A and 4 The device 302 receives serialized video data 311. In some embodiments, the display device 312 includes a display 414, which includes one or more flat panel display panels, such as LCD panels or similar panels configured to receive serialized video frame data. The flat panel display may be a curved display.

[0199] At the display 414, only active pixels are presented, while blanked pixels are ignored or discarded. Alternatively, the display 414 may be provided on a separate device connected to the viewing device 312 that receives the serialized data from the device 302. In some embodiments, the display 414 comprises a near-eye display screen or a screen on a head mounted device.

[0200] The receiver 410 is configured to receive and process a discontinuous or intermittent video data stream 311, and when a pause occurs in the received data, the receiver 410 is configured to add one or more blanking pixels to the end of each received data structure. The receiver 410 then outputs each frame line including a valid pixel group and header or trailer information indicating the row position of the pixel group in the video frame and one or more blanking pixels as a continuous data stream to the display 414, and the display 414 is configured to cause the valid pixels to be correctly presented so as to reconfigure the complete video frame to be presented in the correct order on the display 414. The display 414 discards the added blanking pixels and reconstructs each video frame 416 using the frame line position indicator to correctly order the received valid pixel groups on the display.

[0201] In such Figure 6 In the example embodiment shown, viewing device 312 includes display 416, however, as mentioned above, it will be apparent to anyone skilled in the art that in some embodiments, viewing device 312 may be connected to an external display and configured to output received video stream data to separate display 414 rather than providing received video stream data to an internal display, such as Figure 6 Display 414 is shown.

[0202] Figure 6The illustrated memory 420 may store computer code 600, which, when loaded from the memory 420 and executed by one or more processors 418 of an embodiment of a viewing device 312 according to the disclosed techniques, causes the viewing device 312 to implement an embodiment of a method 600 for receiving serialized video data, such as a continuous or intermittent sequence of data structures for presentation on a display 414. Figure 6 As shown, the computer code 600 includes a module or circuit 602 configured to receive a serialized video stream 311 including a plurality of data structures, each data structure including at least a valid pixel group and a header or trailer indicating the row position of the valid pixel group in a video frame (e.g., when displayed on the display 414). The computer program code also includes a module or circuit 604 configured to add one or more blanking pixels to the end of the data structure, in other words, if the data structure includes valid pixels, additional blanking pixels are added until the next data structure is received. This may result in the generation of an entire blanking pixel video frame row after receiving the data structure of the second-to-last video frame row of the video frame before receiving the first frame row of the next video frame.

[0203] The computer code 600 also includes a module or circuit 604 configured to output the valid pixels of the received data structure with the line position indication information and any added blanked pixels as a continuous stream of variable line length video data to the display 414 for display.

[0204] In some embodiments of the disclosed technology, the computer code 600 may also be provided in the form of a computer program product including the computer code 600 .

[0205] In some embodiments, the computer code 600 is encoded in software as one or more modules, however, in some embodiments, the computer code may be partially or completely hard-coded in circuits. The processor or processing circuits may be configured to control the execution of the computer code that is hard-coded or soft-coded in some embodiments. In some embodiments, one or more of the processors or processing circuits may include a graphics processor or dedicated graphics processing circuits.

[0206] Some embodiments of the viewing device 312 are accordingly configured to cause the computer code 600 to be loaded from the memory 420 and executed by one or more processors or processing circuits 418 to implement the method 600 of processing a received serialized video stream for display. In some embodiments, the received serialized video stream may be received intermittently, or in some other embodiments may be received continuously. Some embodiments of the method include: receiving a received serialized video stream including a plurality of data structures, each data structure including at least a valid pixel group from a video frame row and a frame row position indicator assigned to the valid pixel group; generating a complete row of video data by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure including the next pixel group in the serialized video stream is received; and causing each complete row of video data to be output for presentation on the display at a position in the video frame indicated by the frame row position indicator of the valid pixel assigned to the complete row of video data.

[0207] In some embodiments, the sequenced video stream is received from an apparatus 302 according to any embodiment of the present disclosure.

[0208] In some embodiments, the method further includes determining 605a that the penultimate frame row of a frame has been received, generating 605b a frame row including blanked pixels as the last frame row of the frame, and outputting 606 the frame row of the frame as a serialized data stream of pixels for presentation on the display 414.

[0209] Now watch Figure 7 , which shows in more detail how the viewing device 312 processes incoming intermittent video data, for example, a discontinuous serialized video data stream, and adds blanking pixels so that a continuous serialized video data stream including active pixels and blanking pixels is output to the display 414 when there is a single video source.

[0210] like Figure 7 As shown, module or circuit 602 examines the discontinuous data stream received at viewing device 312 to find header or trailer information indicating whether the associated pixel group is the first horizontal row of pixel groups of a video frame from single video source 104 .

[0211] In some embodiments, the frame row position indicator is a binary tag having a first value indicating whether the pixel group belongs to the first row of the video frame or not, taking different values ​​for all other rows in the video frame. This allows the receiving device to output the received pixel groups in the correct order because the valid pixel groups are received in order in the received serialized video stream 311 based on the preconfigured number of rows in the frame.

[0212] In some embodiments of the disclosed technology, a frame row indicator assigned to each valid pixel group is included in a header of the data structure.

[0213] In some embodiments, the viewing device is configured to interpret the received video data in fixed bit length intervals or blocks. If the header bit size can be matched to the bit size of the parallel width of the serializer of the transmitter at the video source 102 so that it has the same bit size as the transmitted pixel data, the viewing device can process the received data stream more efficiently because it only has to check one set bit size interval at a time to extract the header information.

[0214] In some embodiments, if the header is too small, for example, the device's transmitter 310 may pad the header with additional bits to increase it, so as to allow the header information to be extracted from the pixel data stream in a more efficient manner at the viewing device 312. In this case, the viewing device 312 is configured to only check the data at set bit intervals, rather than being configured to look for smaller or larger bit intervals, which would increase processing and potential latency at the viewing device 312.

[0215] These row indicators are used at the interface of the display 414, on which the continuous stream of video data received from the receiver 410 of the viewing device 412 is presented. The viewing device may buffer or discard the received data until it recognizes that a group of pixels is preceded by a first row indicator or label indicating that it is to output the pixel to the display 414. The pixel header information may also include information on how to present the rows on the display 414 in order to correctly present the image video frame 416, for example, which pixel should be presented first and the row position.

[0216] As described above, in some embodiments, each data structure may provide equivalent information in a trailer associated with a valid pixel group instead of or in addition to having a header. When a header is provided, references to the header herein should be considered to also refer to the trailer after the valid pixel group in the data structure. Regardless of whether the header is provided as a header before the valid pixel group and / or as a trailer after the valid pixel group, an indicator or tag that fills the header field is used to identify which row is the first row, and in some embodiments, this information may be provided using techniques defined by the SDI standard in the form of, for example, a 64-bit header tag. In some embodiments, the data structure of the disclosed technology may use different header or trailer sizes to identify whether the pixel group is the first row of pixel groups.

[0217] In some embodiments, the size of the header or trailer is constructed to be the same as the bit size of the pixel. This means that if the pixel is 20 bits, then the header or trailer can also be 20 bits. This makes it easier to configure the receiver of the viewing device to extract the header or trailer in an efficient manner because the received serialized data stream can be captured every 20 bits, 20 bits, etc. If the bit size of the header or trailer is different from the bit size of the pixel, for example, if the size of the header is 2 bits, this will mean that the receiver will have to capture 2 bits, then 20 bits, 20 bits, 20 bits, etc. In order to make the bit size of the header the same as the pixel, one or more dummy bits can be added in the header by the serializer of the transmitter.

[0218] Figure 6 The module or circuit 602 shown accordingly configures the receiver 410 of the viewing device 312 to buffer data until the first frame line of the video frame is received. Figure 7 As shown, at the viewing device 312, the receiver waits for the first frame row of the video frame in the received serialized video data by checking the received data structure to find the first frame row position indicator in the header. Once all valid pixels have been received for the first row of the video frame, the modules or circuits 604, 605 configure the receiver 410 to add zero, one or more blanking pixels after the valid pixels. In some embodiments, these added blanking pixels cause the viewing device to generate variable line length video data, which allows a continuous stream of serialized frame rows to be output to the display 414. The module or circuit 606 configures the receiver 410 to provide the valid pixels and subsequent blanking pixels as a continuous stream of pixels output to the display 414 until the next valid pixel group is received and streamed to the display, etc. For example, in an example where the display 414 is a raster type display in which video pixel rows are displayed from right to left, the first pixel sent is displayed on the right, and the next pixel of the valid pixel group is displayed from right to left along the horizontal line.

[0219] In some embodiments, for each received pixel group corresponding to a frame row of a video frame, once all valid pixels sent to the display have been displayed, the receiver sends blanking pixels until the next frame row of the video frame can be output. Alternatively, if the last frame row output is the second to last frame row in the frame, the receiver 410 or generates the end of the frame row including blanking pixels. In each case, the frame row data is then provided to the display 414 for presentation. After each frame row of valid pixels has been presented on the display, all subsequent pixels will be treated as blanking pixels and not displayed until the next row indicator label is received.

[0220] The identification of which pixel in the pixel group is the first pixel in the video frame row is performed at the interface to the display and conforms to standard techniques for raster type displays (e.g., LCD or OLED type displays). In some embodiments, this may also be encoded in the header structure of the pixel or pixel group. In other words, it is known in the art to provide another signal to indicate to the display 414 which pixel is the first pixel of the subsequent row (thus, by counting pixels, any additional pixels beyond a certain number of pixels will be discarded by the display 414 as blanked pixels). Such a signal sent for each row to indicate the first pixel may conform to, for example, a standard VGA or HD signal format.

[0221] The form of display 414 may vary in various embodiments, and in some embodiments may include a near-eye display.

[0222] Although the above-described embodiments relate to a single video source, in some embodiments, additional data, such as graphics overlay data, may be provided from one or more other sources and merged with the video image data at video source 102 .

[0223] In some of the above embodiments, as well as in some of the embodiments described below, additional control modules or devices may be provided at the viewing device and / or device to control the receiver and transmitter in response to executing computer code to implement the relevant methods disclosed herein at the transmitter device and the receiver device.

[0224] In the attached figure Figure 8 An example embodiment of a video display system 800 according to the disclosed technology is shown. The video display system 800 includes a video hub, also referred to herein as a video combiner device 810 according to any embodiment of the device 810 disclosed herein, and a viewing device 824 according to any embodiment of the device disclosed herein. The video combiner device 810 may include one or more video sources 812a, 812b, or Figure 8 Video sources 812a, 812b are each configured to simultaneously provide a sequenced video data stream intended for synchronous display on two displays 834a, 834b at viewing device 824.

[0225] The viewing device 824 may be connected to or include at least one of the two displays 834a, 834b. In some embodiments, each display 834a, 834b is configured to present video received from one of the video sources 812a, 812b. The viewing device may be connected to the combiner device via a serialized data link such as a serialized data interface (SDI) data link. The viewing device 824 may be provided as part of the video combiner device 810, in which case, Figure 8The illustrated transmission channel 822 may be provided over an internal wired or wireless data communication channel that may or may not conform to the SDI data standard.The display interfaces are configured to each receive a serialized video data stream and present the received data on a display with no or minimal buffering.

[0226] exist Figure 8 In one embodiment of the system shown, the video display system 800 includes a video combiner device 810 configured to process two serialized video data streams 814a, 814b.

[0227] Each video stream includes video frame data captured by a video source 812a, 812b. The device 810 includes a transmitter 820, which includes a serializer 819, which is configured to combine the two parallel video data streams 814a, 814b into a single serialized data stream on a frame-by-frame basis at the transmitter 820 for onward transmission. The transmitter 820 outputs the combined serialized data for onward transmission to a viewing device 824 on a serialized data transmission channel 822. The output may include a continuous stream or an intermittent, discontinuous data stream, depending on when a complete frame row from either source is ready for transmission.

[0228] The combiner device 810 comprises a memory 815 which may comprise one or more buffers 816a, 816b for buffering received video data streams 814a, 814b at a receiver 818 of the device 810. The device 810 further comprises one or more processors or processing circuits 817 configured to process the valid pixels of the buffered data streams 819a, 818b so as to associate them with information such as their video source and (if not already provided) the row position of each pixel group in the video frame. In some embodiments, the one or more processors or processing circuits may form part of a serializer 819. The serializer 819 may be provided as part of a transmitter 820 and is arranged to output video data from the two buffers 816a, 816b in a serialized combined data stream 822. The buffers 816a, 816b are arranged to process the received data streams 814a, 814b in parallel respectively to forward the data to the serializer 819 in parallel.

[0229] The viewing device 824 includes a receiver / display driver 826 configured to receive the serialized combined video stream 822 from the video combiner device 810. The receiver 826 is configured to check the header information of each received valid pixel group and includes a deserializer 827, which is configured to deserialize the valid pixels into two parallel channels 828a, 828b according to the header information indicating different video sources 812a, 812b. References to "headers" herein should also be interpreted as references to trailers and / or other designated locations, which may be used instead in some embodiments.

[0230] The receiver / LCD driver 826 will also add blanking pixels to each active pixel group and can buffer the output until the first frame row of video data is provided so that the outputs 830a, 830b shown as left output 830a and right output 830b to each display 834a, 834b, respectively, will be displayed correctly.

[0231] Each output data stream 832a, 832b includes a continuous stream of valid pixels and blanked pixels. At each display 834a, 834b, the blanked pixels of each output data stream 832a, 832b are discarded, and only valid pixels are presented in the proper order on each video frame line, and each video frame line is correctly positioned in the frame.

[0232] In some embodiments, Figure 8 The viewing device of the present disclosure may accordingly occasionally or constantly receive a combined data stream 822 that has undergone row-by-row interleaving of pixel groups from different video sources 812a, 812b at the combining device 810. In addition, by omitting at least one blanking pixel, the transmission time for sending video from both sources to the viewing device 824 is reduced because fewer pixels are transmitted over any shared link for onward transmission.

[0233] In some embodiments, the video combiner device 810 may include two or more video sources 812a, 812b, however, in some embodiments they may be provided separately. The video combiner device 810 and a viewing device may also be combined in some embodiments.

[0234] In some embodiments, the viewing device 824 includes two or more displays 834a, 834b, wherein each different display is configured to present video received from a different video source 812a, 812b, the video data of which has been forwarded by the combiner device 810 on the same video data channel 822. However, in some embodiments, the displays may be provided separately.

[0235] In some embodiments, displays 834a, 834b are provided as left and right near-eye displays, such as left and right displays of a head-mounted device. In some embodiments, displays 834a, b may be connected to a viewing device 824, but separated from the viewing device 824, to make the head-mounted device less bulky or reduce the weight of the head-mounted device that accommodates displays 834a, 834b. In some embodiments, the viewing device 824 may be provided as a dedicated device that is part of the display, implemented using a tablet or similar computer-type device, and in some embodiments, outputs may be provided to a real screen or virtual screen partition to replicate left and right displays. However, in other use cases, a head-mounted device or near-eye display may include a left display and a right display, for example, where the viewing device as a whole is sufficiently compact / lightweight, etc. In some embodiments, a combination of different displays may be provided, for example, if a suitable driver is configured, the viewing device 824 may output video to a near-eye display and simultaneously or nearly simultaneously to a large-size display screen.

[0236] In some embodiments, the video combiner device (also referred to herein as a hub) 810 may include a hub co-located with a surgical imaging device. In some embodiments, the hub 810 includes a hub co-located with a surgical imaging device, and the hub 810 is configured to receive serialized video streams from two or more video sources, for example, left view and right view video sources. In some embodiments, each video source includes a camera, and each camera is part of the surgical imaging device. In some embodiments, the hub 810 is a surgical imaging device or a part thereof that includes a surgical microscope. In some embodiments, the hub 810 may be co-located with a surgical microscope and / or may be provided as part of a control system or as other computing components or units of a surgical microscope.

[0237] exist Figure 8 In the illustrated embodiment of the system, there are two separate video sources 812a, 812b each configured to provide separate serialized variable line length video streams 814a, 814b to the combiner device 810. In some embodiments, more than two video sources may be configured to provide serialized video data to the video combiner 810. In some embodiments, the images from each video source are synchronously presented on a different display at the viewing device 824. However, in some embodiments, additional information may be added to one or more of the video data streams, such as overlay image data or information, which may come from another source and be combined with the video image data from one of the image video sources 812a, 812b.

[0238] Overlay information intended to be presented on the same display as the video image from source 812a, 812b can be fused or otherwise integrated with the video information at source 812a, 812b or combiner device 810. For example, such overlay information can modify the presentation of the video on the display, for example, it can highlight or adjust the color or contrast of one of the images presented on one or both displays. Alternatively or in addition, the overlay information can provide text or symbols presented with the video. In some embodiments, the overlay information is synchronized with the video sources of two or more video data so that in some embodiments the same overlay information source can be separated and combined with two serialized video streams 814a, 814b.

[0239] In some embodiments, if the number of video sources does not match the number of separate displays, the video data may be blended so that the effective pixels presented on one of the displays can include data from more than one source. In this case, in some embodiments, the data structure header information may include an identifier that, in some embodiments, indicates that the data structure includes pixels for overlay and image data (or other forms of combined data) from two sources.

[0240] In some embodiments, the hub can process the video data received from the source to add overlays or image enhancements. This can be embedded in the video stream output by the hub, or in some embodiments, it can be provided as a separate stream.

[0241] In some embodiments of the disclosed technology, the video combiner device 810 includes a device for combining video from at least two different video sources 812a, 812b (e.g., from Figure 8The embodiment shown in FIG. 8 is an apparatus 810 for combining multiple serialized video streams 814a, 614b of two different video sources 812a, 812b). The video combiner apparatus 810 includes a receiver 818, which is configured to receive each serialized video stream 814a, 814b from different video sources 812a, 812b through separate channels. Each received serialized video stream includes multiple video frames, including rows of valid pixels and blanked pixels. The combiner apparatus 810 also includes one or more processors or processing circuits 817 and a memory 815. The memory 815 may include multiple buffers 816a, 816b, each buffer being arranged to buffer at least valid pixels of the received serialized video streams 814a, 814b. One or more processors or processing circuits 817 are configured to, for each valid pixel group in a fully received pixel row in a received frame of a received serialized video stream 814a, 814b, assign a video source indicator to the valid pixel group, include or assign a frame line position indicator to the valid pixel group, and generate a data structure including the valid pixel group, the source indicator of the pixel group, and the frame line position indicator assigned to the valid pixel group. The combiner device 810 also includes a transmitter 820, which is configured to transmit the generated data structure from each received serialized video stream 814a, 814b onward as a single serialized video stream 822.

[0242] In some embodiments, one or more processors or processing circuits 817 may be provided as part of the transmitter 820 of the combined device 810. For example, by loading and executing Fig.11 The illustrated computer program code, such as computer code 1106 which may be stored in a suitable form in memory 815 , may configure one or more processors or processing circuits in some embodiments.

[0243] At the viewing device 824, one or more processors or processing circuits 836 may also be provided, for example, in some embodiments as part of the receiver 826 or the deserializer 827. In some embodiments, the one or more processors or processing circuits 836 may be configured, for example, by loading and executing computer program code from, for example, Fig.12Computer code 1200 of the memory 838 in a suitable form as shown is provided to cause a viewing device to implement a method for viewing a plurality of serialized video streams received as a single serialized combined video stream. For example, in some embodiments, the method includes: receiving a serialized video stream including a plurality of data structures, each data structure including an effective pixel group, a source indicator assigned to the pixel group, and a frame line position indicator assigned to the effective pixel group; and processing the received serialized video stream by: generating a complete line of video data by adding one or more blanking pixels to each effective pixel group of the received data structure until a data structure including the next pixel group in the serialized video stream is received, and causing each complete line of video data to be output for presentation on a display associated with the source indicator of the complete line of video data at a position in the video frame indicated by the frame line position indicator of the effective pixel assigned to the complete line of video data.

[0244] In some embodiments, the viewing device 824 includes a receiver 826 configured to receive a serialized video stream 822 including data from a plurality of different sources 812a, 812b, for example, a combined serialized video stream 822 from a combiner device 810. The serialized video stream 822 includes a plurality of data structures, each of which includes an effective pixel group, a source indicator assigned to the pixel group, and a frame line position indicator assigned to the effective pixel group. The viewing device 824 also includes one or more processors or processing circuits 836 configured to generate a complete row of video data by adding one or more blanking pixels to each effective pixel group of the received data structure until a data structure including the next pixel group in the serialized video stream is received. Each complete row of video data is then provided for presentation on a display 834a, 834b associated with the source indicator of the complete row of video at a position in the video frame indicated by the frame line position indicator of the effective pixel assigned to the complete row of video data.

[0245] The attached drawings Fig. 9 Schematically illustrating some embodiments of the operation of the combiner device 810, for example Figure 8 In the embodiment of the apparatus 810 schematically shown in FIG. 8 , the transmitter 820 may occasionally pause forward transmission between data structures.

[0246] exist Fig. 9 , the serializer 819 is not shown for clarity. The pauses are labeled TX PAUSE #1 through #5 and may occur from time to time between transmissions of data structures, or may occur all the time between data structures. Fig. 9, the data structures are labeled A#1, B#1, A#2, B#2, A#3, where A# indicates that the data structure includes valid pixels originating from video data source 812a, and B# indicates that the data structure includes valid pixels originating from 812b.

[0247] The transmission pause may occur regularly or irregularly, depending on the timing of the data structure transmission and the time required for the next data structure to be ready for transmission after the previous data structure transmission is completed.

[0248] Each of the valid pixel groups is assigned a source indicator at the combiner device before it is transmitted forward, the source indicator being included in the header of the pixel group. Additional header information may include a frame line position indicator for the pixel group and any other header information that may be retained from the received serialized data from the video source. In some embodiments, for example, the transmission of a frame line from one source 812a may be completed before the next data structure including a different valid pixel group with a different source indicator #B is ready for transmission, in which case, rather than pausing, if another data structure from the first source is ready for transmission, the other data structure is transmitted rather than waiting for the frame line of the other source to be ready.

[0249] like Fig. 9 As shown, the transmission is from a first source, such as the first frame line of video source 812a (at Fig. 9 1). Then, since there is no subsequent data structure ready for transmission, a transmission pause marked as TX#1 occurs. However, once the subsequent data structure B#1 is ready for transmission, B#1 is output along the serialized video data link 822. In the example shown, the next data structure ready for transmission is marked as B#1, and after B#1 is transmitted, another pause occurs, namely pause #2, data structure #A2 is ready for transmission, then #A2 is transmitted, and another pause #3 occurs before B#2 is ready for transmission, then B#2 is transmitted, after which pause #4 occurs, and data structure A#3 is transmitted, followed by another pause.

[0250] However, the transmitter 820 may send out a continuous stream of pixels by alternating the source pixel groups being transmitted, so that after transmitting a data structure including valid pixels of a video frame from a first video source 812a, if a data structure including valid pixels of a video frame from another video source 812b is ready for transmission when it completes transmitting a data structure including pixels from the first video source 812a, the transmitter next sends out a data structure including valid pixels of a video frame from another video source 812b. In other words, some or none of the pauses shown may occur in some examples of the disclosed technology, and the frame line sequence may not strictly alternate from one source to another.

[0251] In some embodiments, each frame line, when displayed, comprises a horizontal line of a video frame.

[0252] In some embodiments, the video source indicator identifies a spatial relationship between at least two of the plurality of video sources. Figure 8 As shown, in some embodiments, two video sources 812a, 812b provide left and right views of a scene, and a receiver 826 of a viewing device 824 is configured to examine the received video stream 822 to find an indicator for each video source and to allocate valid pixel groups of a data structure associated with the source indicator to one of the two displays 834a, 834b accordingly.

[0253] In some embodiments, the multiple serialized video streams 814a, 814b include a video stream for a left view of the scene obtained from a video source and a video stream for a right view of the scene obtained from the video source, and different video source indicators are used to identify whether a pixel group belongs to a frame for the left view of the scene or a frame for the right view of the scene in a video frame obtained from the video source.

[0254] In some embodiments, the assigned video source indicator and frame row position indicator are included in a header of the data structure, where the bit size of the header matches the parallel width of a serializer of the transmitter 820, the serializer being configured to combine data structures from multiple different video sources 812a, 812b into an outgoing serialized video stream 822.

[0255] In some embodiments, data structures from two sources 812a, 812b are interleaved by the serializer into an outgoing serialized video stream 822.

[0256] In some embodiments, interleaving alternates data structures from one video source with data structures from another video source in the outgoing serialized video stream. In some embodiments, if the data structure from the other video source is not yet ready for transmission, the transmitter 820 will pause transmission until it is ready for transmission. Alternatively, in some embodiments, the transmitter may transmit a data structure derived from a frame row of the same source as a previously transmitted frame row.

[0257] The apparatus 810 may be configured to receive at least one serialized video stream 814a, 814b on a channel having one or more of a different data rate and a different video resolution relative to at least one other channel via which another received video stream 814a, 814b is received.

[0258] While latency reduction at displays 834a, 834b may be better achieved by removing all blanking pixels in received video frame rows at device 810, such that preferably received video frame rows with no blanking pixels are included in the outgoing serialized video stream 822 including the data structure for forward transmission, a similar benefit in latency reduction may be achieved by instead removing at least one blanking pixel for each received video frame row including a valid pixel group, discarding at least one blanking pixel and including the valid pixel group and any remaining blanking pixels for that received video frame row in the outgoing serialized video stream's forward transmission data structure.

[0259] In some embodiments, the video combiner device 810 decouples the source clock of the received serialized video stream in the outgoing serialized video stream by switching between different video sources when combining serialized video data for onward transmission, and / or pausing the transmission of a data structure after a previous data structure has been transmitted if the data structure is not yet ready for transmission. In some embodiments, the receiver clock and the source clock are correspondingly decoupled, and can be further decoupled by varying the pause length between data structures transmitted onward. The transmitter clock of the serializer 819 can also be correspondingly decoupled from the clock at the receiver 826 of the viewing device 824.

[0260] In some embodiments, transmitter 810 is configured to transmit the outgoing serialized video stream over a data communications channel 822 along a wired link and / or over a wireless data communications channel to a viewing device 824 configured to cause display of the received video data.

[0261] The attached drawings Fig.10 An example embodiment of a method 1000 performed by a combiner device 810 configured to combine two parallel streams of serialized video data is shown. Fig.10 As shown, method 1000 includes: from a video source, such as Figure 8 9, and simultaneously receives 1002a video stream from another video source, such as Figure 89 receives 1002b a video stream. Each stream is then buffered 804a, 804b, respectively, until a complete video frame row is received in 1006a, 1006b, at which point, in some embodiments, a header associated with the valid pixels of each video frame row is populated 1008a, 1008b with an indicator or tag that provides an indication of the video source of the video frame for each valid pixel group. The header may already include or be assigned a video frame row indication and any other relevant information, such as which pixel in the valid pixel group is the first pixel that should ultimately be presented on a display of the video frame at the viewing device 824. A data structure including the valid pixel and header information is then generated 1010a, 1010b, and whichever of the two resulting data structures is ready for transmission next is transmitted forward in 1014, for example, toward Figure 8 and 9 824. In some embodiments, the other data structure may be buffered until it can be transmitted, in which case the forward transmission may be continuous from one data structure to another, and in some embodiments, the data structures may alternate sources. If no data structure is ready for transmission from either source after the previous data structure has been transmitted, the sender may pause transmission 1016 until the next data structure is ready for transmission.

[0262] The clock at the viewing device may be decoupled from the transmission clock at the original video source 812a, 812b, since any pause in transmission will interrupt the timing of the pixel groups being transmitted according to their original clock timing.

[0263] In some embodiments, the method 1000 includes: combining a plurality of serialized video streams 814a, 814b from different video sources 812a, 812b for serialized forward transmission over a channel, each serialized video stream being received over a different communication channel. The method includes: in some embodiments, receiving 802a, 802b each of the plurality of serialized video streams, respectively, buffering at least valid pixels of a frame row of a received serialized frame of each of the plurality of received video streams, respectively, processing each buffered valid pixel group in a fully received frame row by at least assigning 808a, 808b a video source indicator to the valid pixel group, and generating 810a, 810b a data structure including a valid pixel group, an indicator of a video source of the pixel group, and a frame row position indicator of the valid pixel group, and forwarding over a common channel the generated data structure including buffered valid pixels of a frame row of a video frame from one of the plurality of received serialized video streams.

[0264] In some embodiments of method 1000, the method further includes: after each data structure has been transmitted, determining whether another data structure of video frame rows of video frames from another video source among the multiple video sources is ready to start forward transmission, and if so, forward transmitting the data structure from the other video source among the multiple video sources on the common channel, otherwise determining whether another data structure from one video source among the multiple video sources is ready to be transmitted, and if so, forward transmitting the data structure from the one video source among the multiple video sources on the common channel.

[0265] In some embodiments of the method, after the data structure has been transmitted, the method further includes: determining that no data structure from any of the multiple video sources is ready for transmission, pausing transmission until the data structure from the video source in the multiple video sources is ready for transmission, and forwarding the data structure on the common channel.

[0266] Fig.11 The schematic diagram shows how the device 812 includes computer code 1106, for example, the computer code is stored in a memory or configured in hardware in the form of a computer program, and the computer code can be executed by one or more processors or processing circuits 817. The computer program code includes one or more modules or circuits in some embodiments, which control various components of the combiner device 810 to implement functions such as Fig.10 Method 1000 and the like are shown.

[0267] Fig.11 It is shown how the computer code 1106 includes one or more modules or circuits 1108 and one or more modules or circuits 1110, which when executed, cause a data structure with a header to be generated, and one or more modules or circuits 1110, when executed, cause the header to be filled. In some embodiments, the computer code 1110 includes one or more modules or circuits 1202, which when executed, are configured to cause the combiner device 810 to determine or check the presence of an indicator of the video line position of the valid pixel group in the frame in the received video data stream. The frame line indicator is included in the data structure transmitted forward and can indicate, for example, whether the valid pixels of the video frame line belong to the first line of the video frame.

[0268] If there is no frame row position indicator associated with a valid pixel group, the header will be filled with the frame row indicator.The frame row indicator may include a binary indicator in some embodiments as to whether the frame row is the first frame row or not the first frame row.

[0269] The computer code 1106 also includes one or more modules or circuits 1204 that are configured to, when executed, cause the header to be populated with any other relevant information. For example, the computer code 1106 may include one or more modules or circuits in 1116 that are configured to, when executed, cause an indicator or tag to be added to the header to identify the video source or indicator of the active pixel group, wherein the viewing device is configured to send the active pixels associated with the particular video source identifier to a particular one of its displays 834a,b. Alternatively or in addition, in some embodiments, the indicator or tag may identify the display on which the active pixel group is to be displayed, in which case, upon detecting the particular display indicator, the viewing device is configured to forward the received active pixels with the particular display indicator (and any added blanked pixels) to that particular display.

[0270] In some embodiments, other relevant information may include indicators or labels added to indicate which pixel is displayed first in each row and / or the direction in which the rows are presented on the display, such as from left to right, etc.

[0271] As is apparent to anyone skilled in the art, in the above-described embodiments, the header may be provided instead of or in addition to the trailer, or located at another predetermined position relative to the valid pixel group in the forwarded data structure.

[0272] Fig.12 Schematically illustrates how a viewing device 824 may include one or more processors or processing circuits 838, a memory 836, and a receiver 828, such as a receiver / driver for a plurality of displays 834a, 834a. Fig.12 As shown, the displays are included in the viewing device, but in some embodiments they may be separate displays. In some embodiments, the viewing device 824 may be configured to receive video routed from the combined device 810. In some embodiments, the displays 834a, 834b include left and right near-eye displays in a head-mounted device or similar near-eye display viewing device.

[0273] like Fig.12 As shown, the memory 836 includes memory storing computer code 1200. The computer code 1200 may include one or more modules or circuits 1202, 1204, 1206 that, when executed by one or more processors or processing circuits 838, are configured to cause the viewing device 824 to perform a method for viewing multiple serialized video streams received as a single serialized video stream.

[0274] For example, in some embodiments, the computer code includes one or more receiver modules or circuits 1202 that, when executed, cause the viewing device 824 to receive a serialized video stream 822 comprising a plurality of data structures, each data structure comprising an active pixel group, a source indicator assigned to the pixel group, and a frame line position indicator assigned to the active pixel group. The computer program code may also include one or more deserialization modules or circuits 1204 that, when executed by the viewing device 824 or a deserializer 827 of the device 824, process the received serialized video stream 822 to generate a complete line of video data by adding one or more blanking pixels to each active pixel group in the received data structure until a data structure comprising the next pixel group in the serialized video stream is received.

[0275] The computer program code may also include one or more data output modules or circuits 1206 that are configured to, when executed, cause each complete line of video data to be output for presentation on a display 834a, 834b at a position in the video frame indicated by a frame line position indicator of valid pixels assigned to the complete line of video data, the display being associated with a source indicator of the valid pixels.

[0276] In the above embodiments, the video received by the device 302, 802 from one or more video sources 102, 812a, 812b may be received from the video source via the SDI link 14. The memory 306 may include appropriate read-only memory and / or random access memory for storing computer code for execution by one or more processors or processing circuits 308. The memory 306 may also include at least one line buffer 305, 816a, 816b for storing each horizontal line of video data as each horizontal line of video data is received from the video source 102, 812a, 812b on the SDI link. When the buffer 306, 816a, 816b is filled, in other words, when a complete horizontal line of valid and blanked pixels has been received, the blanked pixels are discarded or ignored or removed, and the valid pixel group is processed for forward transmission. In some embodiments, the forward transmission may not include any pauses and may be continuous. In some embodiments, some data structures may be transmitted without pause if they are buffered and ready for transmission when a previous data structure has been transmitted by the device 310, 810, but may be paused from time to time if no data structures are ready.

[0277] The display 414, 834a, 834 may include any suitable type of display, such as a raster scan type display, for example, a display that uses a liquid crystal display (LCD) or light emitting diode (LED) or OLED or organic light emitting diode technology to display video.

[0278] Video data is transmitted serially, pixel by pixel, over a serialized data interface (SDI) link or channel. The order in which the bits (usually 24 bits) of each pixel are transmitted may be determined differently by different communication protocols. However, if it is serialized, the most significant bit of each pixel is sent first, and the remaining pixels are sent in sequence until the last pixel is sent. Serialized video streams, depending on the required video resolution, can run at speeds up to 6Gbit / s or 12Gbit / s.

[0279] In some embodiments, the transmission time of each frame is different, and only the average frame period of the forward transmission is matched by the device 310, 810 to the frame period of the video received from the video source 102, 812a, 812b. In other words, the source clock used by the device 310, 810 to transmit each pixel does not need to be the same as the display clock used at the display 314, 834a, 834b to receive each transmitted pixel.

[0280] exist Figure 3A and Figure 8 In the embodiment, the transmitters 312 and 820 are respectively configured to ensure that the average transmission frame period matches the transmission time of each video source. In other words, in some embodiments, the transmitters 312 and 820 of the present disclosure are configured to allow the transmission period of each pixel in the video frame to be different. In the embodiments described herein, due to the clock difference between the device 310, 810 and the source 102 in some embodiments, the idle period during the transmission is used to compensate for the extra / fewer valid pixels from the video source.

[0281] For example, now refer back to Figure 4 , although the following also relates to embodiments where video from two sources is being combined, in pixel group 402a, by way of example embodiments, the clock period for transmitting pixels may be X=0.1000 μs plus δ=0.0001 μs, in other words, the clock is X+δ=0.10001 μs, while the clock period for the receiver of display 312 to receive pixels of a blanking line may be X=0.1000 μs less than δ=0.0001 μs, i.e., X-δ=0.999 μs. In other words, in some embodiments of the disclosed technology, the receiver 410, 824 of the viewing device 410, 824 uses a local clock to drive the output to the display 414, 834a, 834b, in other words, the viewing device may use a local clock to drive the rate at which pixels are presented on the display (e.g., an LCD panel) without the need to obtain a clock from the transmitter 310, 820. This allows the transmission time of individual frames to vary within a certain tolerance level, in other words, as long as the frame transmission duration does not vary excessively on average.

[0282] Variations in the arrival time of each horizontal line of serialized video data are compensated for by varying the blanking period that the display's driver 412, 824 adds to the end of each received active pixel group 402a-402d of the video frame. As described above, these blanking pixels are not displayed, but can compensate for differences between the viewing device's receiver clock 410, 824 and the source clock at the transmitter 310, 820 and / or the source clock of the video source 102, 812a, 812b.

[0283] Because blanking pixels added by the display are used to compensate for the arrival times of different video lines at the display, the pixels forming the pixel columns output on the display panels 414, 834a, 834b of the viewing device 410, 824 can be synchronized at the display even with two or more separate video sources.

[0284] Some embodiments of the disclosed technology are configured to provide a video output to a viewing device 410, 824, the viewing device being configured to drive a display 414, 834a, 834b, for example, an LCD panel display, the video output having variable length horizontal video frame lines. In some embodiments, the variable length horizontal frame lines are set to catch up with the source video horizontal timing, but it is not necessary to lock to the video source clock timing only. Accordingly, some embodiments may use a proprietary serialized data interface type of video transmission method, which does not require locking to the video source clock at the display.

[0285] In some embodiments, viewing device 410, 824 is configured to automatically add a variable number of video blanking pixels to each pixel group including a horizontal line of valid pixels received from device 310, 810 until the last valid pixel of the next complete video line is received. Advantageously, this can allow error correction to be performed on the next received video frame line.

[0286] In some embodiments of the disclosed technology, the device 302 receives two serialized video streams from different sources. Figure 5 An example embodiment of apparatus 302 including apparatus 510 is shown.

[0287] In some embodiments, the video data being transmitted forward is compressed and one or more communication links are configured to have a total active bandwidth that is less than the sum of the active bandwidths of each source.

[0288] In some embodiments, the video sources 12, 512a, 512b may be RGB camera sources. In some embodiments, the display 312 or display system 518 comprises a near-eye display, such as a stereoscopic near-eye display.

[0289] Such displays may include microdisplays or liquid crystal displays that specify a fixed number of blanking pixels after a video line of active pixels, for example, 1920 active pixels followed by 40 blanking pixels, however, it has been found that the viewing device of the present disclosure can provide 1920 active pixels followed by 39 or 41 blanking pixels, etc., without significantly affecting the way the video is displayed. Accordingly, in some embodiments, the viewing device does not need to add a fixed number of blanking pixels to the end of each active pixel group. Advantageously, the disclosed embodiment of the combiner device 810 that receives video from two or more sources enables the two or more video sources 812a, 812b to have independent clocks and still share the same transmission cable. In some embodiments, an additional video frame buffer may be used, but in the absence of the need for such a frame buffer, the video received by the display 16, 518 can have a low or very low latency, for example, on the order of 10s or μs.

[0290] The disclosed embodiments described above may allow a display such as an LCD panel to no longer need to generate its clock from the source of the video it is receiving for display. Compared to using a conventional SDI transmitter that requires the source video clock to be locked to, the disclosed techniques allow video from two or more different video sources to be transmitted on the same or different physical channels, even if the video sources are not clock synchronized. In addition, by transmitting data line by line in the manner disclosed herein, the long delays required when using conventional SDI transmitters can be avoided because there is no need to have a video frame buffer.

[0291] In some embodiments, the displays 834a, 834b comprise near-eye stereoscopic displays based on diode technology, such as LCD or OLED displays.

[0292] The above program code can also be provided as a computer program product, for example, in the form of a data carrier carrying a computer program code or code device, which, when loaded into a processing circuit in a control unit, is used to perform the embodiments herein. The data carrier or computer readable medium can be one of an electronic signal, an optical signal, a radio signal, or a computer readable storage medium. For example, the computer program code can be provided as a pure program code on a control unit or on a server and downloaded to the control unit. Therefore, it should be noted that in some embodiments, the functions of the control unit can be implemented as a computer program stored in a memory, for example, a computer readable storage unit, so as to be executed by a processor or a processing module, for example, a processing circuit in a control unit.

[0293] It should also be understood by those skilled in the art that the above-mentioned processing circuits and memories or computer-readable storage units may refer to a combination of analog circuits and digital circuits, and / or one or more processors configured with software and / or firmware stored, for example, in the memory, and when one or more processors, such as processing circuits, execute the software and / or firmware, the execution is performed as described above. One or more of these processors, as well as other digital hardware, may be included in a single application-specific integrated circuit (ASIC), or multiple processors and various digital hardware may be distributed in several separate components, whether individually packaged or assembled into a system-on-chip (SoC).

[0294] The control unit in some embodiments may also include or be capable of controlling how the signal is sent along the communication link 103, 822, which may be a wired or wireless communication link. If the signal is sent wirelessly, the control unit may also appropriately control the operation of the antenna.

[0295] The communication channel may be point-to-point or a network, such as through a cellular network or a satellite network supporting wireless communications. The wireless communications may conform to one or more public or proprietary communications standards, protocols and / or technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Instant Messaging and Presence Extensions to Session Initiation Protocol (SIMPLE) and / or Instant Messaging and Presence Service (IMPS)) and / or Short Message Service (SMS)), or any other suitable communications protocol, including communications protocols not yet developed as of the date of filing of this document.

[0296] The operating system of the disclosed technology may require various other software components and / or drivers to control and manage general system tasks (e.g., memory management, storage device control, power management, etc.) and, in some embodiments, facilitate communications between various hardware and software components that are not shown in the figures but whose inclusion is apparent for clarity.

[0297] When the disclosed technology is described with reference to the accompanying drawings in the form of block diagrams and / or flow charts, it can be understood that several entities in the drawings, for example, the blocks of the block diagrams, and the combination of entities in the drawings, can be implemented by computer program instructions, which can be stored in a computer-readable memory or loaded onto a computer or other programmable data processing device. Such computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, and / or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer and / or other programmable data processing device can generate a method for implementing the functions / actions specified in the block diagrams and / or flow charts.

[0298] In some implementations, and according to some aspects of the present disclosure, the order in which the functions or steps in the blocks appear may not be consistent with the order described in the operating instructions. For example, depending on the functions / actions involved, two blocks shown in succession may actually be executed substantially concurrently, or may sometimes be executed in reverse order. In addition, the functions or steps indicated in the blocks may be executed continuously in a loop according to some aspects of the present disclosure.

[0299] The description of the example embodiments provided herein has been presented for illustrative purposes. This description is not intended to be exhaustive or to limit the example embodiments to the precise form of the present disclosure, and modifications and variations are possible in light of the above teachings, or may be obtained from the practice of various alternatives to the provided embodiments. The examples discussed herein are selected and described in order to explain the principles and properties of the various example embodiments and their practical applications, so that those skilled in the art can utilize the example embodiments in various ways and with various modifications suitable for specific intended uses. The features of the embodiments described herein may be combined in all possible combinations of methods, devices, modules, systems, and computer program products. It should be noted that the example embodiments presented herein may be practiced in any combination with each other.

[0300] It should be noted that the word "comprising" does not necessarily exclude the existence of other elements, features, functions or steps than those listed, and the word "a" or "an" preceding an element does not exclude the existence of multiple such elements, features, functions or steps. It should be further noted that any figure mark does not limit the scope of the claims, and the example embodiments may be implemented at least in part by means of hardware and software, and multiple "devices", "units" or "equipment" may be represented by the same item of hardware.

[0301] The various example embodiments described herein are described in the general context of methods and may refer to elements, functions, steps, or processes, one or more of which or all may in some aspect be implemented by a computer program product embodied in a computer-readable medium including computer-executable instructions, such as program codes, executed by a computer in a network environment.

[0302] Computer-readable media may include removable and non-removable storage devices, including but not limited to read-only memory (ROM), random access memory (RAM), which may be static RAM, SRAM or dynamic RAM, DRAM. ROM may be a programmable ROM, PROM, or EPROM, an erasable programmable ROM, or an electrically erasable programmable ROM, EEPROM. Suitable storage components of the memory may be integrated as a chip onto a printed circuit board or other substrate connected to one or more processors or processing modules, or provided as a removable component, for example, a flash memory (also known as a USB stick), a compact disk (CD), a digital versatile disk (DVD), and any other suitable form of memory. Unless not suitable for the present application, the memory may also be distributed on various forms of memory and storage components and may be provided remotely on one or more servers, such as may be provided by a cloud-based storage solution. In general, program modules may include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for performing the steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0303] Therefore, any device described herein, regardless of the form of its electronic device, uses a memory including any appropriate device readable and / or writable medium, examples of which include, but are not limited to: any form of volatile or non-volatile computer readable memory, including, but not limited to, persistent memory, solid-state memory, remote mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (such as hard disks), removable storage media (such as flash drives, compact disks (CDs) or digital video disks (DVDs)) and / or any other volatile or non-volatile, non-temporary device readable and / or computer executable storage device, which stores information, data and / or instructions that can be used for processing circuits. The memory can store any appropriate instructions, data or information, including computer programs, software, applications including one or more of logic, rules, codes, tables, etc. and / or other instructions that can be executed by processing circuits, and is used by devices in any form of electronic devices. The memory can be used to store calculations performed by the processing circuit and / or any data received via a user or communication or other type of data interface. In some embodiments, the processing circuit and the memory are integrated. The memory can also be dispersed in one or more system or device components. For example, the memory may include multiple different memory modules, including modules located on other network nodes in some embodiments.

[0304] In the drawings and the specification, exemplary aspects of the present disclosure have been disclosed. However, many changes and modifications may be made to these aspects that fall within the scope of the appended claims. Therefore, the present disclosure should be regarded as illustrative rather than restrictive in terms of supporting the scope of the claims, and the scope of the claims should not be limited to the specific examples of the above-mentioned aspects and embodiments. The present invention is exemplified by the above-mentioned aspects and embodiments, and its scope is limited by the appended claims.

Claims

1. An apparatus (810) for combining a plurality of serialized video streams from different video sources (812a, 812b) for serialized forward transmission on a channel (622), each serialized video stream being received on a different communication channel, the apparatus comprising: a receiver (818) configured to receive each of the plurality of serialized video streams, respectively; a plurality of buffers (816, 816b), each buffer arranged to buffer at least valid pixels of a frame row of a received serialized frame of one of a plurality of received video streams; One or more processors or processing circuits (821) configured to, for each valid pixel group in a completely received pixel row in a received frame of a received serialized video stream, assigning a video source indicator to the active pixel group; as well as generating a data structure including the active pixel groups, source indicators assigned to the pixel groups, and frame line position indicators for the active pixel groups, each data structure omitting at least one blanked pixel of the fully received frame line; as well as The transmitter (620) is configured to transmit the generated data structure of the two received serialized video streams onwards on a common channel (822).

2. The apparatus of claim 1, wherein the transmitter is configured to pause transmission after a data structure from one video source has been transmitted until a next data structure from another video source or from the same video source is ready to begin transmission.

3. The video combiner device (810) of claim 1, wherein the video source indicator identifies a spatial relationship between at least two of the plurality of video sources.

4. A video combiner device (820) as claimed in any one of claims 1 to 3, wherein the allocated video source indicator and the frame line position indicator are included in a header of the data structure, and wherein the bit size of the header matches the parallel width of a serializer (821) of a transmitter (820) configured to combine data structures from multiple different video sources (812a, 612b) into one outgoing serialized video stream (822).

5. A video combiner device (820) as claimed in any one of claims 1 to 4, wherein data structures from two sources (812a, 612b) are interleaved by the serializer (821) in an outgoing serialized video stream (822).

6. The video combiner device (820) of any preceding claim, wherein at least one received serialized video stream (814a, 614b) is received on a channel having a data rate or video resolution relative to at least one other channel having one or more of: Different data rates; and Different video resolutions; Another received video stream is received via the at least one other channel.

7. The video combiner device (820) of claim 1, wherein none of the blanked pixels of the received video frame lines are included in the outgoing serialized video stream (822) including the forwarded data structure.

8. A video combiner device (820) as claimed in any one of claims 1 to 7, wherein the transmitter (810) is configured to transmit the outgoing serialized video stream over a data communication channel along a wired link and / or over a wireless data communication channel to a device (834) configured to cause display of the received video data.

9. A viewing device (824), comprising: A receiver (826) configured to receive a serialized video stream (822) comprising a plurality of data structures, each data structure comprising a valid pixel group, a source indicator assigned to the pixel group, and a frame line position indicator assigned to the valid pixel group; as well as One or more processors or processing circuits (836) configured to: generating a complete line of video data by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure including a next pixel group in the serialized video stream is received; as well as Each complete line of video data is caused to be output for presentation on a display (834a, 834b) associated with a source indicator for the complete line of video at a position in the video frame indicated by a frame line position indicator of valid pixels allocated to the complete line of video data.

10. A viewing device (824) as described in claim 6, wherein the received serialized video stream (822) is received from a device (810) according to any one of claims 1 to 5, wherein the received intermittently transmitted serialized video data is processed by one or more processors or processing circuits (836) to add a variable number of blanking pixels for output to a display as a continuous serialized video stream.

11. A method (800) for combining a plurality of serialized video streams (814a, 814b) from different video sources (812a, 812b) for serialized forward transmission over a shared channel, each serialized video stream being received over a different communication channel, the method comprising: Receiving (802a, 802b) each of a plurality of serialized video streams, respectively; buffering (804a, 804b) at least valid pixels of a frame row of a received serialized frame of each of a plurality of received video streams, respectively; Each buffered valid pixel group in a fully received frame line is processed at least in the following manner: assigning (808a, 808b) a video source indicator to the active pixel group; and generating (810a, 810b) a data structure including the active pixel group, an indicator of a video source for the pixel group, and a frame line position indicator for the active pixel group; as well as The generated data structure comprising buffered valid pixels of a frame row from a video frame of one of the plurality of received serialized video streams is transmitted forward on the common channel.

12. The method (800) of claim 11, wherein the method further comprises: After each data structure has been transmitted, determining whether another data structure of video frame rows from another one of the multiple video sources is ready to begin forward transmission, and if so, forward transmitting the data structure from the other one of the multiple video sources on the common channel; otherwise determining whether another data structure from one of the multiple video sources is ready to be transmitted, and if so, forward transmitting the data structure from the one of the multiple video sources on the common channel.

13. The method of claim 12, wherein after the data structure has been transmitted: determining that no data structure from any of the plurality of video sources is ready for transmission; pausing transmission until a data structure from a video source among the plurality of video sources is ready for transmission; and Transmit the data structure forward on the common channel.

14. A method for viewing a plurality of serialized video streams received as a single serialized video stream, the method comprising: Receiving a serialized video stream (822) comprising a plurality of data structures, each data structure comprising a valid pixel group, a source indicator assigned to the pixel group, and a frame line position indicator assigned to the valid pixel group; as well as The received serialized video stream (822) is processed in the following manner: generating a complete line of video data by adding one or more blanking pixels to each valid pixel group of the received data structure until a data structure including a next pixel group in the serialized video stream is received; as well as Each complete line of video data is caused to be output for presentation on a display (834a, 834b) associated with a source indicator of the active pixel group at a position in the video frame indicated by a frame line position indicator of active pixels assigned to the complete line of video data.

15. A video display system, comprising: The video combiner device according to any one of claims 1 to 8; as well as A viewing device according to any one of claims 9 or 10.

Images