Serialized Video Transmission

Abstract

Disclosed is an apparatus for processing a serialized video stream and related aspects. For example, the apparatus (302) includes a receiver (304) configured to receive a serialized video stream including a plurality of video frames serially transmitted from a video source (102), each video frame including a plurality of frames, each frame including a plurality of frame lines including a group of valid pixels and a group of blank pixels; at least one buffer (305) configured to buffer valid pixels of the received serialized video stream until at least a complete frame line of pixels is received; and for each buffered complete frame line, assigning a frame line position indicator to the buffered complete frame line of pixels and assigning a frame line position indicator to the buffered complete frame line of pixels and assigning a frame line position indicator to the valid pixels and the blank pixels. and one or more processors or processing circuits (308) configured to form data structures including frame line position indicators assigned to effective pixel groups, whereby each data structure includes effective pixels from a received frame line and excludes blank pixels of the received frame line; and a transmitter (310) configured to forward transmit an output serialized video stream including the formed data structures, wherein after a data structure including the effective pixels and assigned pixel position indicators has been transmitted, the forward transmission of the output serialized video stream is paused between data structures until the next data structure including different effective pixel groups and different frame line position indicators is ready to be transmitted. The figure accompanying this abstract is Figure 4.

Description

[Technical Field]

[0001] The present disclosure relates to apparatus and methods and related aspects for serialized video transmission, i.e., intermittently forward transmitting serialized video frame lines received as a continuous pixel stream from at least one video source to a viewing device.

[0002] In particular, but not exclusively, the present disclosure relates to an apparatus configured to discard one or more blank pixels from each received video frame line, and to pause transmission after a group of pixels including valid pixels and zero or more remaining blank pixels from one video frame line has been transmitted until pixels of the next video frame line are ready for forward transmission.

[0003] In particular, but not exclusively, some embodiments of the disclosed technology relate to a system in which video from two or more sources is live streamed with minimal latency to one or more remote displays for synchronized, low-latency output, such as near-eye displays in a headset. [Background technology]

[0004] Some aspects and embodiments of the disclosed technology may be particularly useful in use cases, including but not limited to, those requiring highly accurate and reliable live video streams from two or more separate asynchronous sources for reconstruction with minimal latency at a receiver display. Such images may be used in a variety of applications, particularly those that provide some visual indication of perspective and / or depth within the image, such as can be provided by a stereoscopic display.

[0005] By way of example, some embodiments of the disclosed technology can be used in medical settings, for example, in connection with surgical procedures where a surgeon or associated medical personnel performs or assists in the procedure while observing live video footage of the vicinity of the surgical site displayed on a near-eye display or screen. Such a display advantageously enables image acquisition of a patient undergoing surgery, where one or more imaging modes make it possible to distinguish healthy tissue from unhealthy tissue, enhance body parts and organs, and obtain a magnified view of the particular area being observed and manipulated.

[0006] However, such use cases require any headset-style display that is lightweight and comfortable to wear, as well as extremely low latency and highly stable synchronization of the video feeds from at least two video sources. Synchronization of images from different sources is crucial because any loss of synchronization or jitter in the live video stream can cause significant problems during surgery. For example, if the near-eye video stream for the left view is not synchronized with the video stream containing the near-eye image for the right view. Even when video is received from a single video source, managing the data flow is equally important to avoid problems such as buffer overflow as the video is processed for onward transmission from the source to the display device.

[0007] Displays using liquid crystal display (LCD) panels can be designed to comply with industry-standard video standards such as high-definition HD, ultra-high-definition UHD, etc. These standards require strict adherence and establish a set number of vertical and horizontal pixels in each vertical and horizontal line of a video frame, specify vertical and horizontal blanking periods that contain "pixels" that are not intended to be displayed, and specify transmission rate standards.

[0008] In a raster-scan display, the vertical blanking interval (VBI) is the time from the end of the last visible line of a frame or field to the beginning of the next visible line of the next frame. Modern thin-panel displays, such as liquid crystal displays and organic light-emitting diode (OLED) displays, do not require blanking intervals; however, for legacy reasons, displays are configured to receive video streams with blanking intervals that contain multiple blank pixels. The terms "blank pixel" or "blanking interval" are known in the art to include data that is not intended to be displayed. In other words, a "blanking interval" or "blank pixel" does not necessarily carry any information.

[0009] Video standards also set stringent requirements for, for example, transmission and clock synchronization timing, which further requires that the LCD panel receiver clock be synchronized to the video source clock to ensure proper display of the image. When the source video (e.g., a video camera output video signal) is transmitted using a common standard such as the Serial Digital Interface (SDI) standard, the SDI signal is configured to transfer the video source clock from the video source to the LCD panel for synchronization.

[0010] Each SDI cable can only carry the clock of a single video source, so problems exist when a single SDI cable is used to transmit video from two or more asynchronous video sources, which can be particularly demanding when creating video images that provide perspective or depth, for example.

[0011] Current solutions include an intermediate video frame buffer that separates the input video clock from each video source and synchronizes the SDI transmission to a single clock independent of either source. However, adding an intermediate video frame buffer requires hardware reconfiguration and can increase latency dimensions between one and two frames in the video output on an LCD panel display. Such latency levels are undesirable when the video source is a live stream intended for interactive control, e.g., surgery. When video from a single source is displayed routed through an intermediate hub for onward transmission, similar latency issues can arise due to potential buffer overflow issues in the hub. Summary of the Invention [Problem to be solved by the invention]

[0012] The disclosed technology is intended to mitigate, avoid, reduce or eliminate various problems known in the art that affect the transmission of one or more video streams to, for example, the LCD or LCD-type displays described above. [Means for solving the problem]

[0013] While the present invention is defined by the appended claims, various aspects of the disclosed technology, including the claimed technology, are presented in this summary section along with examples of some preferred embodiments and indications of possible technical advantages.

[0014] a memory configured to buffer the received serialized video data until at least all of the effective pixels of a frame line of pixels of the received video frames have been buffered; one or more processors or processing circuits configured to form, for each effective pixel group of a frame line of pixels, a data structure having at least the effective pixels of that frame line and a frame line position indicator assigned to that effective pixel group, wherein each data structure includes the effective pixels from a received frame line and excludes at least one blank pixel of the received frame line; and a transmitter configured to intermittently forward transmit the formed data structures by pausing transmission after each data structure is transmitted until the next data structure is ready to begin transmission.

[0015] Advantageously, in some embodiments, instead of buffering the received data stream at the device until an entire video frame is received, the receiver buffer is configured to empty after each complete video frame line is received without waiting for an entire video frame to be received, thereby reducing latency delays.

[0016] In some embodiments, from each received video frame line that includes valid pixels and at least one blank pixel, two or more blank pixels are discarded.

[0017] In some embodiments, any blank pixels in the received video frame lines are not included in the output serialized video stream that contains the forward transmitted data structure.

[0018] Advantageously, by removing one or more blank pixels or all blank pixels from each video frame line, the number of blank pixels available for onward transmission is reduced, thereby reducing the transmission time for each video frame line to reach the viewing device.

[0019] In some embodiments, the intermittently transmitted video data stream has a fixed frame line length with variable transmission pause periods.

[0020] Advantageously, by pausing transmission between pixel groups in the device, any transmit clock used to serialize video frames in the video source can be decoupled from any receive clock used in the observation device.

[0021] Advantageously, in the viewing device, the variable transmission pause periods can be compensated for by adding a variable number of blank pixels to the continuous video data stream of variable line lengths for output to the viewing device.

[0022] In some embodiments, the variable number of blank pixels depends on the transmission period of each data structure transmitted forward to the observation device.

[0023] In some embodiments, the frame line indicator assigned to each effective pixel group is included in the header of the data structure containing the effective pixel group.

[0024] Another aspect of the second disclosed technology includes an observation device comprising: a receiver configured to receive an intermittently transmitted serialized video stream including a plurality of data structures, each data structure including at least a group of valid pixels from a video frame line and a frame line position indicator assigned to the group of valid pixels; and one or more processors or processing circuits configured to: for each received data structure, form a complete video data line by adding one or more blank pixels to each group of valid pixels of the received data structure until a data structure including a next group of pixels in the serialized video stream is received; and output each complete video data line so that it is presented on a display at a position within the video frame indicated by the frame line position indicator assigned to the effective pixels of the complete video data line.

[0025] In some embodiments, a serialized video stream is received from an apparatus that processes a serialized video data stream for onward transmission according to the first aspect disclosed herein or any one of its embodiments, and the intermittently transmitted and received serialized video data stream is processed by one or more processors or processing circuits to add a variable number of blank pixels for output to a display as a continuous serialized video stream of variable line lengths.

[0026] In some embodiments, the viewing device further comprises a display.

[0027] In some embodiments, the viewing device may include two or more displays and may split the received video stream into two or more streams, one or more of which may be external to the viewing device. For example, in some embodiments, a large flat panel display may be used to display the same image as is being displayed on the near-eye display, for example, if the large screen can be driven in a similar manner to the near-eye display.

[0028] In some embodiments, the observation device comprises an apparatus for processing the serialized video stream for onward transmission according to the first aspect disclosed herein or any one of its embodiments.

[0029] Advantageously, reducing the number of blank pixels transmitted can reduce the likelihood of buffer overflow in the receiving viewing device.

[0030] Another aspect of a third disclosed technique includes a method of processing a serialized video stream for forward transmission, the method including: receiving a serialized video stream including a plurality of video frames transmitted serially from a video source, each video frame including a plurality of frames, each frame including a plurality of frame lines including effective pixels and blank pixels; buffering at least effective pixels of the received serialized video stream until a complete frame line of pixels is received; processing each buffered complete frame line of pixels, assigning a frame line position indicator to effective pixels including effective pixels of the buffered complete frame line; forming data structures including effective pixels including the frame line position indicator assigned to the effective pixels, each data structure including effective pixels from a received frame line and excluding at least one blank pixel of the received frame line; when each data structure is ready for transmission, intermittently forward transmitting output serialized video data including the formed data structures; and pausing the forward transmission of the output serialized video data between the data structures.

[0031] For example, in some embodiments, the forward transmission of output serialized video data is paused after a data structure containing a group of valid pixels and an assigned frame line position indicator is transmitted until the next data structure containing a different group of valid pixels with a different frame line position indicator is ready to be transmitted.

[0032] In some embodiments, each data structure containing valid pixels from a received frame line excludes all blank pixels of that received video frame line in the output serialized video stream containing the forward transmitted data structure.

[0033] In some embodiments, the intermittently forward transmitted video data stream has a fixed frame line length and variable transmission pause periods.

[0034] Another fourth aspect of the disclosed technology includes a method of processing a received intermittent serialized video stream for display, the method including: receiving a received serialized video stream including a plurality of data structures, each data structure including at least valid pixels from a video frame line and a frame line position indicator assigned to the valid pixels; forming a complete video data line by adding one or more blank pixels to each valid pixel of the received data structure until a data structure including the next valid pixel group in the serialized video stream is received; and outputting each complete video data line to be presented on a display at a position within the video frame indicated by the frame line position indicator assigned to the valid pixels of the complete video data line.

[0035] In some embodiments, the intermittent serialized video stream is received from an apparatus that processes the serialized video stream for onward transmission according to the first aspect or any embodiment disclosed herein.

[0036] In some embodiments, the intermittently transmitted and received serialized video data is processed by one or more processors or processing circuits (1002) to add a variable number of blank pixels for output to a display as a continuous serialized video stream of variable line lengths.

[0037] In some embodiments, the method further includes determining that a final frame line of the frame has been received, forming a frame line including blank pixels as the final frame line of the frame, and outputting the frame line of the frame as a serial data stream of pixels for display.

[0038] Another fifth aspect of the disclosed technology relates to a computer program product including computer code that, when loaded from a memory and executed by one or more processors of an apparatus for processing a serialized video stream for onward transmission according to any one of the aspects or embodiments disclosed herein, causes the apparatus to perform a method for processing a serialized video stream for onward transmission according to any one of the disclosed aspects or embodiments.

[0039] Another sixth aspect of the disclosed technology relates to a computer program product comprising computer code that, when loaded from a memory and executed by one or more processors of an observation device according to any one of the aspects or embodiments disclosed herein, causes the observation device to perform a method for processing a received video stream according to any one of the disclosed aspects or embodiments.

[0040] Another seventh aspect of the disclosed technology includes a signal formed by any one of the disclosed aspects or embodiments of an apparatus for processing a serialized video stream for onward transmission, the signal including a serialized video stream of data structures, each data structure including at least a group of valid pixels from a video frame line and a frame line position indicator assigned to the group of valid pixels.

[0041] In some embodiments, the output serialized video stream containing the forwarded data structure does not contain any blank pixels.

[0042] In some embodiments, the output serialized video stream containing the forward transmitted data structures includes at least one blank pixel in each data structure that is less than the number of blank pixels in the received video frame line containing the valid pixels of that data structure.

[0043] Another eighth aspect of the disclosed technology relates to a video display system, the video display system including an apparatus for processing a serialized video stream for onward transmission according to any one of the aspects or embodiments disclosed herein, and an observation apparatus according to any one of the aspects or embodiments disclosed herein.

[0044] In some embodiments, the apparatus for processing the serialized video stream for onward transmission further includes a video source.

[0045] In some embodiments, the observation device further includes at least one display configured to present video received from the video source via the device.

[0046] Another ninth aspect of the disclosed technology relates to a computer-readable storage medium comprising computer program code that, when executed by one or more processors or processing circuits of an apparatus for processing a serialized video stream for onward transmission according to any one of the aspects or embodiments disclosed herein, causes the apparatus to perform a method for processing a serialized video stream for onward transmission according to any one of the aspects or embodiments disclosed herein.

[0047] Another tenth aspect of the disclosed technology relates to a computer program carrier comprising computer program code that, when loaded from the computer program carrier and executed by a processor or processing circuitry of an apparatus for processing a serialized video stream for onward transmission according to any one of the aspects or embodiments disclosed herein, causes the apparatus to perform a method for processing a serialized video stream for onward transmission according to any one of the aspects or embodiments disclosed herein, wherein the computer program carrier is one of an electronic signal, an optical signal, a wireless signal, or a computer-readable storage medium.

[0048] Another aspect of the disclosed technology includes a means for carrying out a method according to any one of the method aspects disclosed above.

[0049] Another eleventh aspect of the disclosed technology includes an apparatus for combining multiple serialized video streams from different video sources for serialized forward transmission over a channel, the apparatus including: a receiver configured to separately receive each separate channel of the serialized video streams; a plurality of buffers configured to buffer at least valid pixels of received frame lines of serialized frames in each received video stream; one or more processors or processing units configured to: for valid pixels in each received complete pixel line in a received frame of the received serialized video stream, assign the pixels a video source indicator and form a data structure including the valid pixels, excluding at least one blank pixel of the received complete frame line, and the data structure including the source indicator assigned to the pixels and a frame line position indicator representing the valid pixels; and a transmitter configured to forward transmit the formed data structures for the two received serialized video streams as serialized video data over a common channel.

[0050] According to some embodiments, the video device includes a hub co-located with the surgical imaging device.

[0051] In some embodiments including a hub co-located with the surgical imaging device, the hub is configured to receive serialized video streams from two or more video sources, for example, a left view video source and a right view video source.

[0052] According to some embodiments, each video source includes a camera, and each camera is part of a surgical imaging device.

[0053] According to some embodiments, the hub is or is part of a surgical imaging device that includes a surgical microscope.

[0054] According to some embodiments, the hub is disposed with a surgical microscope.

[0055] In some embodiments, the hub is part of a control system or is provided as a separate computing component or unit of the surgical microscope.

[0056] In some embodiments, the method further includes assigning a frame line position indicator to each group of valid pixels prior to forward transmission, although in alternative embodiments, the method may instead include maintaining a frame line position indicator associated with the group of valid pixels.

[0057] In some embodiments, after a data structure from one video source has been transmitted, the transmitter pauses transmission until it is ready to begin transmitting the next data structure from another video source or from the same video source.

[0058] In some embodiments, the received video data stream has a fixed line length, and the transmission pause lengths between forward transmitted data structures vary.

[0059] In some embodiments, the transmitter forwards and transmits the data structure formed from the multiple sources as a continuous stream of combined video data.

[0060] Advantageously, in some embodiments, instead of buffering the incoming data stream in the device until a complete video frame is received from one video source, the latency delay can be reduced, and the receiver buffer is configured to empty after each complete video frame line is received, without waiting for the complete video frame to be received.

[0061] Advantageously, by pausing transmission between groups of pixels in the device, any transmit clock used to serialize the pixels that form the video frame data in the video source can be decoupled from any receive clock used to extract the pixel data in the observation device.

[0062] Advantageously, separating the clocks at the video source and the observation device reduces transmission delays for multiple serialized video streams. Furthermore, receiver circuitry can be simplified because a phase-locked loop does not need to be used to derive the same frequency as the source. This allows for better synchronized presentation of multiple serialized video streams when displayed simultaneously.

[0063] In some embodiments, the video source indicator identifies a spatial relationship between at least two of the multiple video sources.

[0064] In some embodiments, the multiple serialized video streams include a video stream obtained from a video source for a left view of the scene and a video stream obtained from a video source for a right view of the scene, and the video source indicator identifies whether the group of pixels belongs to a video frame obtained from the video source for the left view of the scene or a video frame obtained from the video source for the right view of the scene.

[0065] In some embodiments, the assigned video source indicator and frame line position indicator are included in a header of the data structure, where the bit size of the header is adapted to the parallel width of a serializer of a transmitter configured to combine data structures from multiple different video sources into one output serialized video stream.

[0066] In some embodiments, the data structures from the two sources are interleaved by the serializer in the output serialized video stream.

[0067] In some embodiments, the transmitter is configured to pause forward transmission between data structures after a data structure including a group of valid pixels having a first video source indicator has been transmitted until a data structure having the same or a different video source indicator is ready to be transmitted.

[0068] In some embodiments, interleaving causes data structures from one video source to alternate with data structures from another video source in the output serialized video stream, although this is not always the case; one or more frame lines may be sent consecutively from one source if no other frame lines are available for transmission.

[0069] In some embodiments, at least one received serialized video stream is received on a channel having one or more of a different data rate and a different video resolution relative to the data rate or video resolution of at least one other channel on which another received video stream is received.

[0070] In some embodiments, none of the blank pixels of the received video frame lines are included in the output serialized video stream that contains the forward transmitted data structure.

[0071] However, in some embodiments, only one or more blank pixels are removed, and any remaining blank pixels of the received video frame line are included in the forward transmitted data structure of the output serialized video stream.

[0072] Advantageously, by removing one or more or all blank pixels from each video frame line, the number of blank pixels available for onward transmission is reduced, thereby reducing the transmission time for each video frame line to reach the observation device. Furthermore, the probability of buffer overflow at the observation device's receiver can also be reduced because an entire video frame does not need to be buffered; instead, only one entire video frame line is buffered at the transmitter before being output. This reduces the probability of buffer overflow at the combiner and observation device in addition to the absence or reduction of blank pixels in the received transmission stream and / or when there is only intermittent transmission of data structures.

[0073] In some embodiments, the video combining device can separate the source clock of the received serialized video stream in the output serialized video stream by varying the pause period between forward transmitted data structures or by varying the transmission period of the forward transmitted data structures.

[0074] In some embodiments, the transmitter is configured to transmit the output serialized video stream over a data communication channel along a wired link and / or over a wireless data communication channel to a device configured to display the received video data.

[0075] In some embodiments, for example in the case of a headset display, the weight of the viewing device can be reduced by using a single wired link with a video combining device that carries serialized video data from multiple serialized video sources, thereby reducing the burden of wearing the headset.

[0076] Another twelfth aspect of the disclosed technology relates to an observation device comprising: a receiver configured to receive a serialized video data stream including a plurality of data structures, each data structure including a group of effective pixels, a source indicator assigned to the group of effective pixels, and a frame line position indicator assigned to the group of effective pixels; and one or more processors or processing circuits configured to form complete video data lines by adding one or more blank pixels to each group of effective pixels of the received data structures until a data structure including the next group of pixels in the serialized video stream is received, and to output each complete video data line to be presented on a display in association with the source indicator for that complete video line at a position within a video frame indicated by the frame line position indicator assigned to the effective pixels of that complete video data line.

[0077] In some embodiments, multiple complete frame lines of video data output to one or more displays include video data of variable line lengths, in other words, the video data output to the displays does not have a fixed frame line length of active and blank pixels because the number of blank pixels can vary.

[0078] Advantageously, variable frame line length video data allows data output to one or more displays to be a continuous stream of pixels and blank pixels.

[0079] In some embodiments, the one or more displays include a flat panel display, such as a display based on LCD or OLED technology, configured to receive serialized video data having a raster display format.

[0080] In some embodiments, the received serialized video stream is received from an apparatus that combines multiple serialized video streams, each originating from a different video source, for serialized forward transmission over a single channel according to the eleventh aspect disclosed herein or any embodiment thereof.

[0081] In some embodiments, the video combining device further includes a video source.

[0082] In some embodiments, the observation device further includes at least one display configured to present video received from the video source via the device.

[0083] Another thirteenth aspect of the disclosed technology includes a method for combining multiple serialized video streams from different video sources received over different communication channels for serialized forward transmission over one channel, the method including: separately receiving each one of the multiple serialized video streams; separately buffering at least effective pixels of a received frame line of a serialized frame of each video stream of the received multiple serialized video streams; processing each buffered effective pixel group within the received complete frame line by assigning at least a video source indicator to the effective pixel group and forming a data structure including the effective pixel group, an indicator for the video source of the effective pixel group, and a frame line position indicator for the effective pixel group; and forward transmitting over a common channel the formed data structure including the buffered effective pixels from the frame line of a video frame of one video stream of the received multiple serialized video streams.

[0084] In some embodiments, the intermittently transmitted and received serialized video data is processed by one or more processors or processing circuits of the observation device so that it is appended with a variable number of blank pixels for output to a display as a continuous serialized video stream of variable line lengths.

[0085] In some embodiments, the method further includes, after each data structure is transmitted, determining whether another data structure from a video frame line of a video frame originating from another one of the multiple video sources is ready to begin forward transmission, and if ready, forward transmitting the data structure from the other one of the multiple video sources over the common channel; or determining whether another data structure from one of the multiple video sources is ready to be transmitted, and if ready, forward transmitting the data structure from the one of the multiple video sources over the common channel.

[0086] In some embodiments, after the data structure has been transmitted, the method further includes determining that no data structure from any video source of the plurality of video sources is ready for transmission, pausing transmission until a data structure from a video source of the plurality of video sources is ready for transmission, and forward transmitting the data structure over the common channel.

[0087] In some embodiments, the method includes intermittently forward transmitting a data structure formed from each of the received serialized video streams as a single serialized video stream, and pausing the forward transmission between data structures after a data structure including a group of valid pixels and an assigned frame line position indicator is transmitted until the next data structure including a different group of valid pixels with a different frame line position indicator is ready to be transmitted.

[0088] In some embodiments, the intermittently transmitted combined video data stream has a fixed frame line length with variable transmission pause periods.

[0089] Another fourteenth aspect of the disclosed technology includes a method for viewing multiple serialized video streams received as a single serialized video stream, the method including: receiving a serialized video stream having multiple data structures, each including a group of effective pixels, a source indicator assigned to the group of pixels, and a frame line position indicator assigned to the group of effective pixels; processing the received serialized video stream by adding one or more blank pixels to each group of effective pixels of the received data structures to form a complete video data line until a data structure including the next group of pixels in the serialized video stream is received; and outputting each complete video data line for presentation on a display associated with the source indicator of the complete video line at a position within a video frame indicated by the frame line position indicator assigned to the effective pixels of the complete video data line.

[0090] In some embodiments, the method further includes displaying each complete line of video data on a display.

[0091] In some embodiments, the serialized video stream data that is intermittently transmitted and received may be video stream data that is intermittently transmitted by an aspect of the combining device disclosed in this specification or an embodiment thereof.

[0092] In some embodiments of the method, the received serialized video stream (622) is a video stream output by a combining device according to any one of the aspects disclosed herein or embodiments thereof.

[0093] Another fifteenth aspect of the disclosed technology relates to a video display system having a video combining device according to the eleventh aspect or any one of the embodiments disclosed herein and an observation device according to the twelfth aspect or any one of the embodiments disclosed herein.

[0094] In some embodiments, the video combining device further includes a video source.

[0095] In some embodiments, the observation device further includes at least one display configured to present video received from the video source via the device.

[0096] A sixteenth aspect of the disclosed technology relates to a computer program product comprising computer code that, when loaded from a memory and executed by one or more processors of an apparatus according to the eleventh aspect or any one of its embodiments disclosed herein, causes the apparatus to perform a method according to the thirteenth aspect.

[0097] A seventeenth aspect of the disclosed technology includes a computer program product comprising computer code that, when loaded from a memory and executed by one or more processors of an observation device according to the twelfth aspect or any one of its embodiments disclosed herein, causes the observation device to perform a method according to the fourteenth aspect or any one of its embodiments.

[0098] Another aspect of the disclosed technology includes a signal formed by an apparatus for combing a video stream according to the eleventh aspect disclosed herein or any one of its embodiments, the signal including a serialized video stream of data structures, each data structure including at least a group of valid pixels from a video frame line and a frame line position indicator assigned to the group of valid pixels.

[0099] In some embodiments, the output serialized video stream containing the forwarded data structure does not contain any blank pixels.

[0100] In some embodiments, the output serialized video stream containing the forward transmitted data structures includes at least one blank pixel in each data structure that is less than the number of blank pixels in the received video frame line containing the valid pixels of that data structure.

[0101] Another aspect of the disclosed technology includes a method of viewing one or more serialized video streams received as a single serialized video stream, the method including: receiving a first serialized video stream having multiple data structures, each data structure including a group of effective pixels, a source indicator assigned to the group of pixels, and a frame line position indicator assigned to the group of effective pixels; processing the received serialized video stream by adding one or more blank pixels to each group of effective pixels of the received data structures to form a complete video data line until a data structure including a next group of pixels in the serialized video stream is received; and outputting each complete video data line for presentation on at least one of the at least one display associated with the source indicator of the complete video line at a position within a video frame indicated by the frame line position indicator assigned to the effective pixels of the complete video data line.

[0102] In some embodiments, the method further includes displaying each complete line of video data on a display.

[0103] In some embodiments of the method, the received serialized video stream is a discontinuous video stream output by a combining device according to any one of the aspects or embodiments of the combining device disclosed herein.

[0104] Another aspect of the disclosed technology includes a means for carrying out a method according to any one of the disclosed method aspects.

[0105] Another aspect of the disclosed technology includes a computer program product comprising computer code that, when loaded from a memory and executed by one or more processors of a video combining device according to the eleventh aspect disclosed herein or any one of its embodiments, causes the video combining device to perform a method according to the thirteenth aspect disclosed herein or any one of its embodiments.

[0106] Another aspect of the disclosed technology includes a computer program product comprising computer code that, when loaded from a memory and executed by one or more processors of an observation device according to the twelfth aspect disclosed herein or any one of its embodiments, causes the observation device to perform a method for processing a received video stream according to any one of the disclosed aspects or embodiments.

[0107] Another aspect of the disclosed technology includes a signal formed by an apparatus according to the eleventh aspect disclosed herein or any one of its embodiments, the signal including a serialized video stream of data structures, each data structure including at least a group of valid pixels from a video frame line and a frame line position indicator assigned to the group of valid pixels.

[0108] In some embodiments, the output serialized video stream containing the forwarded data structure does not contain any blank pixels.

[0109] In some embodiments, the output serialized video stream containing the forward transmitted data structures includes at least one blank pixel in each data structure that is less than the number of blank pixels in the received video frame line containing the valid pixels of that data structure.

[0110] Another aspect of the disclosed technology includes a video display system, the video display system including an apparatus according to the eleventh aspect or any one of the embodiments disclosed herein and an observation apparatus according to the twelfth aspect or any one of the embodiments disclosed herein.

[0111] In some embodiments, the apparatus for processing the serialized video stream for onward transmission further includes a video source.

[0112] In some embodiments, the observation device further includes at least one display configured to present video received from the video source via the device.

[0113] In some embodiments, the video display system further includes a plurality of video sources, each configured to provide serialized video data to the device according to the eleventh embodiment.

[0114] Another aspect of the disclosed technology includes a computer-readable storage medium comprising computer program code that, when executed by one or more processors or processing circuits of a video combining device according to any one of the aspects or embodiments disclosed herein, causes the video combining device to perform a method according to the thirteenth aspect or any one of the embodiments disclosed herein.

[0115] Another tenth aspect of the disclosed technology is a computer program carrier comprising computer program code that, when loaded from the computer program carrier and executed by one or more processors or processing circuits of a video combining device according to the eleventh aspect disclosed herein or any one of its embodiments, causes the video combining device to perform a method according to the thirteenth aspect disclosed herein or any one of its embodiments, wherein the computer program carrier comprises a computer program carrier that is one of an electronic signal, an optical signal, a wireless signal, or a computer-readable storage medium.

[0116] The disclosed aspects and embodiments may be combined with each other in any suitable manner apparent to those skilled in the art.

[0117] Some embodiments of the disclosed technology will now be described, by way of example only, with reference to the accompanying drawings in which: [Brief explanation of the drawings]

[0118] [Figure 1A] FIG. 10 is a diagram illustrating schematically how latency can be created when there are two video sources each outputting a serialized video stream to an intermediate device configured to buffer the received video stream frame by frame before forwarding the combined video stream to two remote displays according to the prior art. [Figure 1B] FIG. 10 is a diagram illustrating schematically how latency can be created when there are two video sources each outputting a serialized video stream to an intermediate device configured to buffer the received video stream line by line before forwarding the combined video stream to two remote displays according to the disclosed technology. [Figure 1C]FIG. 10 is a diagram illustrating schematically how latency can be created when there are two video sources each outputting a serialized video stream to an intermediate device configured to buffer the received video stream line by line before forwarding the combined video stream to two remote displays according to the disclosed technology. [Figure 1D] FIG. 10 is a diagram illustrating schematically how latency can be created when there are two video sources each outputting a serialized video stream to an intermediate device configured to buffer the received video stream line by line before forwarding the combined video stream to two remote displays according to the disclosed technology. [Figure 2A] FIG. 1 illustrates a schematic example of a serial video frame transmission; [Figure 2B] FIG. 1 illustrates schematically an example of how serially transmitted video frames can be displayed by an LCD display panel. [Figure 3A] 1A and 1B are schematic diagrams illustrating example apparatuses consistent with some embodiments of the disclosed technology. [Figure 3B] 1A-1C are diagrams illustrating an example of a method according to some embodiments of the disclosed technology. [Figure 4] 1A and 1B are diagrams illustrating examples of serialized transmission of variable video frame lengths from a single video source, in accordance with some embodiments of the disclosed technology; [Figure 5] 3B is a diagram illustrating schematically how an apparatus according to FIG. 3A forms a data structure header, in accordance with some embodiments of the disclosed technology. [Figure 6] 1A and 1B are diagrams illustrating schematic diagrams of exemplary observation devices according to some embodiments of the disclosed technology. [Figure 7] 1A-1C are diagrams that schematically illustrate methods performed by observation devices in accordance with some embodiments of the disclosed technology. [Figure 8] FIG. 1 is a diagram that schematically illustrates a system for displaying video from two video sources, in accordance with some embodiments of the disclosed technology. [Figure 9] 1A and 1B are diagrams illustrating examples of serialized transmission of variable video frame lengths from multiple video sources to multiple displays, in accordance with some embodiments of the disclosed technology; [Figure 10] FIG. 10 is a diagram illustrating a schematic example of another method according to some embodiments of the disclosed technology. [Figure 11] 10A-10C are diagrams illustrating examples of how an apparatus according to some embodiments of the disclosed technology forms a data structure header. [Figure 12] FIG. 10 is a diagram that schematically illustrates another example of an observation device that processes received video data, in accordance with some embodiments of the disclosed technology. DETAILED DESCRIPTION OF THE INVENTION

[0119] Aspects of the present disclosure will be described in more detail below with reference to the accompanying drawings. However, the apparatus and method disclosed herein can be embodied in many different forms and should not be construed as limited to the aspects set forth herein. Steps, whether explicitly or implicitly mentioned, may be reordered or omitted unless essential to some of the disclosed embodiments. Like numbers in the figures refer to like elements throughout, but may have the same or different numbers depending on which page of the drawings the same or different embodiments first appear on.

[0120] The terms used herein are for the purpose of describing particular aspects of the present disclosure only and are not intended to limit the embodiments of the disclosed technology described herein. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise.

[0121] 1A shows a schematic of how latency can be achieved when two video sources each output a serialized video stream to an intermediate device that conforms to the video standard transmission requirements of the serial data interface SDI and is configured to buffer the received video stream frames frame by frame before forward transmitting the combined video stream to two remote displays. In FIG. 1A, two asynchronous video sources #1 and #2 of serialized video data streams each output data over separate 3G serialized video links to a combining device that is configured to merge the two streams into one output serialized video stream for forward transmission according to the SDI video standard over a 6 Gbit / s link to an observation device that includes two displays.

[0122] In Figure 1A, two video sources #1 and #2, labeled 102a and 102b, are both configured to generate video of a scene, in this example, both configured to capture a view of a pendulum. As shown in Figure 1A, video source #1, 102a, outputs an image of the pendulum, which is transmitted to display 106a by a video combiner or hub 108, which combines video from video source #1 with another pendulum video from video source #2 over a serialized data interface (SDI) link 104.

[0123] As shown in Figure 1A, a video frame received from video source #1 is shown stored in a buffer (Buf1 at the top of Figure 1A) in hub 108. However, because video source #2, 102b, is running at a slightly lower clock speed for transmitting video or because link 104b is slower, the transmission of another pendulum frame arrives at hub 108 slightly late. Because the hub is configured to wait until two frames have been completely received from the two sources before forwarding any frame, the result shown in Figure 1A is that every 1.5 frames from video source #1 are buffered in the top row of buffers Buf1 and Buf2 in hub 108 before the bottom buffer Buf1 holds the entire image of the same pendulum from video source #2.

[0124] Because the combiner box cannot lock to the clock of either source, and both image frames must be completely received at the hub 108 before they can be forwarded line-by-line to the corresponding display, buffers at the hub 108 accumulate at different rates to compensate for the asynchronous transmission differences between the two video sources before the video can be forwarded, which increases latency at the hub 108.

[0125] One technique for reducing latency is to configure Hub 108 to buffer the incoming video only line by line. Figures 1B-1C show a schematic of how such an approach can be used to reduce the latency created in Hub 108, in other words, how a buffer underflow problem can arise in Hub 108 if Hub 108 outputs video line by line from two asynchronous video sources.

[0126] 1B-1D show a series of schematic examples of what happens at hub 108 when the hub device is configured not to wait for complete frames to be received from two sources before forwarding the frames line-by-line to displays 106a and 106b. In FIG. 1B-1D, hub 108 instead forwards the video received from two sources #1 and #2 line-by-line to displays 106a and 106b.

[0127] As shown in FIG. 1B, video streams received from two asynchronous video sources can remain fully synchronized, such that when a video frame from source #1A is received by the upper buffer Buf1 of hub 108, the first complete frame line from source #B is already present in the lower buffer Buf1 of the hub. FIG. 1C illustrates how line-by-line transmissions at hub 108 reduce the delay it takes for displays 106a and 106b to receive the first line of their corresponding video frames. However, as the two sources gradually become desynchronized, a situation exists in which only one line can be transmitted from one video source, here source #1. This is because, as shown in FIG. 1D, there is a buffer underflow at hub 108 for the video from the other video source (source #2). In other words, both lower buffers Buf1 and Buf2 in FIG. 1D are empty.

[0128] Therefore, to avoid the risk of buffer underflow, it is desirable to buffer the complete frame at hub 108 before transmitting any of the framelines to the remote display.

[0129] Some embodiments of the disclosed technology aim to provide an alternative solution that can reduce latency and reduce the risk of buffer underflow even when the hub 108 is configured to forward transmit received video data line by line. In some embodiments, similar latency reduction can be achieved even when receiving only one video stream from a single video source.

[0130] Each video frame line in the video source includes a number of effective pixels, where the effective pixels represent pixels to be reproduced on the remote display, and a number of blank pixels are added to form a continuous data stream. The blank pixels form a blanking period, a conventional practice in raster-scan or cathode ray tube (CRT) display types to return to the beginning of the physical display after the end of the previous line to begin the next line of the image, and add a line of blank pixels at the end of the last line of effective pixels in the frame to allow the CRT beam to physically move to the beginning of the top line of the next frame. This practice also remains for other types of displays, including flat-panel displays, including, for example, displays based on organic light-emitting diode (OLED) technology, liquid crystal display (LCD) technology, or optical coupling device technology. References herein to a display refer to a display configured to receive data from a video source that includes blank pixels. The observation device can include one or more displays and can also include one or more processors that process the received serialized video data stream for output to the display.

[0131] The term "blanking" as used herein means only "not intended to be displayed as part of a video image frame line," and does not necessarily mean that the pixels in the "blanking period" contain no data or information. In the prior art, each pixel in a received serialized video data stream, whether it is a valid pixel intended for display or a blank pixel not intended for display, is transmitted at the clock frequency of the video image source 102 along the data link 104 according to standard techniques.

[0132] When there is only one video source, buffer underflow or overflow in the hub 108 or any viewing device having one or more displays 106 is typically eliminated by design. However, at high data transmission rates, particularly when there are two sources, the likelihood of buffer underflow or overflow may increase if the two sources are asynchronous, as explained above. Therefore, if the latency delay resulting from frame-by-frame buffering in the hub 108 is unacceptable, it is advantageous for the hub 108 to be able to perform line-by-line buffering and forward transmit video line-by-line for display.

[0133] For example, a frame-buffered serialized video data transmission rate on the order of 10 milliseconds for transmitting video through the hub 108 can incur a latency on the order of 16 milliseconds from the initiation of image transfer at the source 102 to the remote display 106 until the transferred image is displayed on the remote display 106. Such a latency is excessive in use cases, for example, where a medical procedure is being performed using a camera that captures images of the body area undergoing the procedure, and where the surgeon or one or more assistants are wearing near-eye displays or headsets. Therefore, it would be desirable to have improvements in how a hub device or the like processes received serialized video stream data to reduce the introduction of additional latency or delay and further shorten, and preferably minimize, the time between when an image is captured and when it is displayed on the display 106.

[0134] Figure 2A shows in more detail how a fixed line-length video frame is transmitted over a standard link 104 of frame-by-frame transmission, e.g., a serial data interface (SDI) communication link, e.g., a wired link, from a video source 12, such as a video camera, to a remote display 106. Figure 2A shows the active pixels of a video frame 100 schematically as boxes with numbers, and shows null pixels as blank or blank boxes. Figure 2A also shows schematically how, in addition to including multiple lines, each frame terminates with a blank or blanking video line 202e, where only four of the video lines 202a-202d are shown schematically in Figure 2A by way of example.

[0135] Each video frame 204 contains a fixed number y horizontal rows of pixels and a fixed number x vertical columns of pixels, which are typically set according to a particular video standard. For illustrative purposes only, as shown in Figure 2A, y = 5 and x = 6. This means that in the example shown in Figure 2A, each source video frame 204 contains 30 pixels, organized into five rows of six pixels each, with four valid pixels and two blank pixels in each row, and each vertical column contains five pixels, four of which are valid.

[0136] As shown in Figure 2A, each video frame in the serialized video stream from video source 102 over link 104 has one vertical blanking period and two horizontal banking intervals. The lower section of Figure 2A shows how each group of pixels, including active and blank pixels, forms a horizontal line of video frame 100 and how video frame 100 is serially transmitted horizontally frame line by horizontal frame line as a continuous data stream over SDI transport link 104 to remote display 106. The time to serialize each video frame containing 30 pixels is determined by the clock cycles required to transmit each pixel multiplied by the number of pixels in each video frame, which in this example is 30x clock cycles.

[0137] 2A and 2B show in more detail how the top line 202a is transmitted first, with the active pixels transmitted before any blank pixels, then the next line 202b, the third line 202c, and the next line 202d, where the last line 202e contains only blank pixels that are transmitted last. After the blank pixel line 202e is transmitted, transmission of the next image frame begins with the first horizontal line of the current frame, then the second, third, fourth, and finally the blanking line of the next video frame, and the process is repeated as video data is generated by the video source 202.

[0138] As mentioned above, each pixel takes a certain amount of time to transmit, determined by the clock period used by the video source 102 to transmit the video. If the video source clock frequency is one pixel per clock period, and has a clock period of 1 μs, by way of example only, each pixel will take 1 μs to be ready for transmission. Because each horizontal line 202a-202e consists of six pixels, each line takes 6 μs to be ready for transmission. Because the video frame 204 has five lines 202a-202e, the total serialization time to transmit the entire video frame 204 along the SDI link 104 is 30 μs.

[0139] When a video stream from a video source 102 is sent over an SDI transport link 104 to a display 106, such as an LCD panel display, the LCD panel at the remote display 106 must synchronize the start of each received line 202a-202d in order to properly reconstruct the frame 204 for display. Metadata is provided in the pixels of each video frame line, providing information that enables the display 106 to properly present the image carried by the serialized video frame data.

[0140] FIG. 2B shows a schematic example of how the serially transmitted video frames shown in FIG. 2A are processed for more detailed display on the display 106. In the example shown in FIG. 2B, a video source 102 of a serialized video data stream provides a fixed-length sequence of lines. The received video stream is processed in a receiver 202 of the display 106, which receives and forwards each data line for display on a screen 204, such as an LCD panel 204. The clock period of the video source 102 is used to synchronize pixel columns as they are displayed. This synchronization requires that the clock in the display 106 be locked in frequency to the video source clock used to transmit pixel data over the SDI link 104. For example, if the source clock period is 1 μs, then the display video clock frequency must have a clock period based on the source clock period. Locking the display clock frequency to the source clock frequency ensures that the first column of pixels in each frame, such as frame 204, is vertically aligned when reconstructed on LCD panel 204, as shown schematically in Figure 2B. If the timing of the two clocks is incorrect, buffer overflow or underflow can occur, depending on the extent to which the two clocks are out of synchronization with each other.

[0141] 2A and 2B, the source clock cannot be separated from the display clock in order to properly display captured images on display 106. This imposes various constraints on the design of a display system with at least one video source and at least one video display. First, only one source clock frequency is physically transported using the serialized video data stream sent over each SDI link 14, limiting the ability to feed each video data stream with video captured by a single video source 102.

[0142] One example of how this can create problems occurs when it is desirable to reduce the number of cables, for example, when the hub 108 is attached to a headset-type viewing device including one or more displays 106 to reduce the level of viewing device clutter. Second, for example, any processing that adds a video overlay to the video stream received at the intermediate device or hub, or any processing that performs a similar effect, such as image enhancement that results in two video streams, cannot be transmitted from the hub 108 to the display 106 using the same video link. Furthermore, transmission delays, even in the case of line-by-line transmission, can mean that the latency of displaying real-time streamed video can create problems in some embodiments that rely on real-time streamed video to perform actions. One example of such a use case is video surgery, where a surgeon or their assistant uses a headset including a near-eye display to receive an enhanced view of the person or animal undergoing surgery.

[0143] It should be noted that such use cases can benefit from a stereoscopic display in which the presented images provide depth perception. In an exemplary embodiment herein, the images can be reconstructed on one or two near-eye displays of a headset-type viewing device. Each display can receive a video stream from a separate source, and one or both of the video streams can undergo additional processing in an intermediate device to add image enhancements, such as color overlays. Undesirable latency, jitter, and different frame rates can cause the displayed images to lose synchronization, even when the images are acquired from two or more synchronized video sources.

[0144] Generally, if two video sources need to synchronize their images for remote viewing, two separate feeds must be provided, and additional mechanisms may be required to ensure that each reconstructed video frame is properly aligned, which can add additional delay. For example, if two or more video sources are provided to a viewing device that includes displays in a near-eye display headset, e.g., a left-eye display and a right-eye display, the video sources cannot share a single SDI cable because the clock frequencies of these video sources are never exactly equal, meaning the images displayed on each display would not be synchronized. However, providing two separate cables on such a device is undesirable due to the resulting headset complexity and extra weight. Figure 6, described in more detail below, illustrates an exemplary embodiment in which video is provided from two or more video sources.

[0145] 3A is a schematic diagram of an example of a device 302 for processing a serialized video stream. In some embodiments, the device 302 may include a hub co-located with the surgical imaging device. In some embodiments in which the device 302 includes a hub co-located with the surgical imaging device and is configured to receive the serialized video stream from one or more video sources, the video sources may include one or more cameras that are part of the surgical imaging device. According to some embodiments, the device 302 is or is part of a surgical imaging device that includes a surgical microscope. In some embodiments, the hub may be co-located with the surgical microscope and / or may be part of a control system or other computing component or unit of the microscope.

[0146] The device 302 includes a receiver 304 configured to receive a serialized video stream from a single video source 101 via a serialized video data link 103. However, instead of forwardly transmitting a continuous video data stream that can be used to align the source clock and the display clock, the device 302 decouples the source clock and the display clock by forming a discontinuous stream of serialized video data in accordance with the disclosed techniques. The discontinuous stream of serialized video data includes at least valid pixels for each horizontal line of a video frame, followed by a transmission pause. In some embodiments, for each received video frame line that includes at least valid pixels, at least one blank pixel is omitted when forwarding the valid pixels. This reduces the transmission time of the video frame because there are fewer blank pixels to transmit. Alternatively, the transmitter can pause transmission until the next complete pixel line, including valid and blank pixels, is buffered, at which point, when the buffer is empty, the transmitter transmits the valid pixels and excludes at least one blank pixel. In some embodiments, one or more blank pixels are omitted. In some embodiments, the number of blank pixels transmitted or excluded is fixed for all video lines. After transmitting a data structure containing valid pixels from a received video frame line and any blank pixels from that received frame line that are not excluded from onward transmission, the transmitter is configured to wait until the next line of video frame data is ready to be transmitted. This causes a transmission pause and a discontinuous stream of serialized video frame data, transmitted intermittently over the serialized video transmission link, is sent to the observation device 410.By removing one or more blank pixels from the end of each horizontal line of a video frame and / or by removing or reducing the blank pixels that form the last blank pixel row in each video frame, intermittent transmission of serialized video data occurs, thereby allowing the clock of the remote viewing device to be decoupled from the clock of the video source 101.

[0147] 3A, the received serialized video frame data is buffered line by line by buffer 305 configured to discard one or more blank pixels in each video frame of variable line length and forward at least the valid pixels of each complete received video line as a group of pixels having a regular line length to transmitter 310. By removing at least one blank pixel from each line received by device 302 from source 101, the transmitter of device 302 decouples the forward transmitted data from the video source clock, thereby allowing the receiver clock to be reset accordingly.

[0148] Some embodiments of device 302 include a device that processes a serialized stream of video data for onward transmission, for example, to observation device 312. The serialized video data stream can include multiple video frames transmitted serially from a video source, each video frame including multiple frame lines, with at least one frame line including active pixels and one or more blank pixels.

[0149] The apparatus may include a receiver 304 configured to receive a serialized video data stream; a memory 306, which may include a memory 305 configured to buffer the received serialized video data until at least all effective pixels of a frame line of pixels of the received video frame have been buffered; one or more processors or processing circuits 308 configured to form, for each effective pixel group of a frame line of pixels, a data structure including at least the effective pixels of the frame line and a frame line position indicator assigned to that effective pixel group; and a transmitter 310 configured to intermittently forward transmit the formed data structure by pausing transmission after each data structure is transmitted until the next data structure is ready to begin transmission.

[0150] In some embodiments of the device 302 for processing a serialized video stream, the receiver 304 is configured to receive the serialized video stream 104, e.g., a video stream transmitted from the video source 102 over an SDI link. The serialized video stream 104 consists of multiple video frames transmitted serially from the video source 102, each video frame including multiple video frame lines. Each video frame line from the first to the last video frame line includes active pixels and blank pixels. The last frame line of each video frame includes blank pixels. The received video frame data can use a transmission link according to a video transmission standard, such as the Serialized Data Interface SDI transmission link 103, but can use a different transmission link 311 for onward transmission to the observation device 312.

[0151] The memory 306 can include various types of memory, such as memory storing data including computer program code and at least one buffer 305 configured to buffer at least valid pixels of the received serialized video stream 104 until a complete frame line of pixels is received from the video source 101. The one or more processors or processing circuits 308 of the device 302 are configured to include a frame line position indicator in the forwardly transmitted data for each complete frame line of buffered pixels. In some embodiments, the valid pixels and the frame line position indicator form a data structure. In other words, the forwardly transmitted data includes intermittently transmitted data structures, each data structure including the valid pixels and a frame line position indicator assigned to the valid pixels. Each data structure includes the valid pixels from a received frame line, and in some embodiments, excludes at least one blank pixel from the received frame line; however, in some embodiments, none of the blank pixels from the received video frame line are included in the output serialized video stream including the forwardly transmitted data structure.

[0152] In some embodiments, since the number of lines in a frame is known, the frame line position indicator can be configured to indicate whether the frame line is the first frame line of the frame or a subsequent frame line in the form of a binary indicator of "first line" or "not first line." However, in other embodiments, a specific line position within the frame can be indicated. In some embodiments, the frame line indicator can be received in the incoming video data stream from the source and carried in the forward transmitted output data, while in other embodiments, the frame line position indicator can be assigned by a processor or processing circuit. Thus, display of the first row of effective pixels in a received video frame can properly reconstruct the entire image on the display of the observation device 312.

[0153] In some embodiments, the frame line indicator assigned to each effective pixel group is included in a header or footer of a data structure containing that effective pixel group, and the bit size of the header or footer is adapted to the parallel width of a serializer of a transmitter (310) configured to serialize the data structure of the output video stream data.

[0154] The transmitter 310 is configured to intermittently forward transmit output serialized video data comprising a series of data structures in the case where a pause is provided between data structures if the next data structure is not ready to be transmitted when the transmission of the last data structure has finished. In other words, the transmitter is further configured to pause the forward transmission of the output serialized video stream between data structures after transmitting a data structure comprising a group of valid pixels and an assigned frame line position indicator until the next data structure comprising a different group of valid pixels having a next frame line position indicator is ready to be transmitted.

[0155] The transmitter 310 can forward the data structure as a serialized video stream 311 to a viewing device 312 including a display such as, for example, a suitable raster-type display, a display including one or more LCD panels 204 shown in FIG. 2.

[0156] As used herein, references to video frame lines, or simply frame lines, refer to horizontal video frame lines that are stacked vertically and that form a video frame when displayed sequentially, for example, as shown in Figures 1A-1D, 2A, and 2B. In other words, in some embodiments, each video frame line comprises a horizontal line of a video frame when displayed in the normal orientation of the video screen, although it is possible to display the frame lines in different orientations.

[0157] Referring now to Figure 4, there is shown in more detail a schematic of how device 302 may pause transmission between forwardly transmitted data structures. In Figure 4, a plurality of serialized data structures 402a-402d form a video frame when displayed on display 414 of observation device 312, with serialized data structures 406a-406b associated with subsequent image frames. Data structures 404a-404d and 406a-406b are shown being serially transmitted by device 302 to observation device 312.

[0158] As shown in FIG. 4, each data structure includes a group of effective pixels and associated header information (the header data structure is not shown in FIG. 4 for clarity). Each pixel group of data structures 402a, 404b, 402c, and 402d is transmitted in turn with metadata, e.g., a video frame line position indicator in the header or footer of the data structure. When reconstructing the serialized image on display 414, the video frame line indicator indicates to the viewing device 312 and / or display 414 which data structure to display first. Additional metadata may be provided to allow the effective pixels of the data structures to be displayed in the correct line order and pixel order on display 414 when reconstructing video frame image 416 on the display. In the example of FIG. 4, each video frame includes only four horizontal lines, each line having four effective pixels.

[0159] 4 also shows schematically how device 310 may pause transmissions 404a-404d, 408a-408b between successive data structures 402a-402d, 406a-406d. Because there will typically be a video frame line containing only blank pixels following the last frame line containing valid pixels, pause 404d may be longer than the other pauses 404a-404c before the first row of the next video frame is transmitted in data structure 406a.

[0160] In some embodiments, the intermittently transmitted video data stream has a fixed frame line length with variable transmission pause periods, such that the pauses between transmissions can vary based on the number of blank pixels received at device 302, which are excluded from the data structure forwarded by device 302 to observation device 312 and therefore not forwarded.

[0161] In some embodiments, the received video data stream has variable line lengths, with varying pause periods between forward transmitted data structures. The invention can also be implemented so that the video data stream has a fixed number of pixels in each frame line.

[0162] Transmission pauses in the outgoing serialized video stream between the data structures of the first video frame are labeled 404a, 404b, 404c, 404d, and after the last data structure, the first data structure of the next video frame is transmitted, followed by a transmission pause 408a, then the next data structure of the next video frame 406b, followed by the next transmission pause labeled 408b, and so on.

[0163] Referring now to FIG. 3B of the drawings, this figure schematically illustrates an example of a method 300 performed by an apparatus 312 for processing a serialized video stream 104 for onward transmission, in accordance with some embodiments of the disclosed technology.

[0164] As shown in FIG. 3B, a method 300 for processing a serialized video stream for onward transmission includes receiving 314 a serialized video stream. The serialized video stream may include multiple video frames transmitted serially from a video source, such as video source 101 of FIG. 3B. Each video frame has multiple frame lines. In embodiments with fixed line lengths, the first through last frame lines of each video frame include a fixed number of effective pixels for display on a display, and the last frame line includes a fixed number of blank pixels. Method 300 also includes buffering at least effective pixels of the received serialized video stream at 316 until a complete frame line of pixels is received at 318. Method 300 further includes processing each buffered complete frame line of pixels to associate a frame line position indicator with the effective pixels of that frame line, where associating may include assigning a frame line position indicator or reusing or retaining an existing frame line position indicator from the received input video data. Method 300 further includes forming a data structure 322 including the effective pixels, including frame line position indicators assigned to the effective pixels. Method 300 further includes forward transmitting 326 the data structure as output serialized video data as soon as it is ready for transmission at 324. The method then includes checking whether the next data structure is ready for transmission at 324, and if not, pausing the forward transmission of the output serialized video stream 328 until the next data structure is ready for transmission at 324, at which point it is transmitted at 326.

[0165] Ongoing transmission is intermittent by pausing transmission between data structures 328 after transmission of the data structure containing valid pixels and assigned frame line position indicators until the next data structure containing a different valid pixel group with a different frame line position indicator is ready to be transmitted at 324. Note that in some embodiments, ongoing transmission can be continuous, with the next data structure always ready to be transmitted after the previous data structure is transmitted. In this case, since device 302 does not wait to receive each video frame, and latency depends on the delay required to receive each video frame line, the forward transmitted data can be considered to be transmitted more efficiently with lower latency; by removing one or more or all blank pixels from each video frame line, each frame line has fewer pixels to transmit, i.e., less data to transmit to the destination carrying the frame line.

[0166] As used herein, the term "data structure" refers to structured data in an onward transmitted serialized video stream, the structured data including groups of effective pixels and one or more indicators, which may also be referred to as tags, associated with the groups of effective pixels. Metadata, such as one or more indicators or tags, may be provided in some embodiments as a header before the associated groups of effective pixels or, in some embodiments, as a footer after the associated groups of pixels. In some embodiments, one or more indicators or tags may precede the associated groups of pixels, or one or more indicators or tags may follow the associated groups of pixels. References herein to a header also apply to indicators or tags provided in other locations relative to the associated groups of effective pixels, such as in a footer, unless the context clearly indicates otherwise.

[0167] Thus, references to apparatus 302 forming a data structure refer to apparatus 302 preparing structured data for output in an onward transmitted serialized video stream.

[0168] Some embodiments of the method 300 performed by the device 302 do not include blank pixels from the received video frame lines of the output serialized video stream that includes the forward transmitted data structure.

[0169] In some embodiments, the method 300 performed by the device 302 further includes excluding or discarding at least one blank pixel of each received video frame line that includes valid pixels and at least one blank pixel, whereby the valid pixels and any remaining blank pixels of the received video frame line are included in the forward transmitted data structure of the output serialized video stream.

[0170] In some embodiments, the pause period in the forward transmission varies according to the number of blank pixels present in each frame line, for example, because the last frame line consists of only blank pixels, there is a very long pause between the transmission of the data structure corresponding to the last frame line of a frame and the transmission of the data structure corresponding to the first line of the next frame in the forward transmitted video data stream.

[0171] In some embodiments, the method 300 performed by the device 302 includes separating the source clock of the received serialized video stream 104 in the output serialized video stream 311 .

[0172] In some embodiments, method 300 enables components of the observation device to use a clock for processing a received serialized video stream that is not based on the clock of the device that includes the serializer used to form the serialized video stream.

[0173] In some embodiments of the method 300, the onward transmission of the output serialized video stream 311 includes transmitting the output serialized video stream over a data communication channel along a wired link to an observation device (312) configured to cause a display 416 of the received video data. This provides a secure link to the display 414, but it may not be the fastest link. In some embodiments, alternatively or additionally, the onward transmission of the output serialized video stream 311 includes transmitting the output serialized data stream 311 over a wireless data communication channel to an observation device 312 configured to cause a display 416 of the received video data.

[0174] In some embodiments of the device 302 shown in Figures 3 and 4, none of the blank pixels of the received video frame lines are included in the output serialized video stream, including the forward transmitted data structure. However, in some embodiments, at least one blank pixel may be discarded and one or more blank pixels may be retained from each line, so that each received variable line length can be regularized to a fixed number of valid pixels and blank pixels. In other words, in some embodiments, at least one blank pixel of each received video frame line that includes valid pixels and at least one blank pixel is discarded, and the valid pixels and any remaining blank pixels of that received video frame line are included in the forward transmitted data structure of the output serialized video stream.

[0175] 4, while the average frame period is aligned with the average frame period of the video source 102, buffering and removal of blank pixels in the serialized transmission of the output video 311 allows the video stream received by the observation device 312 to not be locked to the same clock period used by the video source 104. In other words, in some embodiments, the device 312 decouples the source clock of the received serialized video stream 104 in the output serialized video stream 311 by varying the pause period between forward transmitted data structures.

[0176] In some embodiments, the device's transmitter 310 is configured to transmit the output serialized video stream as a discontinuous serialized video stream over a data communication channel along a wired link with the observation device 312. However, in some embodiments, alternatively or additionally, the forward transmitted serialized video stream data is transmitted intermittently over a wireless data communication channel to the observation device 312 configured to cause a display of the received video data.

[0177] In some embodiments, as shown in FIG. 4, separation is achieved by device 302 using a slightly different clock period X+δ, where δ is a small percentage difference in the clock period for each transmitted pixel in the forward transmitted serialized video stream 311.

[0178] In accordance with the disclosed technology, the clock of the receiver 410 in the observation device 312 can also be a clock that is not locked to the clock period used by the device 302 or to the clock period used by the video source 102. For example, as shown in Figure 4, the clock at the receiver 410 can be run at X-δ, where δ can be the same or a different amount of drift of the clock at the receiver from the source video clock X.

[0179] The observation device 312 adds blank pixels after each valid pixel of a received data structure until the next data structure is received in the serialized transmission of the video 311, forming a video line of variable length. In other words, during pauses when no data structures are received, the receiver 410 adds one or more blank pixels. This allows the receiver 410, in some embodiments, to form a continuous video data stream as output 412 to the display 414 by adding a sufficient number of blank pixels to the end of the last valid pixel group received for display.

[0180] FIG. 5 illustrates another schematic diagram of an apparatus 302 for serializing data for onward transmission in accordance with some embodiments of the disclosed technique. In FIG. 5, the illustrated apparatus 302 includes a memory 306, one or more processors or processing circuits 308, a receiver 304, and a transmitter 310. The memory 306 may include a buffer memory 305, but also includes suitable memory for storing computer code 500, which is illustrated schematically in FIG. 5. The computer program code 500, when loaded from the memory 306 and executed by the one or more processors or processing circuits 308 of the apparatus 302 for processing a serialized video data stream for onward transmission, causes the apparatus 302 to perform the embodiment of the method 300 illustrated in FIG. 3B or a method in accordance with any suitable one of the disclosed embodiments or aspects of a method for processing a serialized video data stream for onward transmission.

[0181] In the embodiment shown in Figure 5, computer code 500 includes one or more software modules or circuits that, when executed by one or more processors or processing units, cause device 302 to form a data structure at 502 and transmit the data structure at 504. As shown in Figure 5, the data structure formation module or circuit includes a module or circuit 503 that associates metadata, e.g., data provided in a header or footer of the data structure, with valid pixels of the data structure. While the following references may refer only to the header of the data structure, those skilled in the art will recognize that a footer or other designated location within a data structure may be used instead of a header, and unless expressly stated to the contrary, references to the term header herein should be construed accordingly.

[0182] The computer code 500 also includes a module or circuit 504 that populates the header and uses the valid pixels and associated metadata to form structured data, also referred to herein as a data structure, at 504. The metadata can include metadata provided by the video source 102 or metadata formed by the device 302.

[0183] In some embodiments, the header population and header association module or circuit 503, 504 may comprise the same module. The header population module or circuit 504 populates the header with one or more indicators or tags. For example, at 506, the header may be populated with a first indicator or tag to indicate whether the pixels of the data structure include valid pixels forming a first line of a displayed video frame. Other information required for proper display of valid pixels on a display may also be captured in the header by adding other indicators 508 (shown in FIG. 5 as adding tag 508). In some embodiments, a footer may be provided instead of or in addition to a header, and references to a header should be interpreted accordingly. The data structure transmission module or circuit 510 outputs a data structure for onward transmission to an observation device, such as observation device 312. The data structures may be output as a continuous stream of video data 311, where after one data structure has been transmitted, the next data structure is ready for transmission. If not, the data structure transmission module 510 pauses transmission until the next data structure is ready to be transmitted.

[0184] 6 illustrates a schematic representation of an exemplary embodiment of observation device 312, where observation device 312 includes a receiver 410, one or more processors or processing circuits 418, such as a graphics processing unit, and memory 420. Receiver 410 is configured to receive serialized video data 311 from a device such as device 302 of FIGS. 3A and 4. In some embodiments, display device 414 includes a display 414 including one or more flat-panel display panels, such as an LCD panel or similar panel, configured to receive the serialized video frame data. The flat-panel display may be a curved display.

[0185] Only valid pixels are displayed on the display 414, and blank pixels are ignored or discarded. Alternatively, the display 414 may be on a separate device connected to the observation device 312 that receives the serialized data from the device 302. In some embodiments, the display 414 may include a near-eye display screen or screen on a headset.

[0186] The receiver 410 is configured to receive and process a discontinuous or intermittent stream of video data 311, and when pauses in the received data occur, the receiver 410 is configured to append one or more blank pixels to the end of each received data structure. The receiver 410 then outputs each frame line, including valid pixels and header or footer information indicating the line position of the pixels within the video frame, along with one or more blank pixels as a continuous data stream to the display 414, which is configured to correctly present the valid pixels so that the entire video frame presented to it is reconstructed in the correct order. The display 414 discards the appended blank pixels and reconstructs each video frame 416 using the frame line position indicators to place the received valid pixels in the correct order on the display.

[0187] In the exemplary embodiment shown in FIG. 6, the observation device 312 includes a display 416, however, as noted above, it will be apparent to those skilled in the art that in some embodiments the observation device 312 may be connected to an external display and configured so that the received video stream data is output to a separate display 414 rather than being provided to an internal display such as the display 414 shown in FIG. 6.

[0188] The memory 420 shown in Figure 6 can store computer code 600 that, when loaded from the memory 420 and executed by one or more processors 418 of an embodiment of the observation device 312 in accordance with the disclosed technology, causes the observation device 312 to perform an embodiment of a method 600 for receiving serialized video data, e.g., a continuous or intermittent sequence of data structures, for presentation on the display 414. As shown in Figure 6, the computer code 600 includes a module or circuit 602 configured to receive the serialized video stream 311, which includes a plurality of data structures, each data structure including at least a set of valid pixels and a header or footer indicating, e.g., the line location of the valid pixels within a video frame as it is displayed on the display 414. The computer program code also includes a module or circuit 604 configured to add one or more blank pixels to the end of a data structure, or in other words, if a data structure includes valid pixels, add additional blank pixels until the next data structure is received. This results in an entire video frame line of blank pixels being formed after the data structure for the last video frame line of a video frame is received and before the first frame line of the next video frame is received.

[0189] The computer code 600 also includes a module or circuit 604 configured to output the valid pixels of the received data structure, with line position indicator information and any blank pixels appended, as a continuous stream of variable line length video data for display on the display 414.

[0190] The computer code 600 may, in some embodiments of the disclosed technology, be provided in the form of a computer program product that includes the computer code 600.

[0191] In some embodiments, the computer code 600 is coded in software as one or more modules, but in some embodiments may instead be partially or completely hard-coded in circuitry. A processor or processing circuitry may be configured to control the execution of hard-coded or soft-coded computer code in some embodiments. In some embodiments, one or more of the processors or processing circuits may include a graphics processor or dedicated graphics processing circuitry.

[0192] Accordingly, the observation device 312 of some embodiments is configured to load computer code 600 from memory 420 and execute it by one or more processors or processing circuits 418 to perform the method 600 for processing a received serialized video stream for display. In some embodiments, the received serialized video stream may be received intermittently, while in some other embodiments, it may be received continuously. Some embodiments of the method include receiving a received serialized video stream including a plurality of data structures, where each data structure includes at least valid pixels from a video frame line and a frame line position indicator assigned to the valid pixels; forming a complete video line of data by appending one or more blank pixels to each valid pixel of the received data structure until a data structure including the next set of pixels of the serialized video stream is received; and outputting each complete video line of data for presentation on a display at a position within the video frame indicated by the frame line position indicator assigned to the valid pixels of the complete video data line.

[0193] In some embodiments, the serialized video stream is received from a device 302 according to any one of the disclosed embodiments.

[0194] In some embodiments, the method further includes determining 605 a that a final frame line of the frame has been received, forming 605 b a frame line including blank pixels as the final frame line of the frame, and outputting 606 the frame line of the frame as a serial data stream of pixels for display on the display 414.

[0195] Referring now to FIG. 7, there is shown in more detail how the observation device 312 can process incoming intermittent video data, e.g., a discontinuous stream of serialized video data, and, in the case where a single video source is present, add blank pixels so that a continuous stream of serialized video data comprising active and blank pixels is output to the display 414.

[0196] As shown in FIG. 7, a module or circuit 602 examines the discontinuous data stream received by the observation device 312 for header or footer information that indicates whether the associated group of pixels is the first horizontal line of pixels of a video frame from a single video source 104.

[0197] In some embodiments, the frame line position indicator is a binary tag having a first value indicating whether the pixels belong to the first line of the video frame, or if not, a binary tag that takes a different value for all other lines in the video frame, thereby allowing the receiving device to output the received pixels in the correct order, since valid pixels in the received serialized video stream 311 are received in order based on the preconfigured number of lines in the frame.

[0198] In some embodiments of the disclosed technology, the frame line indicator assigned to each valid pixel group is included in the header of the data structure.

[0199] In some embodiments, the observer is configured to sample the received video data at regular bit-size intervals or chunks. The bit size of the header can be matched to the bit size of the parallel width of the serializer of the transmitter at the video source 102, so that if the header has the same bit size as the transmitted pixel data, the observer can process the received data stream more efficiently because it only needs to look at one configured bit-size interval once to extract the header information.

[0200] In some embodiments, if the header is too small, for example, the observer transmitter 310 can pad the header with additional bits to increase it so that the header information can be more efficiently extracted from the pixel data stream by the observer 312. In this case, the observer 312 is configured to only examine data at the set bit interval, rather than being configured to search for smaller or larger bit intervals, which would increase processing and potential latency in the observer 312.

[0201] These line indicators are used in the interface to the display 414 when the continuous video data stream received from the receiver 410 of the observation device 412 is presented on the display 414. The observation device can buffer or discard the received data until it recognizes that a group of pixels is preceded by a first line indicator or tag that indicates the point at which the pixels should be output to the display 414. The pixel header information can include information regarding how the lines are to be presented on the display 414 to properly present the image video frame 416, such as line position as well as information regarding which pixel should be presented first.

[0202] As mentioned above, in some embodiments, instead of or in addition to having a header, each data structure can provide equivalent information in a footer associated with the effective pixel group. References to a header herein should also be considered to apply when the header is provided as a footer following the effective pixel group in the data structure. An indicator or tag populated in the header field is used to identify which line is the first line, regardless of whether the header is provided as a header before the effective pixel group and / or as a footer after the effective pixel group, and in some embodiments, techniques defined in the SDI standard can be used to provide this information, for example, in the form of a 64-bit header tag. In some embodiments, the data structures of the disclosed technology can use different header or footer sizes to identify whether pixels are the first line of pixels.

[0203] In some embodiments, the header or footer is the same bit size as a pixel. That is, if a pixel is 20 bits, the header or footer can also be 20 bits. This makes it easy to configure the receiver of the observation device to extract the header or footer in an efficient format that can capture 20 bits every 20 bits, etc., of the received serialized data stream. If the header or footer has a bit size different from the bit size of a pixel, for example, if the header has a size of 2 bits, this means that the receiver needs to capture 20 bits, 20 bits, 20 bits, etc., then 2 bits. To make the header the same bit size as a pixel, one or more dummy bits can be added to the header by the serializer of the transmitter.

[0204] Thus, the module or circuit 602 shown in Figure 6 configures the receiver 410 of the observation device 312 to buffer data until the first frame line of a video frame is received. As shown in Figure 7, in some embodiments, in the observation device 312, the receiver waits for the first frame line of a video frame in the received serialized video data and checks the received data structure against the first frame line position indicator in the header. Once all valid pixels for the first line of the video frame have been received, the module or circuit 604, 605 configures the receiver 410 to append zeros or one or more blank pixels following the valid pixels. These appended blank pixels, in some embodiments, cause the observation device to form variable line length video data, which results in a continuous stream of serialized frame lines being output to the display 414. Module or circuit 606 configures receiver 410 to provide the valid pixels and the following blank pixels as a continuous stream of pixels that are output to display 414 until the next group of valid pixels is received and streamed to the display, etc. For example, in an example where display 414 is a raster display that displays video lines of pixels from right to left, the first pixel transmitted is displayed on the right and the next pixel in the group of valid pixels is displayed from right to left along a horizontal line.

[0205] In some embodiments, for each group of received pixels corresponding to a frame line of a video frame, once all valid pixels transmitted to the display have been displayed, the receiver transmits blank pixels until the next frame line of that video frame can be output. Alternatively, if the last frame line output is the last frame line in the frame, the receiver 410 forms the end of the frame line with blank pixels. In either case, the frame line data is provided to the display 414 for presentation. After each frame line of valid pixels is displayed on the display, all subsequent pixels are treated as blank pixels and are not displayed until the next line indicator tag is received.

[0206] The recognition of which pixel of a group of pixels is the first pixel in a video frame line occurs at the interface to the display and conforms to standard techniques for raster-type displays, e.g., LCD or OLED-type displays. In some embodiments, this can be coded within the header structure for a pixel or group of pixels. In other words, it is known in the art to provide a separate signal that indicates to the display 414 which pixel is the first pixel of a subsequent line (i.e., by counting more than a predetermined number of pixels, any additional pixels are discarded by the display 414 as blank pixels). Such a signal sent for each line is used to indicate the first pixel and may, for example, follow a standard VGA or HD signal format.

[0207] The form of the display 414 may vary in various embodiments, and in some embodiments may include a near-eye display.

[0208] While the above embodiments relate to a single video source, in some embodiments, additional data, such as graphic overlay data, may be provided from one or more other sources and blended with the video image data at video source 102.

[0209] In some embodiments described above and below, the observation device and / or the device may be provided with an additional control module or device that controls the receiver and transmitter in response to execution of computer code in the transmitter and receiver for performing the associated methods disclosed herein.

[0210] FIG. 8 illustrates an exemplary embodiment of a video display system 800 according to the disclosed technology. The video display system 800 includes a video hub, also referred to herein as a video combining device 810, according to any one of the embodiments of the device 810 disclosed herein, and an observation device 824 according to any one of the disclosed embodiments of the device disclosed herein. The video combining device 810 may include one or more video sources 812a, 812b, or these may be provided separately as shown in FIG. 8. The video sources 812a, 812b are each configured to provide a synchronized serialized video data stream intended for synchronized display on two displays 834a, 834b in the observation device 824. The observation device 824 may be connectable to or include at least one of the two displays 834a, 834b. In some embodiments, each display 834a, 834b is configured to present video received from one of the video sources 812a, 812b. The observation devices are connected to the combining device via a serialized data link, such as a serialized data interface (SDI) data link. The observation devices 824 may be provided as part of the video combining device 810, in which case the transmission channel 822 shown in FIG. 8 may be provided via an internal wired or wireless data communication channel, which may or may not conform to the SDI data standard. The display interfaces are configured to receive the serialized video data streams and present the received data on the display with minimal or no buffering.

[0211] In one embodiment of the system shown in FIG. 8, a video display system 800 includes a video combiner 810 configured to process two serialized video data streams 814a, 814b.

[0212] Each video stream includes video frame data obtained from video sources 812b and 812c. Device 810 includes a transmitter 820, which includes a serializer 819, that combines two parallel streams of video data 814a and 814b frame line by frame line at transmitter 820 to form a single serialized data stream for onward transmission. Transmitter device 820 outputs the combined serialized data to observation device 824 for onward transmission via serialized data transmission channel 822. The output may include a continuous stream or a discontinuous data stream that is transmitted intermittently depending on when a complete frame line from either source is ready for transmission.

[0213] The combining device 810 includes a memory 815, which may include one or more buffers 816a, 816b for buffering the received video data streams 814a, 814b at a receiver 818 of the device 810. The device 810 also includes one or more processors or processing circuits 817 configured to process the buffered data streams 819a, 819b to associate effective pixels with information such as their video source and, if not already provided, the line position within a video frame of each pixel group. In some embodiments, the one or more processors or processing circuits may form part of a serializer 819. The serializer 819 may be provided as part of a transmitter 820 configured to output the video data from the two buffers 816a, 816b in a serialized combined data stream 822. Buffers 816 a and 816 b are configured to process received data streams 814 a and 814 b in parallel, respectively, and forward the data to serializer 819 simultaneously.

[0214] The observation device 824 includes a receiver / display driver 826 configured to receive the serialized combined video stream 822 from the video combiner 810. The receiver 826 is configured to examine header information for each group of received effective pixels and includes a deserializer 827 configured to deserialize the effective pixels into two parallel channels 828a, 828b based on the header information indicating that the effective pixels have different video sources 812a, 812b. References herein to a "header" may, in some embodiments, be interpreted as references to a footer and / or other designated location instead of a header.

[0215] The receiver / LCD driver 826 can also add blank pixels to each group of valid pixels and buffer the output until the first frame line of video data is provided, so that the outputs 830a, 830b, shown as LEFT OUT 830a and RIGHT OUT 830b for the respective displays 834a, 834b, are displayed correctly.

[0216] Each output data stream 832a, 832b includes a continuous stream of valid and blank pixels. At each display 834a, 834b, the blank pixels in each output data stream 832a, 832b are discarded and only the valid pixels in each video frame line are presented in the proper order, with each video frame line correctly positioned within the frame.

[0217] 8 may occasionally or always receive a combined data stream 822, which is a line-by-line interleaving of pixels from different video sources 812a, 812b at combiner 810. Additionally, by excluding at least one blank pixel, fewer pixels need to be transmitted over any shared link used for forward transmission, thereby reducing the transmission time required to transmit video from the two sources to observer 824.

[0218] In some embodiments, the video combining device 810 may include two or more video sources 812a, 812b, which may be separate in some embodiments. In some embodiments, the video combining device 810 and the observation device may be combined.

[0219] In some embodiments, the observation device 824 includes two or more displays 834a, 834b, where each different display is configured to present video of video data received from different video sources 812a, 812b and transmitted forward over the same video data channel 822 by the combining device 810, although the displays may be provided separately in some embodiments.

[0220] In some embodiments, displays 834a, 834b are provided as left and right near-eye displays, such as the left and right displays of a headset. In some embodiments, displays 834a, 834b are connected to, but may be separate from, the observation device 824 to make the headset more manageable or to reduce the weight of the headset housing displays 834a, 834b. In some embodiments, observation device 824 may provide a dedicated device, i.e., a portion of a display, implemented using a tablet or similar computer-type device, and in some embodiments, may provide output to sections of a real or virtual screen that replicate the left and right displays. Note that in other use cases, the headset or near-eye display may include both left and right displays, for example, if the entire observation device is sufficiently compact or lightweight. Various display combinations may be provided in some embodiments; for example, observation device 824, when configured with appropriate drivers, may output video to a near-eye display and simultaneously or substantially simultaneously to a large display screen.

[0221] In some embodiments, the video combining device (also referred to herein as a hub) 810 may include a hub co-located with the surgical imaging device. In some embodiments including a hub co-located with the surgical imaging device, the hub 810 is configured to receive serialized video streams from two or more video sources, e.g., left-view and right-view video sources. In some embodiments, each video source includes a camera, and each camera is part of the surgical imaging device. According to some embodiments, the hub 810 is or is part of the surgical imaging device, including a surgical microscope. In some embodiments, the hub 810 can be co-located with the surgical microscope and / or can be part of a control system or can provide another computing component or unit for the surgical microscope.

[0222] 8, there are two separate video sources 812a, 812b, each configured to provide a separate variable line length serialized video stream 814a, 814b to the combining device 810. In some embodiments, more than two video sources can be configured to provide serialized video data to the video combining device 810. In some embodiments, images from each video source are presented synchronously on different displays in the observation device 824. However, in some embodiments, additional information can be added to one or more video data streams, such as overlay image data or information that comes from another source and can be combined with the video image data from one of the image video sources 812a, 812b.

[0223] Overlay-type information intended to be displayed on the same display as the video images from sources 812a, 812b can be blended or otherwise integrated with the video information in sources 812a, 812b or combining device 810. Such overlay information can, for example, modify how the video is displayed on the displays, such as highlighting or adjusting the color or contrast of one of the images displayed on one or both of the displays. Alternatively or additionally, overlay information can provide text or symbols to accompany the video presentation. In some embodiments, overlay information is synchronized with two or more video sources of video data; thus, in some embodiments, the same overlay information source can be split and combined with both serialized video streams 814a, 814b.

[0224] In some embodiments, if the number of video sources does not match the number of separate displays, the video data may be fused so that an effective pixel presented on one of the displays may contain data from more than one source. In some embodiments, in this case, the data structure header information may include an identifier indicating a data structure containing pixels for overlay in some embodiments and image data (or other forms of combined data) from the two sources.

[0225] In some embodiments, the hub may process the video data received from the source to add overlays or image enhancements, which in some embodiments may be embedded in the video stream output by the hub or provided as a separate stream.

[0226] In some embodiments of the disclosed technology, a video combining device 810 includes a device 810 for combining multiple serialized video streams 814a, 814b from at least two different video sources 812a, 812b, e.g., two different video sources 812a, 812b, as shown in the embodiment of FIG. 8. The video combining device 810 includes a receiver 818 configured to receive each serialized video stream 814a, 814b from the different video sources 812a, 812b via a separate channel. Each received serialized video stream includes multiple video frames consisting of lines of active pixels and blank pixels. The combining device 810 also includes one or more processors or processing circuits 817 and a memory 815. The memory 815 can include multiple buffers 816a, 816b, each configured to buffer at least active pixels of the received serialized video streams 814a, 814b. The one or more processors or processing circuits 817 are configured to: for each effective pixel group in a completely received line of pixels in a received frame of the received serialized video streams 814 a, 814 b, assign a video source indicator to the effective pixel group, include or assign a frame line position indicator to the effective pixel group, and form a data structure including the effective pixel group, the source indicator for the pixel group, and the frame line position indicator assigned to the effective pixel group. The combining device 810 also includes a transmitter 820 configured to forward transmit the formed data structure from each received serialized video stream 814 a, 814 b as a single serialized video stream 822.

[0227] In some embodiments, the one or more processors or processing circuits 817 may be provided as part of the transmitter 820 of the synthesizer 810. In some embodiments, the one or more processors or processing circuits may be configurable, for example, by loading and executing computer program code, shown in Figure 11 as computer code 1106, which may be stored in a suitable form of memory 815.

[0228] In the observation device 824, in some embodiments, one or more processors or processing circuits 838 may be provided, for example as part of the receiver 826 or the deserializer 827. In some embodiments, the one or more processors or processing circuits 838 are configurable by loading and executing computer program code, for example, as shown in FIG. 12 as computer code 1200 from a suitable form of memory 836, to cause the observation device to perform a method for observing multiple serialized video streams received as a single serialized combined video stream. For example, in some embodiments, a method includes receiving a serialized video stream including a plurality of data structures, each including a group of effective pixels, a source indicator assigned to the group of pixels, and a frame line position indicator assigned to the group of effective pixels; processing the received serialized video stream by adding one or more blank pixels to each group of effective pixels of the received data structures to form a complete line of video data until a data structure including a next group of pixels of the serialized video stream is received; and outputting each complete line of video data for presentation on a display at a position within a video frame indicated by the frame line position indicator assigned to the effective pixels of the complete line of video data, in association with the source indicator of the complete line of video data.

[0229] In some embodiments, the observation device 824 includes a receiver 826 configured to receive a serialized video stream 822 including data from multiple different sources 812 a, 812 b, such as a composite serialized video stream 822 from the combiner device 810. The serialized video stream 822 is comprised of multiple data structures, each including a group of active pixels, a source indicator assigned to the group of active pixels, and a frame line position indicator assigned to the group of active pixels. The observation device 824 also includes one or more processors or processing circuits 838 configured to form a complete line of video data by adding one or more blank pixels to each active pixel of the received data structure until a data structure including the next group of pixels in the serialized video stream is received. Each complete line of video data is then provided for display on a display 834 a, 834 b associated with the complete video line's source indicator at a position within the video frame indicated by the frame line position indicator assigned to the active pixels of the complete video data line.

[0230] FIG. 9 illustrates, in some embodiments of the operation of the combining device 810, such as the embodiment of the device 810 illustrated generally in FIG. 8, how the transmitter 820 can subsequently pause forward transmission between data structures.

[0231] In Figure 9, the serializer 819 is not shown for clarity. Pauses are labeled TX PAUSE #1 through #5 and may occur occasionally between transmissions of data structures or may occur constantly between data structures. In Figure 9, the data structures are labeled A#1, B#1, A#2, B#2, and A#3, where A# indicates a data structure containing valid pixels from video data source 812a and B# indicates a data structure containing valid pixels from video data source 812b.

[0232] The transmission pauses may occur at regular or irregular intervals depending on when the data structures are transmitted and how long it takes for the next data structure to be ready to transmit after the previous data structure has finished transmitting.

[0233] Each group of valid pixels is assigned a source indicator in the combiner that is included in the header of that group of pixels before being transmitted forward. Additional header information may include a frame line position indicator for that group of pixels and any other header information that may be retained from the serialized data received from the video source. In some embodiments, for example, transmission of a frame line from one source 812a may be completed before the next data structure containing a different group of valid pixels with a different source indicator #B is ready for transmission; in this case, instead of pausing, if another data structure from the first source is ready for transmission, transmission occurs without waiting for the other source frame line to be ready.

[0234] As shown in Figure 9, the first frame line from a first source, e.g., video source 812a, is transmitted (this transmission is labeled A#1 in Figure 9). A transmission pause, labeled TX#1, then occurs because no subsequent data structures are ready for transmission. However, as soon as the subsequent data structure, B#1, is ready for transmission, it is output along serialized video data link 822. In the illustrated example, the next data structure ready for transmission is labeled B#1, after which another pause, i.e., pause#2, occurs; when data structure A#2 is ready for transmission, it is transmitted; before B#2 is ready for transmission, another pause#3 occurs; then after pause#4, data structure A#3 is transmitted; then another pause occurs.

[0235] It should be noted that the transmitter 820 can emit a continuous stream of pixels by alternating between determining which source's pixels are to be transmitted, so that after a data structure containing valid pixels of a video frame from a first video source 812 a is transmitted, the transmitter will transmit a data structure containing valid pixels of a video frame from the other video source 812 b if it is ready for transmission by the time the data structure containing pixels from the first video source 812 a is completed. In other words, some or all of the illustrated pauses may not occur in some examples of the disclosed technology, and the sequence of frame lines may not strictly alternate from one source to another.

[0236] In some embodiments, each frame line comprises a horizontal line of a video frame when displayed.

[0237] In some embodiments, the video source indicator identifies a spatial relationship between at least two of the multiple video sources. For example, as shown in Figure 8, in some embodiments, two video sources 812a, 812b provide left and right views of a scene, and a receiver 826 of an observation device 824 is configured to examine a received video stream 822 for an indicator for each video source and, accordingly, assign active pixels of a data structure associated with that source indicator to one of two displays 834a, 834b.

[0238] In some embodiments, the multiple serialized video streams 814a, 814b include a video stream obtained from a video source of a left view of the scene and a video stream obtained from a video source of a right view of the scene, and different video source indicators are used to identify whether a group of pixels belongs to a video frame obtained from the video source of the left view or the video source of the right view of the scene.

[0239] In some embodiments, the assigned video source indicators and frame line position indicators are included in a header of the data structure, and the bit size of the header matches the parallel width of a serializer of a transmitter 820 configured to combine data structures from multiple different video sources 812a, 812b into one output serialized video stream 822.

[0240] In some embodiments, the data structures from the two sources 812 a , 812 b are interleaved by a serializer in the output serialized video stream 822 .

[0241] In some embodiments, interleaving alternates data structures from one video source with data structures from another video source in the output serialized video stream. In some embodiments, if a data structure from another video source is not yet ready to be transmitted, the transmitter 820 pauses transmission until it is ready to be transmitted. Alternatively, in some embodiments, the transmitter can transmit data structures derived from frame lines of the same source as a previously transmitted frame line.

[0242] The device 810 can be configured to receive at least one serialized video stream 814a, 814b over a channel having one or more of a different data rate and a different video resolution relative to at least one other channel receiving another received video stream 814a, 814b.

[0243] While latency at displays 834a, 834b is better reduced by removing all blank pixels in a received video frame line at device 810, whereby preferably none of the blank pixels of the received video frame line are included in the output serialized video stream 822, including the forward transmitted data structure, an equivalent benefit in latency reduction is instead achieved by removing at least one blank pixel of each received video frame line that includes valid pixels, whereby the at least one blank pixel is discarded and the valid pixels of that received video frame line and all remaining blank pixels are included in the forward transmitted data structure of the output serialized video stream.

[0244] In some embodiments, the video combiner 810 decouples the source clock of the received serialized video stream in the output serialized video stream by switching between various video sources as serialized video data is combed for forward transmission and / or by pausing transmission of a data structure if the data structure is not yet ready for transmission after the previous data structure has been transmitted. In some embodiments, the receiver clock and the source clock can be correspondingly decoupled by varying the pause period between forward transmitted data structures. Furthermore, the transmitter clock of the serializer 819 can be correspondingly decoupled from the clock at the receiver 826 of the observation unit 824.

[0245] In some embodiments, the transmitter 820 is configured to transmit the output serialized video stream via a data communication channel 822 along a wired link and / or via a wireless data communication channel to an observation device 824 configured to cause a display of the received video data.

[0246] FIG. 10 illustrates an exemplary embodiment of a method 1000 performed by a combining device 810 configured to combine two parallel streams of serialized video data. As shown in FIG. 10, the method 1000 includes receiving 1002a a video stream from a video source, e.g., the video source 812a shown in FIG. 8 or FIG. 9, and simultaneously receiving 1002b a video stream from another video source, e.g., the video source 812b shown in FIG. 8 or FIG. 9. Each stream is buffered separately 804a, 804b until a complete video frame line is received 1006a, 1006b, at which point, in some embodiments, a header associated with the effective pixels of each video frame line is assigned an indicator or tag of the video source of each effective pixel group of the video frame. The header may already include or be assigned an indication of the video frame line and any other relevant information, such as information regarding which pixel in the effective pixel group is the first pixel to ultimately be presented on the display of the video frame at the observation device 824. A data structure containing the valid pixels and header information is then formed as 1010a, 1010b, and whichever of the two resulting data structures is ready for next transmission is then forward transmitted at 1014, for example, to the observation device 824 shown in Figures 8 and 9. Other data structures may be buffered until they are ready for transmission in some embodiments, in which case forward transmission may be sequential from one data structure to another, and the data structures may be from alternate sources in some embodiments. If, after the previous data structure has been transmitted, no data structures from either source are ready for transmission, the transmitter may pause transmission 1016 until the next data structure is ready for transmission.

[0247] The clock at the observation device is separable from the transmission clock at the originating video source 812a, 812b, since any pause in transmission decouples the timing of the transmitted pixels from the originating clock timing.

[0248] In some embodiments, the method 1000 includes combining multiple serialized video streams 814a, 814b from different video sources 812a, 812b, where each serialized video stream is received over a different communication channel for serialized forward transmission over the channel. In some embodiments, the method includes separately receiving 802a, 802b a respective one of a plurality of serialized video streams; separately buffering 804a, 804b at least effective pixels of a received frame line of a received frame of each of the plurality of received video streams; processing each buffered effective pixel group in a completely received frame line by at least assigning 808a, 808b a video source indicator to the effective pixel group; and forming 810a, 810b a data structure including the effective pixel group, an indicator for the video source of the pixel group, and a frame line position indicator for the effective pixel group; and forward transmitting over a common channel the formed data structure including the buffered effective pixels from the frame line of a video frame of one of the plurality of received serialized video streams.

[0249] In some embodiments of method 1000, after each data structure has been transmitted, the method further includes determining whether another data structure from a video frame line of a video frame originating from another one of the multiple video sources is ready to begin forward transmission, and if so, forward transmitting the data structure from the other one of the multiple video sources over the common channel; or determining whether another data structure from one video source of the multiple video sources is ready to begin forward transmission, and if so, forward transmitting the data structure from the one video source of the multiple video sources over the channel.

[0250] In some embodiments of the method, after the data structure has been transmitted, the method further includes determining that no data structure from any of the multiple video sources is ready for transmission, pausing transmission until a data structure from one of the multiple video sources is ready for transmission, and forward transmitting the data structure over the common channel.

[0251] 11 shows schematically how the device 812 may include computer code 1106, for example in the form of a computer program stored in a memory or configured in hardware executable by one or more processors or processing circuits 817. The computer program code, in some embodiments, includes one or more modules or circuits that control various components of the coupling device 810 to perform a method, such as the method 1000 shown in FIG.

[0252] 11 illustrates how computer code 1106 may include one or more modules or circuits 1108 that, when executed, cause a data structure having a header to be formed, and one or more modules or circuits 1110 that, when executed, cause the header to be populated. In some embodiments, computer code 1110 includes one or more modules or circuits 1202 that, when executed, cause combining device 810 to determine or check whether a video line position indicator within a frame of valid pixels is present within the received video data stream. The frame line indicator may be included in the forward transmitted data structure and may indicate, for example, whether a video frame line's valid pixels belong to the first line of the video frame.

[0253] If there is no frame line position indicator already associated with the valid pixel group, the header is populated with a frame line indicator. In some embodiments, the frame line indicator may include a binary indicator representing whether the frame line is the first frame line.

[0254] The computer code 1106 also includes one or more modules or circuits 1204 configured, when executed, to populate the header with any other relevant information. For example, the computer code 1106 may include one or more modules or circuits configured, when executed, to add an indicator or tag to the header in 1116 to identify the video source or indicator of the valid pixels, where the observation device is configured to send the valid pixels associated with a particular video source identifier to a particular one of the displays 834a, 834b. Alternatively or additionally, the indicator or tag may identify the display on which the valid pixels are to be displayed in some embodiments, where the observation device is configured, upon detecting a particular display indicator, to forward the received valid pixels having the particular display indicator (and any added blank pixels) to the particular display.

[0255] In some embodiments, other relevant information may include additional indicators or tags to indicate which pixels per line are displayed first and / or to indicate the direction in which the lines are presented on the display, e.g., from left to right.

[0256] As will be apparent to those skilled in the art, in the above embodiments, the header may alternatively or additionally be provided as a footer or may be located in another predetermined position in the data structure relative to the active pixels being forwarded.

[0257] 12 shows a schematic of how the viewing device 824 may include one or more processors or processing circuits 838, memory 836, and receivers 828, e.g., receivers / drivers for multiple displays 834a, 834. As shown in FIG. 12, the displays are included in the viewing device, but in some embodiments, they may be separate displays. In some embodiments, the viewing device 824 may be configured to receive video routed from the combining device 810. In some embodiments, the displays 834a, 834b comprise left and right near-eye displays in a headset or similar near-eye display viewing device.

[0258] 12, memory 836 includes memory that stores computer code 1200. Computer code 1200 may include one or more modules or circuits 1202, 1204, 1206 that, when executed by one or more processors or processing circuits 838, cause observation device 824 to perform a method for observing multiple serialized video streams received as a single serialized video stream.

[0259] In some embodiments, for example, the computer code includes one or more receiver modules or circuits 1202 that, when executed, cause the observation device 824 to receive a serialized video stream 822 including a plurality of data structures, where each data structure includes a group of effective pixels, a source indicator assigned to the group of pixels, and a frame line position indicator assigned to the group of effective pixels. The computer program code may also include one or more deserializing modules or circuits 1204 that, when executed by the observation device 824 or a deserializer 827 of the device 824, process the received serialized video stream 822 to form a complete line of video data by appending one or more blank pixels to each group of effective pixels of the received data structure until a data structure including the next group of pixels in the serialized video stream is received.

[0260] The computer program code may also include one or more data output modules or circuits 1206 configured, when executed, to output each complete video data line for display on a display 834 a, 834 b associated with the source indicator of the effective pixel at a position within the video frame indicated by a frame line position indicator assigned to the effective pixel of that complete video data line.

[0261] In the above embodiments, video received by device 302, 802 from one or more video sources 102, 812a, 812b may be received from the video sources via SDI link 14. Memory 306 may include suitable read-only memory and / or random-access memory for storing computer code executed by one or more processors or processing circuits 308. Memory 306 may include at least one line buffer 305, 816a, 816b for storing each horizontal line of video data as it is received from video source 102, 812a, 812b via the SDI link. When buffer 306, 816a, 816b is full, in other words, when a complete horizontal line of valid and blank pixels has been received, the blank pixels are discarded or otherwise ignored or removed, and the valid pixels are processed for forward transmission. In some embodiments, the forward transmission may be continuous, without any pauses. In some embodiments, if several data structures are buffered and ready to be sent after the device 310, 810 has completed sending a previous data structure, these data structures can be sent without a pause, but a pause can be inserted at any time if no data structures are ready to be sent.

[0262] Displays 414, 834a, 834 may include any suitable type of display, for example, a raster-scan display, such as a display that displays video using liquid crystal display (LCD) or light emitting diode (LED) or organic light emitting diode (OLED) technology.

[0263] Video data is transmitted serially, pixel by pixel, over a serialized data interface (SDI) link or channel. The order in which the bits that make up each pixel, typically 24 bits, are transmitted can be determined differently by different communication protocols. Note that with serialization, the most significant bit of each pixel is transmitted first, and the remaining pixels are transmitted in order until the last pixel is transmitted. Serialized video streams can run at speeds of up to 6 Gbit / s or 12 Gbit / s, depending on the required video resolution.

[0264] In some embodiments, the transmission time for each frame varies, and only the average frame period for forward transmission is adapted to the frame period of the video received from the video source 102, 812a, 812b by the device 310, 810. In other words, the source clock used to transmit each pixel by the device 310, 810 need not be the same as the display clock used to receive each transmitted pixel at the display 414, 834a, 834b.

[0265] 3A and 8, transmitters 310 and 820, respectively, are configured to ensure that the average transmission frame period matches the transmission time of the or each video source. In other words, the disclosed transmitters 310 and 820 are configured, in some embodiments, to allow for different transmission periods for individual pixels within a video frame. In the embodiments described herein, idle periods during transmission are used to compensate for additional or fewer valid pixels from a video source due to clock differences between devices 310, 810 and source 102 in some embodiments.

[0266] 4, for example, for pixel group 402a, although the following also pertains to embodiments in which video from two sources is combined, in an exemplary embodiment, the clock period used to transmit pixels may be X=0.1000 μs plus δ=0.0001 μs, i.e., the clock is X+δ=0.10001 μs, and the clock period of the receiver of display 414 used to receive blanking line pixels is X=0.1000 μs minus δ=0.0001 μs, i.e., X−δ=0.999 μs. In other words, in some embodiments of the disclosed technology, the receiver 410, 824 of the observation device 410, 824 uses a local clock to drive the output to the display 414, 834a, 834b; in other words, the observation device can use a local clock to drive the rate at which pixels are presented to the display, e.g., an LCD panel, without having to derive a clock from the transmitter 310, 820. In other words, the transmission times of individual frames may vary within a predetermined tolerance level, as long as the transmission duration of the frames does not vary too much on average.

[0267] Varying the blanking period added by the display driver 412, 824 to the end of each received valid pixel group 402a-402d of the video frame compensates for variations in the arrival time of each horizontal line of serialized video data. As noted above, these blank pixels are not displayed, but can compensate for differences between the receiver clock of the observation device 410, 824 and the source clock at the transmitter 310, 820 and / or the receiver clock of the observation device 410, 824 and the source clock at the video source 102, 812a, 812b.

[0268] The blank pixels added by the display are used to compensate for different video line arrival times at the display, so that the pixels forming the pixel strings output to the display panels 414, 834a, 834b of the viewing device 410, 824 can be synchronized at the display even when there are two or more separate video sources.

[0269] In some embodiments of the disclosed technology, the video output is configured to be provided to an observation device 410, 824, which is configured to drive a display 414, 834a, 834b, e.g., an LCD panel display, having variable length horizontal video frame lines. In some embodiments, the variable length horizontal frame lines are configured to keep up with the horizontal timing of the source video, although this is not required to lock to the video source clock timing. Thus, in some embodiments, a proprietary serial data interface type video transport method can be used that does not require locking to the video source clock at the display.

[0270] In some embodiments, observation unit 410, 824 is configured to automatically add a variable number of video blank pixels to each group of pixels comprising a horizontal line of valid pixels received from unit 310, 810 until the last valid pixel of the next complete video line is received, thereby advantageously allowing error correction to be performed in the next received video frame line.

[0271] In some embodiments of the device 302 of the disclosed technology, two serialized video streams are received from different sources. An exemplary embodiment of the device 302 including a device 510 is shown in Figure 5 of the accompanying drawings.

[0272] In some embodiments, the forward transmitted video data is compressed and one or more communication links are configured to have a total effective bandwidth that is less than the sum of the effective bandwidths of each source.

[0273] In some embodiments, the video sources 12, 512a, 512b may be RGB camera sources. In some embodiments, the display 414 or display system 518 includes a near-eye display, for example a stereoscopic near-eye display.

[0274] While a display may have effective pixels, e.g., a microdisplay or liquid crystal display that defines a video line of 1920 effective pixels followed by a fixed number of blank pixels, followed by 40 blank pixels, it has been found that the disclosed observation device can provide 39 or 41 blank pixels, etc., for 1920 effective pixels without visually affecting the way the video is displayed. Thus, in some embodiments, the observation device does not need to add a fixed number of blank pixels to the end of each group of effective pixels. Advantageously, disclosed embodiments of a combining device 810 that receives video from two or more sources 812a, 812b allow the combining device 810 to provide independent clocks to the two or more video sources 812a, 812b and still share the same transmission cable. In some embodiments, an additional video frame buffer can be used, but if such a frame buffer is not required, the video received by the display 16,518 may have low or very low latency, e.g., on the order of 10 μs.

[0275] The embodiments disclosed above enable displays, such as LCD panels, to avoid the need for their clocks to be supplied by the source of the video they are receiving for display. In contrast to using conventional SDI transmitters, which require source video clocks to be locked, the disclosed technology allows for the transmission of video from two or more different video sources over the same or different physical channels, even if the video sources are not clock-synchronized. Furthermore, transmitting data line-by-line in the manner disclosed herein eliminates the need for a video frame buffer, thereby avoiding the long latency required when using conventional SDI transmitters.

[0276] In some embodiments, displays 834a, 834b include near-eye stereoscopic displays based on diode technology, such as LCD or OLED displays.

[0277] The above-described program code may also be provided as a computer program product in the form of a data carrier carrying computer program code or code means for carrying out the embodiments herein when loaded into, for example, a processing circuit in the control unit. The data carrier or computer-readable medium may be one of an electronic signal, an optical signal, a radio signal, or a computer-readable storage medium. The computer program code may be provided as pure program code, for example, in the control unit or on a server, and downloaded to the control unit. It should therefore be noted that the functionality of the control unit may in some embodiments be implemented as a computer program stored in memory, for example, in a computer-readable storage unit, for execution by a processor or processing module, for example, a processing circuit in the control unit.

[0278] Those skilled in the art will also understand that the processing circuitry and memory or computer-readable storage unit described above may refer to a combination of analog and digital circuitry and / or one or more processors configured by software and / or firmware, e.g., stored in memory, that, when executed by one or more processors such as processing circuitry, perform as described above. One or more of these processors and other digital hardware may be included in a single application-specific integrated circuit (ASIC), or multiple processors and various digital hardware may be distributed among multiple separate components, whether individually packaged or assembled into a system-on-chip (SoC).

[0279] The control unit of some embodiments may also include or control how signals are transmitted along the communication link 103, 822, which may be a wired or wireless communication link. If the signals are transmitted wirelessly, the control unit may also appropriately control the operation of the antenna.

[0280] The communication channel may be point-to-point or over a network, such as a cellular or satellite network, that supports wireless communications, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet, and the like. The communication protocol may conform to one or more public or proprietary communication standards, protocols, and / or technologies, including Voice over IP (VoIP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), and / or Instant Messaging Service (IMPS), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this application.

[0281] The operating system of the disclosed technology may require various other software components and / or drivers that control and manage general system tasks (e.g., memory management, storage device control, power management, etc.) to facilitate communication between the various hardware and software components, in some embodiments not shown in the drawings where inclusion is apparent for clarity.

[0282] When the disclosed technology is described with reference to drawings in block diagram and / or flowchart form, it will be understood that multiple entities in the drawings, e.g., blocks of the block diagrams and combinations of entities in the drawings, can be implemented by computer program instructions, which can be stored in a computer-readable memory and loaded into a computer or other programmable data processing apparatus. Such computer program instructions can be supplied to a processor in a general-purpose computer, a special-purpose computer, and / or other programmable data processing apparatus, thereby producing a machine, such that the instructions, executed via the processor of the computer and / or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the block diagrams and / or flowcharts.

[0283] In some implementations, according to some aspects of the disclosure, the functions or steps noted in the blocks may be performed in an order different from the order noted in the illustrations of the operations. For example, two blocks shown in succession may in fact be performed substantially concurrently, or the blocks may sometimes be performed in the reverse order, depending on the functionality / acts involved. Also, the functions or steps noted in the blocks may be performed sequentially in a loop, according to some aspects of the disclosure.

[0284] The descriptions of exemplary embodiments provided herein are presented for illustrative purposes. The description is not intended to be exhaustive or to limit the exemplary embodiments to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be obtained from practicing various alternatives of the provided embodiments. The examples described herein are chosen and described in order to explain the basic scheme and nature of various exemplary embodiments and their practical applications, enabling those skilled in the art to utilize the exemplary embodiments in various ways and with various modifications suited to the particular use intended. Features of the embodiments described herein can be combined in all possible combinations of methods, apparatus, modules, systems, and computer program products. It is understood that the exemplary embodiments presented herein can be used in any combination with each other.

[0285] It should be noted that the word "comprising" does not necessarily exclude the presence of other elements, features, functions or steps than those listed, and the words "a" or "an" preceding an element do not exclude the presence of a plurality of elements, features, functions or steps. Furthermore, it should be noted that reference signs do not limit the scope of the claims, and that the exemplary embodiments can be implemented at least partly by means of both hardware and software means, and that several "means," "units" or "apparatus" can be represented by the same item of hardware.

[0286] Various exemplary embodiments described herein are described in the general context of methods, which may further refer to elements, functions, steps, or processes, one or more or all of which may in one aspect be implemented by a computer program product embodied in a computer-readable medium that includes computer-executable instructions, e.g., program code, that are executed by computers in a network environment.

[0287] Computer-readable media can include removable or non-removable storage devices, including, but not limited to, static RAM (SRAM) or dynamic RAM (DRAM). ROM can be programmable ROM, PROM or EPROM, erasable programmable ROM, or electrically erasable programmable ROM, EEPROM. Storage components suitable for memory can be embedded as chips on a printed circuit board or other substrate connected to one or more processors or processing modules, or can be provided as removable components, such as flash memory (also known as USB sticks), compact discs (CDs), digital versatile discs (DVDs), and any other suitable form of memory. If not suitable for the application at hand, memory can be distributed across various forms of memory and storage components, or provided remotely on one or more servers, such as may be provided by a cloud-based storage solution. Generally, program modules can include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0288] Accordingly, memory used by any form of electronic device described herein includes any suitable device, i.e., readable and / or writable medium, examples of which include, but are not limited to, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (e.g., flash drives, compact discs (CDs), or digital video discs (DVDs)), and / or any other volatile or non-volatile non-transitory, machine-readable and / or computer-executable memory devices that store information, data, and / or instructions that can be used by a processing circuit. The memory can store any suitable instructions, data, or information executable by an application and / or processing circuit, including one or more of computer programs, software, logic, rules, codes, tables, etc., as well as other instructions utilized by the device in any form of electronic device. The memory can be used to store any calculations performed by the processing circuit and / or any data received from a user or via a communication or other type of data interface. In some embodiments, the processing circuit and the memory are integrated. The memory may be distributed among one or more system or device components, for example, the memory may in some embodiments include multiple different memory modules, including modules located on other network nodes.

[0289] While exemplary aspects of the present disclosure have been disclosed in the drawings and specification above, many changes and modifications can be made to these aspects that fall within the scope of the appended claims. Accordingly, the present disclosure is not limited to the specific example aspects and embodiments described above, which should be considered illustrative rather than restrictive with respect to supporting the claims. The present invention, as exemplified herein by the various aspects and embodiments described above, has a scope defined by the following claims.

Claims

1. A device (302) for processing a serialized video data stream for forward transmission, The serialized video data stream includes a plurality of video frames transmitted serially from a video source, each video frame includes a plurality of frame lines (202a to 202e), and at least one frame line (202a to 202d) includes a group of active pixels and one or more blank pixels. The aforementioned device (302) is A receiver (304) configured to receive the serialized video data stream, A memory (306) configured to buffer the received serialized video data until at least all of the valid pixels in the frame line of the received video frame's pixels are buffered, One or more processors or processing circuits (308) are configured to form a data structure for each group of valid pixels in a pixel frame line, having at least the group of valid pixels in the frame line and a frame line position indicator assigned to the group of valid pixels, wherein each data structure includes a group of valid pixels from a received frame line, and at least one blank pixel in the received frame line is excluded. A transmitter (310) configured to intermittently transmit the formed data structures by pausing transmission after each data structure has been transmitted until preparations for starting the transmission of the next data structure are complete, A device (302) equipped with the following:

2. Two or more blank pixels are discarded from each received video frame line that contains valid pixels and at least one blank pixel. The apparatus (302) according to claim 1.

3. None of the blank pixels in the received video frame line are included in the output serialized video stream, which contains the forward-transmitted data structure. The apparatus (302) according to claim 1.

4. The intermittently transmitted video data stream has a fixed frame line length with a variable transmission pause period. The apparatus (302) according to claim 1.

5. The frame line indicator assigned to each group of valid pixels is included in the data structure containing the group of valid pixels. The apparatus (302) according to claim 1.

6. An observation device (312), wherein the observation device (312) is A receiver (410) configured to receive a serialized video data stream containing a plurality of data structures (402) transmitted intermittently, wherein each data structure includes at least a group of valid pixels from a video frame line and a frame line position indicator assigned to the group of valid pixels, One or more processors or processing circuits (1002), Equipped with, The aforementioned one or more processors or processing circuits (1002) For each received data structure, one or more blank pixels are added to each valid pixel group of the received data structure to form a complete video data line until a data structure containing the next group of pixels in the serialized video stream is received. Each complete video data line is output so that it is presented on the display at a position within the video frame indicated by a frame line position indicator assigned to the effective pixels of the complete video data line. It is structured in such a way. Observation device (312).

7. The serialized video stream is received from the apparatus (302) according to any one of claims 1 to 5. The intermittently transmitted and received serialized video data stream is processed by one or more processors or processing circuits (1002) such that a variable number of blank pixels are added for output to the display as a continuous serialized video stream of variable line length. The observation apparatus according to claim 6.

8. The observation device further comprises the display, The observation apparatus (312) according to claim 6.

9. A method for processing a serialized video stream for forward transmission, the method being: Receiving a serialized video stream (314) which includes multiple video frames transmitted serially from a video source, wherein each video frame includes multiple frames and multiple frame lines which include a set of valid pixels and blank pixels, Buffering at least the valid pixels of the received serialized video stream until a complete frame line of pixels is received (318) (316), Processing each complete frame line of buffered pixels, Includes, The aforementioned process is performed by Assigning the frame line position indicator to the group of valid pixels that include the valid pixels of the buffered complete frame line (320), (322) Forming a data structure including the group of valid pixels, which includes a frame line position indicator assigned to the group of valid pixels, wherein each data structure includes the group of valid pixels from the received frame line, and at least one blank pixel of the received frame line is excluded. When each data structure is ready to be transmitted (324), the output serialized video data including the formed data structures is transmitted intermittently forward. Pausing forward transmission of output serialized video data between data structures (328), This is done by method.

10. Each data structure, including the set of valid pixels from the received frame line, is obtained by removing all blank pixels from the received video frame line in the output serialized video data, which includes the forward-transmitted data structures. The method according to claim 9.

11. A video data stream transmitted intermittently forward has a fixed frame line length and a variable transmission pause period. The method according to claim 10.

12. A method for processing a received serialized video stream for display, wherein the method is Receiving an intermittently received serialized video stream containing multiple data structures, each data structure containing at least a set of valid pixels from a video frame line and a frame line position indicator assigned to the set of valid pixels, Until a data structure containing the next set of pixels in the serialized video stream is received, one or more blank pixels are added to each set of valid pixels in the received data structure to form a complete video data line. Each complete video data line is output so that it is presented on the display at a position within the video frame indicated by the frame line position indicator assigned to the effective pixels of the complete video data line. A method that includes this.

13. The intermittent serialized video stream is received from the apparatus (302) according to any one of claims 1 to 5. The intermittently transmitted and received serialized video data is processed by one or more processors or processing circuits (1002) so that a variable number of blank pixels are added for output to a display as a continuous serialized video stream of variable line length. The method according to claim 12 (600).

14. The aforementioned method, Determining that the last frame line of the frame has been received (605a), (605b) The end of a frame line containing blank pixels is formed as the last frame line of the frame, Outputting the received frame line, including the end of the frame line, to the display as a serial data stream of pixels (606), Further including, The method according to claim 12 (600).

15. A computer program product including computer code (500), The computer code (500) is loaded from memory (306) and executed by one or more processors (308) of the device (302) according to any one of claims 1 to 5, causing the device (302) to execute the method according to claim 9 or 10. Computer program products.

Images