SYSTEMS, METHOD AND DEVICES FOR VIDEO STREAM SYNCHRONIZATION

Abstract

Systems, methods, and devices perform synchronization operations for video streams. Methods include receiving a video frame at a frame capture unit, wherein the video frame is contained in a stream of video data; buffering the video frame using a buffer coupled between the frame capture unit and a frame display unit; and determining an offset value based at least partially on an update rate associated with the frame capture unit. Methods further include reading the video frame from the buffer using the frame display unit after a specified time period defined by the offset value.

Description

TECHNICAL AREA

[0001] This disclosure relates to video streaming and, in particular, to improving the synchronization of components used in such video streaming. BACKGROUND

[0002] Video display systems and devices can utilize video streaming to transmit such video data from a video source to a video display. Accordingly, a stream of video data can be received as a video input, and one or more components can process the video data before providing it as a video output to a video display. Conventional techniques for processing a received video input and preparing it for display remain limited because they use multiple buffers and additional hardware resources to perform warping and scaling operations on the video output generated for the video display. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates an example of a video streaming system configured according to some embodiments. Fig. Figure 2 illustrates an example of a method for video stream synchronization, which is carried out according to some embodiments. Fig. Figure 3 illustrates another example of a method for video stream synchronization, which is carried out according to some embodiments. Fig. Figure 4 illustrates an additional example of a method for video stream synchronization, which is carried out according to some embodiments. Fig. Figure 5 illustrates another example of a method for video stream synchronization, which is carried out according to some embodiments. DETAILED DESCRIPTION

[0003] The following description presents numerous specific details to provide a thorough understanding of the concepts presented. The concepts presented can be implemented without some or all of these specific details. In other cases, well-known process operations have not been described in detail to avoid unnecessarily obscuring the concepts described. While some concepts are described in conjunction with specific examples, it is understood that these examples are not intended to be restrictive.

[0004] The embodiments disclosed herein provide efficient buffering for processing video streams. As discussed in more detail below, a single buffer can be used to buffer video data for a video stream under a variety of operating conditions. For example, a single buffer can be used even if there are differences in the input and output sizes of video data or if there are different numbers of channels. Accordingly, the embodiments disclosed herein avoid the use of multiple buffers while ensuring that synchronization between an input stream and an output stream is maintained. In this way, efficient video streaming modalities are provided that utilize reduced hardware resources.

[0005] Fig. Figure 1 illustrates an example of a video streaming system configured according to several embodiments. As discussed in more detail below, a system such as System 100 can be implemented to perform efficient synchronization for video streaming components. In particular, synchronization operations and time offsets can be used to ensure that a video frame capture unit and a video frame display unit remain synchronized and do not experience errors and / or video data artifacts.

[0006] In various embodiments, the system 100 comprises the vehicle 101. Accordingly, embodiments disclosed herein can be implemented in an operating environment, which may be a vehicle, such as an automobile. In various embodiments, the vehicle 101 comprises the main unit 102, which may be a main unit of an infotainment system. In various embodiments, the main unit 102 supports various applications and associated functionalities, such as navigation and geolocation, video camera displays, and various other vehicle information systems. Accordingly, the main unit 102 comprises the host processor 104 and the storage system 106, which are configured to execute applications underlying these functions and to store data associated with such applications.

[0007] In various embodiments, the main unit 102 additionally includes the display engine 108. In these embodiments, the display engine 108 is an augmented reality engine configured to receive data from components of the main unit 102 and generate a stream of video data that forms an augmented reality display of such data, which can be shown in a head-up display (HUD) of the vehicle 101. As discussed in more detail below, such a HUD can be configured to project a reflected image onto a windshield of the vehicle 101, so that a user sitting in a driver's seat of the vehicle 101 is able to see the projected image as a superimposition of what the user sees through the windshield. Accordingly, the display engine 108 is configured to generate video data that is projected onto the inner surface of the windshield.It goes without saying that any suitable rendering engine can be used to generate the video data. In particular, any suitable augmented reality software configured to generate video data for an augmented reality display can be used.

[0008] In various embodiments, the system 100 additionally comprises the HUD projector module 110, which can be configured to receive video data from the display engine 108 and can further be configured to generate a video output stream that is provided to one or more displays. In particular, the HUD projector module 110 can include the processing device 112, which can include one or more processors configured to perform buffering and synchronization operations disclosed herein. In one example, the processing device 112 includes processing elements configured to implement a frame capture unit, such as the capture unit 113, and a frame display unit, such as the display unit 119, which can handle input and output operations associated with video frames.Furthermore, the processing device 112 can include the buffer 114, which is configured to buffer a received video frame. As discussed in more detail below, the processing device 112 is configured to manage the operation of the frame capture unit and a frame display unit, ensuring synchronization of an input video stream and an output video stream, and utilizing a single buffer. The processing device 112 can also include one or more video processing layers, such as layer 116 and layer 118. In one example, layer 116 and layer 118 can be warping layers configured to perform warping operations on video data to prepare it for display on, for example, a windshield.

[0009] The processing device 112 can provide generated video outputs to one or more displays, such as the display module 120 and the display module 122. In various embodiments, the display module 120 and the display module 122 are projector elements configured to receive a stream of video data and project the video data via one or more optical elements, which may include, for example, one or more lenses and mirrors contained in an optical projection system, such as the optical projection system 124. Accordingly, the display module 120 and the display module 122 can each display video data that is part of a HUD.In various embodiments, the display module 120 and the display module 122 are configured such that the display module 120 displays a near image, which is perceived as being close to the user, and the display module 122 displays a far image, which is perceived as being farther away from the user.

[0010] Fig. Figure 2 illustrates an example of a video stream synchronization method performed according to some embodiments. As discussed in more detail below, a method such as Method 200 can be used to synchronize video streaming components. In particular, the use of a buffer can be managed such that a single buffer can be used while ensuring that a video frame capture unit and a video frame display unit remain synchronized and do not experience errors and / or video data artifacts.

[0011] Method 200 can perform operation 202, during which a video frame can be received at a frame capture unit. In various embodiments, the frame capture unit can retrieve video data frames from a received video stream and can provide the video data frame to downstream components used for video data processing. Accordingly, the frame capture unit can operate as a retrieval unit that has retrieved a video data frame, for example, when a buffer level indicates that a frame should be retrieved. As discussed in more detail below, a synchronization operation can be implemented in response to the received video frame. Such a synchronization operation can synchronize a pixel clock of the frame capture unit with a pixel clock of the frame display unit.

[0012] Method 200 can perform operation 204, during which the frame can be buffered using a buffer coupled to the frame acquisition unit. Accordingly, the received video frame can be provided to and stored in the buffer. In various embodiments, the buffer size can be configured to be large enough to store one video frame.

[0013] Method 200 can perform operation 206, during which the frame can be read from the buffer at a frame display unit after a specified time period. Accordingly, the frame display unit can be configured to read the video frame from the buffer and provide the read data to a downstream component, such as a warping layer. In various embodiments, the frame display unit's read operation is delayed by a specified time period from the corresponding write operation of the frame capture unit. As discussed in more detail below, the specified time period can be a calculated offset value, determined to ensure that the video output and video input do not become desynchronized.

[0014] Fig. Figure 3 illustrates another example of a video stream synchronization method performed according to some embodiments. As discussed in more detail below, a method such as Method 300 can be used to synchronize video streaming components. In particular, the use of a buffer can be managed such that a single buffer can be used while ensuring that a video frame capture unit and a video frame display unit remain synchronized and do not experience errors and / or video data artifacts.

[0015] Procedure 300 can perform operation 302, during which a video frame can be received at a frame capture unit. As discussed similarly above, the frame capture unit can retrieve video data frames from a received video stream and provide the video data frame to downstream components used for video data processing. Accordingly, during operation 302, the next video frame can be received, and the frame capture unit can prepare to begin writing the video frame to the buffer.

[0016] Method 300 can perform operation 304, during which a synchronization operation can be performed for the frame capture unit and a frame display unit. In various embodiments, the synchronization operation is performed in response to the received video frame. Furthermore, the synchronization operation can synchronize a pixel clock of the frame capture unit with a pixel clock of the frame display unit. In some embodiments, the synchronization operation is a VSynch operation. Additionally, a video-in update rate can be set to be the same as a video-out update rate. Accordingly, the update rates of the frame capture unit and the frame display unit can be set to be the same.

[0017] Method 300 can perform operation 306, during which a specific offset value can be determined. In various embodiments, a specific offset value can be calculated to ensure that synchronicity and stability are maintained for video frame processing. The offset value can be determined based on one or more parameters assigned to downstream components. For example, the number of hold lines specified by a downstream warping layer can be used to calculate the specific offset value. In this example, the specific offset value is specified as a number of buffer lines.

[0018] In another example, the specified offset value can be calculated based on a refresh rate or other buffer parameters. For instance, if the refresh rate is 60 Hz, a reference value of 1 / 60 can be used as a limit. Accordingly, the specified offset value can be limited by this value and can be set to a value of 0 ms < T_Offset < 16.6 ms. In this example, the specified offset value between the limits can be set to a value of 8.3 s.

[0019] Method 300 can perform operation 308, during which the video frame can be read from the buffer at the frame display unit after a specific time period, identified by the specified offset value. As discussed similarly above, the frame display unit can be configured to read the video frame from the buffer and provide the read data to a downstream component, such as a warping layer. In various embodiments, the frame display unit's read operation is delayed by a specific time period from the corresponding write operation of the frame capture unit. The specified time period can be the specified offset value calculated during operation 306.

[0020] Method 300 can perform operation 310, during which it can be determined whether an offset compensation operation should be performed. In various embodiments, such a determination can be made by monitoring the value of the offset value and the pixel clock values ​​of the frame acquisition unit and the frame display unit to see if their clocks are drifting too far apart. Such a determination can be made based on a comparison of the difference in their timings with a specific threshold that can be set to the offset value. Accordingly, if the difference in clock values ​​exceeds the offset value, it can be determined that a compensation operation can be performed.

[0021] Method 300 can perform operation 312, during which an offset compensation operation can be performed. In various embodiments, the offset compensation operation can be an additional synchronization operation. Accordingly, an additional synchronization operation can be performed, as discussed similarly above with reference to operation 304, to synchronize the pixel clocks of the frame acquisition unit and the frame display unit and restore their synchronicity.

[0022] Fig. Figure 4 illustrates an additional example of a video stream synchronization method performed according to some embodiments. As discussed in more detail below, a method such as Method 400 can be used to synchronize video streaming components. In particular, the use of a buffer can be managed such that a single buffer can be used while ensuring that a video frame capture unit and a video frame display unit remain synchronized and do not experience errors and / or video data artifacts.

[0023] Procedure 400 can perform operation 402, during which a video frame can be received at a frame capture unit. As discussed similarly above, the frame capture unit can retrieve video data frames from a received video stream and provide the video data frame to downstream components used for video data processing. Accordingly, during operation 402, the next video frame can be received, and the frame capture unit can prepare to begin writing the video frame to the buffer.

[0024] Method 400 can perform operation 404, during which it can be determined that a video input size is smaller than a video output size. In various embodiments, such a determination can be made based on known resolutions of the frame capture unit and the frame display unit. Accordingly, the known resolutions can be compared, and the appropriate determination can be made.

[0025] Method 400 can perform operation 406, during which a synchronization operation can be performed for the frame capture unit and a frame display unit. In various embodiments, the synchronization operation is performed in response to the received video frame. Furthermore, the synchronization operation can synchronize a pixel clock of the frame capture unit with a pixel clock of the frame display unit. In some embodiments, the synchronization operation is a VSynch operation.

[0026] Procedure 400 can perform operation 408, during which a specific offset value can be determined based on a variety of reference values. For example, the reference values ​​can include a first reference value T_Value1 and a second reference value T_Value2. The determined offset value can be limited by the two reference values ​​such that T_Value1 < T_Offset < T_Value2. In this example, the determined offset value can be set halfway between the reference values. Furthermore, the reference values ​​can be determined based on equations 1-3 shown below: T=1 / f T_Value1=(720−(k+480)) / 720*T T_Value2=(720−k) / 720*T

[0027] In equations 1-3 shown above, a video input can have 480 lines, and a video output can have 720 lines. Furthermore, f can be a refresh rate, which might be, for example, 60 Hz. Additionally, k can be an identifier or index value that identifies a buffer line storing the first line of a frame. Accordingly, a received video input can have a video frame containing 480 lines stored in the buffer and ultimately be displayed in an output frame containing 720 lines. The k line can identify a position, such as a line from which the display of the buffer contents should begin. In this example, T_Value1 can be a time relative to the receipt of the next video frame and can be a reference value that limits the specific offset value so that it does not overlap with the activity of the display unit.Furthermore, T_Value2 can be a time relative to the reception of the next video frame and can be a reference value that limits the specified offset value, preventing it from overlapping with the activity of the capture unit. As discussed above, T_Offset can be set between T_Value1 and T_Value2. In this way, the specified offset value can prevent buffer accesses from crossing between the capture unit and the display unit, while also tolerating some variation in the behavior of the capture unit and the display unit.

[0028] Method 400 can perform operation 410, during which the video frame can be read from the buffer at the frame display unit after a specified time period, identified by the specified offset value. As discussed similarly above, the frame display unit can be configured to read the video frame from the buffer and provide the read data to a downstream component, such as a warping layer. In various embodiments, the frame display unit's read operation is delayed by a specified time period from the corresponding write operation of the frame capture unit. The specified time period can be the specified offset value calculated during operation 408.

[0029] Method 400 can perform operation 412, during which an offset compensation operation can be performed. In various embodiments, the offset compensation operation can be an additional synchronization operation. Accordingly, if it is determined that an offset compensation operation should be performed, an additional synchronization operation can be performed to synchronize the pixel clocks of the frame acquisition unit and the frame display unit.

[0030] Fig.Figure 5 illustrates another example of a video stream synchronization method performed according to some embodiments. As discussed in more detail below, a method such as Method 500 can be used to synchronize video streaming components. In particular, the use of a buffer can be managed such that a single buffer can be used while ensuring that a video frame capture unit and a video frame display unit remain synchronized and do not experience errors and / or video data artifacts. As discussed in more detail below, such synchronization can also be maintained for multiple display outputs.

[0031] Procedure 500 can perform operation 502, during which a video frame can be received at a frame capture unit. As discussed similarly above, the frame capture unit can retrieve video data frames from a received video stream and provide the video data frame to downstream components used for video data processing. Accordingly, during operation 302, the next video frame can be received, and the frame capture unit can prepare to begin writing the video frame to the buffer.

[0032] Procedure 500 can perform Operation 504, during which it can be determined that a video output has more video channels than a video input. For example, a video input might have a single channel, and a video output might have multiple channels. In an example where input is received from an augmented reality engine, the input might have one data channel but could ultimately be split into two channels for different display modules capable of projecting near and far images.

[0033] Method 500 can perform operation 506, during which a synchronization operation can be performed for the frame capture unit and a frame display unit. In various embodiments, the synchronization operation is performed in response to the received video frame. Furthermore, the synchronization operation can synchronize a pixel clock of the frame capture unit with a pixel clock of the frame display unit. In some embodiments, the synchronization operation is a VSynch operation.

[0034] Procedure 500 can perform operation 508, during which a first offset value can be determined based on a first set of reference values ​​for a first display output channel. In one example, the reference value T_Value2 can be used to limit the first offset value. For instance, the first offset value can be determined such that 0 < T_Offset1 < T_Value2. In this example, the first offset value can be set halfway between the reference values. In various embodiments, T_Value2 can be determined based on the following equations 4, 5, and 6: Tk=1 / f T_Value1=k / 1200*T T_Value2=j / 1200*T

[0035] In equations 4, 5, and 6 shown above, a video input can have a video frame comprising 1200 lines, and a video output can have 720 lines. Furthermore, f can be a refresh rate, which might be, for example, 60 Hz. As discussed similarly above, k can represent an index value that identifies a buffer line, and j can be k + 720. Accordingly, if k = 0, it is understood that T_Value1 is 0 and T_Value2 = 720 / 1200 * T. Furthermore, T_Value2 can be used to determine T_Offset1 by setting T_Offset1 halfway between T_Value2 and T_Value, which can be 0. Finally, T_Offset1 can be used for an initial display output, which can have two display output channels, as might be implemented in an augmented reality display.

[0036] Procedure 500 can perform operation 510, during which a second offset value can be determined based on a second set of reference values ​​for a second display output channel. For example, the reference value T_Value1 can be used to limit the second offset value. The second offset value can be determined, for instance, such that T_Value < T_Offset2 < 16.6 ms. In this example, the second offset value can be set halfway between T_Value1 and T, as given by Equation 6 below. Thus, in various embodiments, T_Value can be determined based on Equations 6, 7, and 8 below: T=1 / f T_Value1=k / 1200*T T_Value2=j / 1200*T=T

[0037] In equations 7, 8, and 9 shown above, a video input can consist of a video frame with 1200 lines, and video outputs can consist of 720 lines. Furthermore, f can be a refresh rate, which might be, for example, 60 Hz. As discussed similarly above, k can represent an index value that identifies a buffer line. In this example, k can be 720, and j can be 1200. Accordingly, T_Valuel can be 720 / 1200*T, and T_Value2 can be 1200 / 1200*T, which is also expressed as T. As discussed above, T_Valuel can be used to determine T_Offset2, which can be set halfway between T_Valuel and T. Similarly, T_Offset2 can be used for a second display output, which can also have two display output channels.

[0038] Method 500 can perform operation 512, during which one or more offset compensation operations may be performed. In various embodiments, the offset compensation operation may be an additional synchronization operation. Accordingly, if it is determined that an offset compensation operation should be performed, an additional synchronization operation may be performed to synchronize the pixel clocks of the frame acquisition unit and the frame display unit.

[0039] Although the foregoing concepts have been described in detail for the purpose of understanding, it is obvious that certain changes and modifications can be made within the scope of the appended claims. It should be noted that there are many alternative ways to implement the processes, systems, and devices. Accordingly, the examples provided here should be considered illustrative and not limiting.

Claims

[1] A procedure that includes the following: Receiving a video frame at a frame capture unit, wherein the video frame is contained in a stream of video data; Buffering the video frame using a buffer coupled between the frame capture unit and a frame display unit; Determining an offset value that is at least partially based on an update rate associated with the frame capture unit; and Reading, using the frame display unit, the video frame from the buffer after a specific time period defined by the offset value. [2] The method according to claim 1, further comprising: Synchronizing a pixel clock of the frame capture unit and the frame display unit. [3] Method according to claim 1, wherein the buffering further comprises: Writing, using the frame capture unit, the video frame into a multitude of lines of the buffer. [4] The method according to claim 1, further comprising: Perform a compensation operation to update the offset value. [5] Method according to claim 1, wherein the size of a video input to the frame capture unit is smaller than the size of a video output to the frame display unit. [6] The method according to claim 5, further comprising: Determining the offset value based on a variety of reference values. [7] Method according to claim 1, wherein the number of channels included in a video input of the frame capture unit is smaller than the number of channels included in a video output of the frame display unit. [8] The method according to claim 7, further comprising: Determining the offset value based on an initial set of reference values; and Determining an additional offset value based on a second set of reference values. [9] Method according to claim 1, wherein the video data stream includes augmented reality video data. [10] A system that includes the following: a storage device that includes a buffer, wherein the buffer is configured to buffer video data; and a processing device comprising a frame capture unit and a frame display unit, wherein the processing device is configured for the following: Receiving a video frame at the frame capture unit, wherein the video frame is contained in a stream of video data; Determining an offset value that is at least partially based on an update rate associated with the frame capture unit; and Reading, using the frame display unit, the video frame from the buffer after a specific time period defined by the offset value. [11] System according to claim 10, wherein the processing device is further configured for the following: Synchronizing a pixel clock of the frame capture unit and the frame display unit. [12] System according to claim 10, wherein the size of a video input to the frame capture unit is smaller than the size of a video output to the frame display unit. [13] System according to claim 12, wherein the processing device is further configured for the following: Determining the offset value based on a variety of reference values. [14] System according to claim 10, wherein the number of channels included in a video input of the frame capture unit is less than the number of channels included in a video output of the frame display unit. [15] System according to claim 14, wherein the processing device is further configured for the following: Determining the offset value based on an initial set of reference values; and Determining an additional offset value based on a second set of reference values. [16] A device comprising the following: Processing elements configured for the following: Receiving a video frame at the frame capture unit, wherein the video frame is contained in a stream of video data; Buffering the video frame using a buffer coupled between the frame capture unit and a frame display unit; Determining an offset value that is at least partially based on an update rate associated with the frame capture unit; and Reading, using the frame display unit, the video frame from the buffer after a specific time period defined by the offset value. [17] Device according to claim 16, wherein the processing elements are further configured for the following: Synchronizing a pixel clock of the frame capture unit and the frame display unit. [18] Device according to claim 16, wherein the size of a video input to the frame capture unit is smaller than the size of a video output to the frame display unit, and wherein the processing elements are further configured for the following: Determining the offset value based on a variety of reference values. [19] Device according to claim 16, wherein the number of channels included in a video input of the frame capture unit is smaller than the number of channels included in a video output of the frame display unit. [20] Device according to claim 19, wherein the processing elements are further configured for the following: Determining the offset value based on an initial set of reference values; and Determining an additional offset value based on a second set of reference values.

Images