METHOD AND SYSTEM FOR TIME CORRECTION OF VIDEO STREAMS AND FOR LATENCY DETECTION

Abstract

A method for video streaming comprising: receiving video streaming data from a wireless camera (50) on a controller (34) of a vehicle (10), which is in wireless communication with the wireless camera (50) via a receiver (40) of the vehicle (10); identifying a stable video range in the video streaming data received from the wireless camera (50); detecting outdated video content in the video streaming data based on the stable video range; and, in response to the detection of outdated video content in the video streaming data, providing a notification to a vehicle occupant via a display device (48) of the vehicle (10), the notification indicating that a video being streamed to the vehicle's display device is outdated;characterized in that, to identify the stable video area in the video streaming data, an average µ of a predetermined number of past keyframe time intervals is determined; wherein, to detect outdated video content, it is checked whether the next keyframe arrival time lies between tnextmax = tn + µ + ε and tnextmin = tn + µ - ε, where tn is a timestamp of an immediately preceding keyframe and ε is a threshold to control the sensitivity of the detection of outdated video content, and if this is not the case: a current amount of video streaming traffic per unit of time is determined; an amount of video streaming traffic per unit of time for a stable video is determined; and the current amount of video streaming traffic per unit of time is compared with the amount of video streaming traffic per unit of time for the stable video;where, in the event that the current amount of video streaming traffic per unit of time is less than the amount of video streaming traffic per unit of time for the stable video, it is determined that outdated video content is present in the video streaming data.

Description

INTRODUCTION

[0001] The present disclosure relates to systems and methods for time correction of video streams and for latency detection.

[0002] This introduction sets out the general content of the disclosure. The work of the presently named inventors, to the extent described in this introduction, as well as those aspects of the description which do not otherwise qualify as prior art at the time of filing, are neither expressly nor implicitly admitted as prior art against the present disclosure.

[0003] Wireless cameras sometimes stream video and audio signals to a display device in a vehicle. However, the wireless camera stream within the vehicle can be delayed. To address this delay, it is desirable to accurately calculate the latency at the frame level. Without support from the camera manufacturer, however, accurately calculating the latency at the frame level is quite challenging. An incorrect representation at the frame level can lead to an error in the playback of the video stream. SUMMARY

[0004] According to one aspect of the present disclosure, the method for video streaming includes receiving video streaming data from a wireless camera. The camera is in wireless communication with a vehicle. The method further includes identifying a stable video region in the video streaming data received from the wireless camera, detecting outdated video content in the video streaming data based on the stable video region, and in response to detecting the outdated video content in the video streaming data, providing a notification to a vehicle occupant via a display device of the vehicle. The notification indicates that a video streamed to the vehicle display device is outdated.The method described in this paragraph improves video and vehicle technology by identifying outdated video content and notifying the vehicle occupant that the video is outdated, thereby preventing the vehicle occupant from relying on outdated video content.

[0005] The method may comprise performing a reconstruction of video frame timestamps, wherein performing the reconstruction of video frame timestamps comprises determining a video frame presentation time of the video streaming data using the following equation: vpt = 1 / r, where r is the video frame rate and vpt is the video frame presentation time of the video streaming data. Identifying the stable video region in the video streaming data received from the wireless camera comprises determining an average of a predetermined number of past keyframe time intervals. Identifying the stable video region in the video streaming data received from the wireless camera comprises using one of the following equations: t nextmax ≤ tn + µ + ε; t nextmin ≥ tn + µ - ε, where t next-max is a maximum timestamp for a next keyframe arrival time; t next-minis a minimum timestamp for a next keyframe arrival time; t n is a timestamp of an immediately preceding keyframe; µ is the average of the predetermined number of past keyframe time intervals; and ε is a threshold to control the sensitivity of the detection of outdated video content. The method may comprise: determining that the next keyframe arrival time is between t nextmax and t nextmin and in response to determining that the next keyframe arrival time is between t nextmax and t nextminDetermining that the video streaming data from the wireless camera is displayed correctly on the vehicle's display. This notification indicates that a video streamed to the vehicle's display is out of date.The method may include: determining a current amount of video streaming traffic per unit time; determining an amount of video streaming traffic per unit time for a stable video; comparing the current amount of video streaming traffic per unit time to the amount of video streaming traffic per unit time for the stable video to determine if the current amount of video streaming traffic per unit time is less than the amount of video streaming traffic per unit time for the stable video; and in response to determining that the current amount of video streaming traffic per unit time is not less than the amount of video streaming traffic per unit time for the stable video, determining that the video streaming data from the wireless camera is correctly displayed on the display device of the vehicle.

[0006] The present disclosure also describes a tangible non-transitory machine-readable medium containing machine-readable instructions that, when executed by a processor, cause the processor to perform the method described above.

[0007] The present disclosure also describes a system including a transmitter and a controller in communication with the transmitter. The controller is programmed to perform the method described above.

[0008] Further areas of applicability of the present disclosure will become apparent from the detailed description provided below. It should be understood that the detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.

[0009] The above-described features and advantages and other features and advantages of the presently disclosed system and method will become apparent from the following detailed description, including the claims and exemplary embodiments, when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure will be more fully understood from the detailed description and the accompanying drawings, in which: Fig. 1 a schematic representation of a system for video streaming; and Fig. 2 a flowchart for a method for video streaming. DETAILED DESCRIPTION

[0011] Reference will now be made in detail to some examples of the disclosure illustrated in the accompanying drawings. Where possible, the same or similar reference numerals are used throughout the drawings and the description to refer to the same or similar parts or steps.

[0012] With reference to Fig. 1, a system 11 for video streaming includes a vehicle 10 and a wireless camera in communication with the vehicle 10. While the wireless camera 50 is shown outside the vehicle 10, it is contemplated that the wireless camera 50 may be located inside the vehicle 10. Regardless of its location, the wireless camera 50 is in wireless communication with the vehicle 10 via a network such as the Internet or using a short-range wireless technology such as BLUETOOTH or Wi-Fi. Although the vehicle 10 is shown as a sedan, it is contemplated that this vehicle 10 may be another type of vehicle such as a pickup truck, a coupe, an SUV, a recreational vehicle (RV), etc.

[0013] The vehicle 10 includes a vehicle controller 34, a receiver 40, and a display device 48. The display device 48 is in communication with the vehicle controller 34. The display device 48 is configured to display images (e.g., video) and may include speakers to emit sound. The receiver 50 is in communication with the vehicle controller 34 and is configured to wirelessly receive video data from the wireless camera 50. The vehicle 10 may additionally include a video decoder 38 (e.g., a hardware-based video decoder) configured to convert baseband video signals to digital video. The video decoder 38 is in communication with the vehicle controller 34.

[0014] The vehicle controller 34 includes at least one vehicle processor 44 and a non-transitory computer-readable vehicle storage device or vehicle storage medium 46. The vehicle processor 44 may be a dedicated processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among multiple processors associated with the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions. The computer-readable vehicle storage device or vehicle storage medium 46 may include, for example, volatile and non-volatile memory in read-only memory (ROM), random access memory (RAM), and retained memory (RAM).KAM is a persistent or non-volatile memory that can be used to store various operating variables while the vehicle processor 44 is powered off. The computer-readable vehicle storage device or vehicle storage medium 46 of the vehicle controller 34 can be implemented using a number of storage devices such as PROMs (programmable read-only memories), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represents executable instructions used by the vehicle controller 34 in controlling the vehicle 10.

[0015] The instructions may include one or more separate programs, each comprising an ordered list of executable instructions for implementing logical functions. When executed by the vehicle processor 44, the instructions receive and process signals from the cameras, perform logic, calculations, methods, and / or algorithms to automatically control the components of the vehicle 10, and generate control signals to the actuators to automatically control the components of the vehicle 10 based on the logic, calculations, methods, and / or algorithms. Although in Fig. 1, a single vehicle controller 34 is shown, the vehicle 10 may include multiple controllers 34 that communicate via a suitable communication medium or combination of communication media and that cooperate to process the sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control features of the system 11.

[0016] The wireless camera 50 is in wireless communication with the vehicle 10 and includes a camera controller 54, a transmitter 58, and a video encoder 60. The transmitter 58 is configured to send video data to the vehicle 10 and is in communication with the camera controller 54. The video encoder 60 is in communication with the camera controller 54 and is configured to compress raw digital video data into compressed digital video signals.

[0017] The camera controller 54 includes at least one camera processor 55 and a non-transitory computer-readable camera storage device or camera storage medium 56. The camera processor 55 may be a dedicated processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among multiple processors associated with the camera controller 54, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions. The vehicle computer-readable storage device or vehicle storage medium 46 may include, for example, volatile and non-volatile memory in read-only memory (ROM), random access memory (RAM), and retained memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operating variables while the camera processor 55 is powered off.The computer-readable camera storage device or camera storage medium 56 of the camera controller 54 may be implemented using a number of storage devices such as PROMs (programmable read-only memories), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represent executable instructions used by the camera controller 54 in controlling the wireless camera 50. The non-transitory computer-readable camera storage device or camera storage medium 56 contains machine-readable instructions (e.g., those described in . Fig. 2) which, when executed by the one or more processors, cause the camera processors 55 to execute the method 100 ( Fig. 2) to be carried out.

[0018] Fig.2 shows a method 100 for video streaming. Specifically, the method 100 can be used for time correction of video streams and for latency detection. The method 100 begins in block 102. In block 102, the vehicle controller 34 receives video streaming data from the wireless camera. As explained above, the wireless camera 50 is in wireless communication with the vehicle controller 34. The method 100 then proceeds to block 104.

[0019] In block 104, the vehicle controller 34 performs video image time stamp reconstruction. This step is optional and may be performed if the video streaming device does not correctly set the video image presentation time stamps, such as with wireless cameras 50 that use the Real-Time Messaging Protocol (RTMP). To perform video image time stamp reconstruction, the video image presentation time of the video streaming data is determined using the following equation: VPT=1 / R where R is the video frame rate; and VPT is the video frame display time.

[0020] In this scenario, it is assumed that the keyframes have a fixed interval. Furthermore, the video servers receive a number of frames at the native streaming rate in the stable video region. After determining the video frame presentation time, the method 100 proceeds to block 106. In block 106, the vehicle controller 34 determines the average of a predetermined number of past keyframe time intervals to identify a stable video region in the video streaming data received from the wireless camera 50. The following equation may be used: μ=avg(l1,⋯,ln) µ is the average of the given number of past keyframe time intervals; I1, ..., I n is the specified number of past keyframe time intervals; n is the specified number.

[0021] After block 106, the method 100 continues with block 108. In block 108, the vehicle controller 34 determines when the next keyframe should arrive. To do so, the vehicle controller 34 may use the following equations: tnextmax≤tn+μ+∈ tnextmin≥tn+μ−∈ where: t nextmax is a maximum timestamp for the next keyframe arrival time; t nextmin is a minimum timestamp for the next keyframe arrival time; t n is a timestamp of an immediately preceding keyframe; µ is the average of the specified number of past keyframe time intervals; and ε is a threshold to control the sensitivity of detecting outdated video content.

[0022] If the previous keyframe at time t n arrives, the next keyframe should be between [t n + µ - ε, tn + µ + ε], where ε is a threshold to control the sensitivity of the detection of outdated video content. The vehicle controller 34 determines whether the next keyframe arrival time is between t nextmax and t nextmin If the next keyframe arrival time is between t nextmax and t nextmin , then the method 100 continues to block 110. In block 110, the vehicle controller 34 determines that the video streaming data received from the wireless camera 50 is correctly displayed on the display device (ie, the video is in good condition). If the next keyframe arrival time is not between t nextmax and t nextmin then the method 100 continues with block 112.

[0023] In block 112, the vehicle controller 34 determines the amount of video streaming traffic per unit time for a stable video (known based on the original video streaming data) and the current amount of video streaming traffic per unit time. The vehicle controller 34 then compares the current amount of video streaming traffic per unit time with the amount of video streaming traffic per unit time for the stable video to determine if the current amount of video streaming traffic per unit time is less than the amount of video streaming traffic per unit time for the stable video. If the current amount of video streaming traffic per unit time is not less than the amount of video streaming traffic per unit time for the stable video, the method 100 proceeds to block 110.As explained above, in block 110, the vehicle controller 34 determines that the video streaming data received from the wireless camera 40 is displayed correctly (i.e., the video is in good condition). If the current amount of video streaming traffic per unit time is less than the amount of video streaming traffic per unit time for the stable video, then the method 100 proceeds to block 114. In block 114, the vehicle controller 34 determines that outdated video content is present in the video streaming data and, in response, commands the display device 48 to provide a notification to the vehicle occupant. The notification indicates that a video streamed to the vehicle display device is outdated.The notification may be provided via display device 48 and may be provided whenever the video stream delay is greater than a video stream delay threshold. The video stream delay may be determined using the following equation:. ΔT=t′−(tn+μ) where: ΔT is the delay of the video stream; t n is a timestamp of an immediately preceding keyframe; and µ is the average of the given number of time intervals of past keyframes.

[0024] While exemplary embodiments are described above, these embodiments are not intended to describe all possible forms encompassed by the claims. The words used in the application text are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the disclosure. As previously described, the features of various embodiments may be combined to form further embodiments of the presently disclosed system and method, which need not be expressly described or illustrated.While various embodiments may have been described as providing advantages or being preferred over other prior art embodiments or implementations with respect to one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system characteristics depending on the specific application and implementation. These features may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, usability, weight, manufacturability, ease of assembly, etc.Therefore, embodiments that are described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics are not outside the scope of the disclosure and may be desirable for certain applications.

[0025] The drawings are presented in simplified form and are not exactly to scale. For convenience and clarity only, directional terms such as upper, lower, left, right, upward, over, above, below, beneath, rear, and front may be used with reference to the drawings. These and similar directional terms should not be construed as limiting the scope of the disclosure in any way.

[0026] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and further embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the presently disclosed system and method.As will be understood by those skilled in the art, various features illustrated and described with reference to any of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments for typical applications. However, various combinations and modifications of the features consistent with the teachings of this disclosure may be desirable for particular applications or implementations.

[0027] Embodiments of the present disclosure may be described herein in terms of functional and / or logical block components and various processing steps. It should be appreciated that such block components may be implemented by multiple hardware, software, and / or firmware components configured to perform the specified functions. For example, an embodiment of the present disclosure may employ various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, lookup tables, or the like, capable of performing a variety of functions under the control of one or more microprocessors or other control devices.In addition, those skilled in the art will understand that embodiments of the present disclosure may be practiced in connection with multiple systems and that the systems described herein are merely exemplary embodiments of the present disclosure.

[0028] For brevity, techniques related to signal processing, data fusion, signaling, control, and other functional aspects of the systems (and the individual operational components of the systems) need not be described in detail here. Furthermore, the interconnections shown in the various figures included herein are intended to represent examples of functional relationships and / or physical couplings between the various elements. It should be noted that alternative or additional functional relationships or physical connections may be present in an embodiment of the present disclosure.

[0029] This description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure may be implemented in various forms. Therefore, while this disclosure contains specific examples, the true scope of the disclosure should not be so limited, since other modifications will become apparent upon a study of the drawings, the specification, and the following claims.

Claims

[1] Video streaming method that includes: Receiving video streaming data from a wireless camera, the wireless camera being in wireless communication with a vehicle; Identifying a stable video region in the video streaming data received from the wireless camera; Detecting stale video content in the video streaming data based on the stable video range; and in response to detecting the outdated video content in the video streaming data, providing a notification to a vehicle occupant via a display device of the vehicle, the notification indicating that a video streamed to the display device of the vehicle is outdated. [2] The method of claim 1, further comprising performing reconstruction of the video frame timestamps, wherein performing reconstruction of the video frame timestamps comprises determining a video frame presentation time of the video streaming data using the following equation: VPT=1 / R, where R is the video frame rate and VPT is the video frame presentation time of the video streaming data. [3] The method of claim 2, wherein identifying the stable video region in the video streaming data received from the wireless camera comprises determining an average of a predetermined number of past key frame time intervals. [4] The method of claim 3, wherein identifying the stable video region in the video streaming data received from the wireless camera comprises using one of the following equations: tnextmax≤tn+μ+∈ tnextmin≤tn+μ−∈ where t nextmax is a maximum timestamp for the next keyframe arrival time; t nextmin is a minimum timestamp for the next keyframe arrival time; t n is a timestamp of an immediately preceding keyframe; µ is the average of the given number of time intervals of past key frames; and E a threshold to control the sensitivity of detecting outdated video content. [5] The method of claim 4, further comprising: Determine that the next keyframe arrival time is between t nextmax and t nextmin lies; and in response to determining that the next keyframe arrival time is between t nextmax and t nextmin Check that the video streaming data from the wireless camera is displayed correctly on the vehicle's display. [6] The method of claim 5, further comprising: Determining a current amount of video streaming traffic per unit of time; Determining an amount of video streaming traffic per unit time for a stable video; Comparing the current amount of video streaming traffic per unit time with the amount of video streaming traffic per unit time for the stable video to determine whether the current amount of video streaming traffic per unit time is less than the amount of video streaming traffic per unit time for the stable video; and in response to determining that the current amount of video streaming traffic per unit time is less than the amount of video streaming traffic per unit time for the stable video, providing the notification to the vehicle occupant via the vehicle display device, the notification indicating that a video streamed to the vehicle display device is out of date. [7] The method of claim 6, further comprising: Determining a current amount of video streaming traffic per unit of time; Determining an amount of video streaming traffic per unit time for a stable video; Comparing the current amount of video streaming traffic per unit time with the amount of video streaming traffic per unit time for the stable video to determine whether the current amount of video streaming traffic per unit time is less than the amount of video streaming traffic per unit time for the stable video; and in response to determining that the current amount of video streaming traffic per unit time is not less than the amount of video streaming traffic per unit time for the stable video, determining that the video streaming data from the wireless camera is correctly displayed on the display device of the vehicle. [8] Tangible non-transitory machine-readable medium containing machine-readable instructions which, when executed by a processor, cause the processor to: receive video streaming data from a wireless camera, the wireless camera being in wireless communication with a vehicle; to identify a stable video region in the video streaming data received from the wireless camera; detect outdated video content in the video streaming data based on the stable video range; and in response to detecting the outdated video content in the video streaming data, provide a notification to a vehicle occupant via a display device of the vehicle, the notification indicating that a video streamed to the display device of the vehicle is outdated. [9] The tangible non-transitory machine-readable medium of claim 8, wherein the tangible non-transitory machine-readable medium further includes machine-readable instructions that, when executed by a processor, cause the processor to: perform reconstruction of the video frame timestamps, wherein performing reconstruction of the video frame timestamps comprises determining a video frame presentation time of the video streaming data using an equation: VPT=1 / R, where R is the video frame rate and VPT is the video frame presentation time of the video streaming data. [10] The tangible non-transitory machine-readable medium of claim 9, wherein the tangible non-transitory machine-readable medium further includes machine-readable instructions that, when executed by a processor, cause the processor to: to determine an average of a given number of past keyframe time intervals.