Method and system for video stream time correction and delay detection

Abstract

A method for video streaming includes receiving video streaming data from a wireless camera. The camera is in wireless communication with the vehicle. The method further includes identifying a stable video region in the video streaming data received from the wireless camera, detecting expired video content in the video streaming data based on the stable video region, and providing a notification to the vehicle occupant through the vehicle display in response to detecting the expired video content in the video streaming data. The notification indicates that the online playing video in the vehicle display has expired. The method described in this segment improves video and vehicle technology by identifying expired video content and notifying the vehicle occupant that the video is expired, thereby preventing the vehicle occupant from relying on the expired video content.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for video stream time correction and delay detection. Background Art

[0002] This introduction generally introduces the background of the present disclosure. No work by the presently designated inventors is admitted, either expressly or impliedly, to be prior art to the present disclosure to the extent it is described in this introduction and to the extent that it may not qualify as prior art at the time of filing.

[0003] Wireless cameras sometimes transmit video and audio to a vehicle display. However, in-vehicle wireless camera streams may be delayed. To account for this delay, accurate calculation of frame-level latency is required. However, accurate calculation of frame-level latency is quite challenging without support from the camera manufacturer. Incorrect frame-level representation may cause video stream playback failure. Summary of the invention

[0004] In one aspect of the present disclosure, a method for video streaming includes receiving video streaming data from a wireless camera. The camera is in wireless communication with a vehicle. The method also includes identifying a stable video area in the video streaming data received from the wireless camera, detecting expired video content in the video streaming data based on the stable video area, and providing a notification to a vehicle occupant via a vehicle display in response to detecting the expired video content in the video streaming data. The notification indicates that a video playing online in the vehicle display has expired. The method described in this paragraph improves video and vehicle technology by identifying expired video content and notifying the vehicle occupant that the video is expired, thereby preventing the vehicle occupant from relying on expired video content.

[0005] The method may include performing video frame timestamp reconstruction, wherein performing video frame timestamp reconstruction includes determining a video frame presentation time of the video streaming data using the following equation: vpt=1 / r, wherein r is the video frame rate and vpt is the video frame presentation time of the video streaming data. Identifying a stable video region in the video streaming data received from the wireless camera includes determining an average of a predetermined number of past key frame time intervals. Identifying a stable video region in the video streaming data received from the wireless camera includes using the following equation: t nextmax ≤t n +μ+∈,t nextmin ≥t n +μ-∈, where t nextmax is the maximum timestamp of the next key frame arrival time, t nextmin is the minimum timestamp of the next key frame arrival time, t nis the timestamp of the immediately preceding key frame, μ is the average of a predetermined number of past key frame time intervals, and ∈ is a threshold value for controlling the sensitivity of outdated video content detection. The method may include: determining the next key frame arrival time at t nextmax and t nextmin and in response to determining that the next key frame arrival time is between t nextmax and t nextmin It is determined that the video streaming data from the wireless camera is being correctly displayed in the vehicle display. The notification indicates that the video played online in the vehicle display has expired. The method may include: determining the current video streaming traffic per unit time; determining the video streaming traffic per unit time of the stable video; comparing the current video streaming traffic per unit time with the video streaming traffic per unit time of the stable video to determine whether the current video streaming traffic per unit time is less than the video streaming traffic per unit time of the stable video; and in response to determining that the current video streaming traffic per unit time is not less than the video streaming traffic per unit time of the stable video, determining that the video streaming data from the wireless camera is being correctly displayed on the vehicle display.

[0006] The present disclosure also describes a tangible, non-transitory machine-readable medium including machine-readable instructions, which, when executed by a processor, cause the processor to perform the above method.

[0007] The present disclosure also describes a system, which includes a transmitter and a controller in communication with the transmitter. The controller is programmed to execute the above method.

[0008] Further areas of applicability of the present disclosure will become apparent from the detailed description provided hereinafter.It should be understood that the detailed description and specific examples are intended for illustration purposes only and are not intended to limit the scope of the present disclosure.

[0009] The above features and advantages and other features and advantages of the presently disclosed systems and methods are apparent from the detailed description, including the claims and exemplary embodiments, when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure will be more fully understood from the detailed description and accompanying drawings, wherein:

[0011] Figure 1 is a schematic diagram of a system for video streaming; and

[0012] Figure 2 is a flow chart of a method for video streaming. DETAILED DESCRIPTION

[0013] Reference will now be made in detail to several examples of the present disclosure as illustrated in the accompanying drawings. Whenever possible, the same or similar reference numerals are used in the drawings and description to refer to the same or similar parts or steps.

[0014] refer to Figure 1 , a system 11 for video streaming includes a vehicle 10 and a wireless camera in communication with the vehicle 10. Although the wireless camera 50 is shown as being outside the vehicle 10, it is contemplated that the wireless camera 50 may be inside the vehicle 10. Regardless of its location, the wireless camera 50 communicates wirelessly with the vehicle 10 via a network (e.g., the Internet) or using short-range wireless technology (e.g., Bluetooth or Wi-Fi). Although the vehicle 10 is shown as a sedan, it is contemplated that the vehicle 10 may be another type of vehicle, such as a pickup truck, a coupe, a sport utility vehicle (SUV), a recreational vehicle (RV), etc.

[0015] The vehicle 10 includes a vehicle controller 34, a receiver 40, and a display 48. The display 48 is in communication with the vehicle controller 34. The display 48 is configured to present images (e.g., video) and may include a speaker for emitting sound. The receiver 40 is in communication with the vehicle controller 34 and is configured to wirelessly receive video data from a wireless camera 50. The vehicle 10 may also include a video decoder 38 (e.g., a hardware-based video decoder). The video decoder 38 is configured to convert a baseband video signal into a digital video. The video decoder 38 is in communication with the vehicle controller 34.

[0016] The vehicle controller 34 includes at least one vehicle processor 44 and a vehicle non-transitory computer readable storage device or medium 46. The vehicle processor 44 can be a custom processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor in a plurality of processors associated with the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions. The vehicle computer readable storage device or medium 46 can include volatile and non-volatile memory such as read-only memory (ROM), random access memory (RAM), and keep alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the vehicle processor 44 is powered off. The vehicle computer readable storage device or medium 46 of the vehicle controller 34 can be implemented using a variety of storage devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or other electrical, magnetic, optical or combination storage devices capable of storing data, some of which represent executable instructions used by the vehicle controller 34 when controlling the vehicle 10.

[0017] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by the vehicle processor 44, the instructions receive and process signals from the camera, execute logic, calculations, methods and / or algorithms for automatically controlling components of the vehicle 10, and generate control signals to actuators to automatically control components of the vehicle 10 based on the logic, calculations, methods and / or algorithms. Although Figure 1 A single vehicle controller 34 is shown, but the vehicle 10 may include multiple controllers 34 that communicate via a suitable communication medium or combination of communication media and cooperate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals to automatically control features of the system 11.

[0018] The wireless camera 50 communicates wirelessly with the vehicle 10 and includes a camera controller 54, a transmitter 58, and a video encoder 60. The transmitter 58 is configured to transmit video data to the vehicle 10 and communicates with the camera controller 54. The video encoder 60 communicates with the camera controller 54 and is configured to compress raw digital video into a compressed digital video signal.

[0019] The camera controller 54 includes at least one camera processor 55 and a camera non-transitory computer readable storage device or medium 56. The camera processor 55 can be a custom processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor of multiple processors associated with the camera controller 54, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions. The vehicle computer readable storage device or medium 46 may include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the camera processor 55 is powered off. The camera computer readable storage device or medium 56 of the camera controller 54 can be implemented using a variety of storage devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or other electrical, magnetic, optical or combination storage devices capable of storing data, some of which represent executable instructions used by the camera controller 54 when controlling the wireless camera 50. The camera non-transitory computer-readable storage device or medium 56 includes machine-readable instructions (e.g., Figure 2 ), when the machine-readable instructions are executed by one or more processors, the camera processor 55 executes method 100 ( Figure 2 ).

[0020] Figure 2 The method 100 is for video streaming. Specifically, the method 100 can be used for video stream time correction and delay detection. The method 100 begins at box 102. At box 102, the vehicle controller 34 receives video streaming data from the wireless camera. As described above, the wireless camera 50 communicates wirelessly with the vehicle controller 34. Then, the method 100 proceeds to box 104.

[0021] At box 104, the vehicle controller 34 performs video frame timestamp reconstruction. This step is optional and can be performed when the video streaming device (such as a wireless camera 50 using the Real Time Messaging Protocol (RTMP)) does not correctly set the video frame presentation timestamp. To perform video frame timestamp reconstruction, the video frame presentation time of the video streaming data is determined using the following equation:

[0022] VPT = 1 / R Equation 1

[0023] Where R is the video frame rate; and

[0024] VPT is the Video Frame Presentation Time.

[0025] In this case, it can be understood that the key frames have a fixed interval. In addition, in the stable video area, the video server receives multiple frames at the native streaming rate. After determining the video frame presentation time, the method 100 continues to box 106. At box 106, the vehicle controller 34 determines the average of a predetermined number of past key frame time intervals to identify the stable video area in the video streaming data received from the wireless camera 50. The following equation can be used:

[0026] μ=avg(I1,…,I n ) Equation 2

[0027] μ is the average value of a predetermined number of past keyframe time intervals;

[0028] I1,…,I n is a predetermined number of past key frame time intervals; and

[0029] n is a predetermined number.

[0030] After frame 106, method 100 continues to frame 108. At frame 108, vehicle controller 34 determines when the next key frame should arrive. To do this, vehicle controller 34 may use the following equation:

[0031] t nextmax ≤t n +μ+∈ Equation 3

[0032] t nextmin ≥t n+μ-∈ Equation 4

[0033] in,

[0034] t nextmax is the maximum timestamp of the next key frame arrival time;

[0035] t nextmin is the minimum timestamp of the next key frame arrival time;

[0036] t n is the timestamp of the immediately preceding keyframe;

[0037] μ is the average of the predetermined number of past key frame time intervals; and

[0038] ∈ is the threshold that controls the sensitivity of outdated video content detection.

[0039] If the previous keyframe is at timestamp t n Arrived, the next key frame should be at [t n +μ-∈,t n +μ+∈], where ∈ is a threshold that controls the sensitivity of outdated video content detection. The vehicle controller 34 determines whether the next key frame arrival time is between t nextmax and t nextmin If the next key frame arrives at time t nextmax and t nextmin , the method 100 proceeds to block 110. At block 110, the vehicle controller 34 determines that the video streaming data received from the wireless camera 50 is being displayed correctly (ie, the video is healthy). nextmax and t nextmin If the method 100 is between 1 and 2, the method 100 proceeds to box 112.

[0040] At box 112, the vehicle controller 34 determines the video streaming flow rate per unit time of the stable video (which is known based on the initial video streaming data) and the current video streaming flow rate per unit time. Then, the vehicle controller 34 compares the current video streaming flow rate per unit time with the video streaming flow rate per unit time of the stable video to determine whether the current video streaming flow rate per unit time is less than the video streaming flow rate per unit time of the stable video. If the current video streaming flow rate per unit time is not less than the video streaming flow rate per unit time of the stable video, the method 100 proceeds to box 110. As described above, at box 110, the vehicle controller 34 determines that the video streaming data received from the wireless camera 40 is being displayed correctly (i.e., the video is healthy). If the current video streaming flow rate per unit time is less than the video streaming flow rate per unit time of the stable video, the method 100 proceeds to box 114. At box 114, the vehicle controller 34 determines that there is expired video content in the video streaming data, and in response thereto, commands the display 48 to provide a notification to the vehicle occupants. The notification indicates that the video being played online in the vehicle display has expired. The notification may be provided by the display 48 and may be provided whenever the video stream delay is greater than a video stream delay threshold. The video stream delay may be determined using the following equation:

[0041] ΔT=t′-(t n +μ) Equation 5

[0042] in,

[0043] ΔT is the video stream delay;

[0044] t n is the timestamp of the immediately preceding keyframe; and

[0045] μ is the average value of a predetermined number of past key frame time intervals.

[0046] Although exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms covered by the claims. The words used in the specification are descriptive rather than restrictive, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously mentioned, the features of the various embodiments can be combined to form another embodiment of the currently disclosed system and method that may not be explicitly described or shown. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations in terms of one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. Therefore, in terms of one or more features, embodiments that are not ideal compared to other embodiments or prior art implementations do not exceed the scope of the present invention and may be ideal for specific applications.

[0047] The drawings are in simplified form and are not drawn to exact scale. Directional terms such as top, bottom, left, right, up, above, above, below, below, rear, and front may be used with respect to the drawings for convenience and clarity only. These and similar directional terms should not be construed as limiting the scope of the present disclosure in any way.

[0048] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms. The drawings are not necessarily drawn to scale, and certain features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but merely as a representative basis for teaching those skilled in the art to adopt the currently disclosed system and method in different ways. As will be understood by those of ordinary skill in the art, the various features shown and described with reference to any of the accompanying drawings may be combined with the features shown in one or more other drawings to produce embodiments that are not explicitly shown or described. The combination of the features shown provides a representative embodiment of a typical application. However, for a specific application or embodiment, various combinations and modifications of features consistent with the teachings of the present disclosure may be expected.

[0049] Embodiments of the present disclosure may be described herein in terms of functional and / or logical block components and various processing steps. It should be understood that such block components may be implemented by multiple hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of the present disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which may perform multiple functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will appreciate that embodiments of the present disclosure may be practiced in conjunction with a variety of systems, and the systems described herein are merely exemplary embodiments of the present disclosure.

[0050] For the sake of brevity, technologies related to signal processing, data fusion, signal transmission, control and other functional aspects of the system (as well as the various operating components of the system) may not be described in detail herein. In addition, the connecting lines shown in the various figures contained herein are intended to represent example functional relationships and / or physical couplings between the various elements. It should be noted that alternative or additional functional relationships or physical connections may exist in the embodiments of the present disclosure.

[0051] This description is merely illustrative in nature and is in no way intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in many forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, as other modifications will become apparent upon studying the drawings, the specification and the appended claims.

Claims

1. A method for video streaming, comprising: receiving video streaming data from a wireless camera, wherein the wireless camera is in wireless communication with the vehicle; identifying a stable video region in the video streaming data received from the wireless camera; detecting outdated video content in the video streaming data based on the stable video region; and In response to detecting the expired video content in the video streaming data, providing a notification to a vehicle occupant via a vehicle display, wherein the notification indicates that the video being played online in the vehicle display has expired.

2. The method according to claim 1 further comprises reconstructing the video frame timestamp, wherein: Performing the video frame timestamp reconstruction includes determining the video frame presentation time of the video streaming data using the following equation: VPT=1 / R, where R is the video frame rate and VPT is the video frame presentation time of the video streaming data.

3. The method according to claim 2, wherein: Identifying the stable video region in the video streaming data received from the wireless camera includes determining an average of a predetermined number of past key frame time intervals.

4. The method according to claim 3, wherein: Identifying the stable video region in the video streaming data received from the wireless camera includes using the following equation: t nextmax ≤t n +μ+∈ t nextmin ≥t n +μ-∈ t nextmax is the maximum timestamp of the next key frame arrival time; t nextmin is the minimum timestamp of the next key frame arrival time; t n is the timestamp of the immediately preceding keyframe; μ is said average value of said predetermined number of past key frame time intervals; as well as ∈ is the threshold that controls the sensitivity of outdated video content detection.

5. The method according to claim 4, further comprising: Determine the arrival time of the next key frame at t nextmax and t nextmin between; as well as In response to determining that the next key frame arrival time is at t nextmax and t nextmin and determining that the video streaming data from the wireless camera is being properly displayed in the vehicle display.

6. The method according to claim 5, further comprising: Determine the current video streaming traffic per unit time; Determine the video streaming flow rate per unit time of the stable video; Comparing the current video streaming transmission flow rate per unit time with the video streaming transmission flow rate per unit time of the stable video to determine whether the current video streaming transmission flow rate per unit time is less than the video streaming transmission flow rate per unit time of the stable video; as well as In response to determining that the current video streaming traffic per unit time is less than the video streaming traffic per unit time of the stable video, providing the notification to the vehicle occupant through the vehicle display, wherein the notification indicates that the video played online in the vehicle display has expired.

7. The method according to claim 6, further comprising: Determine the current video streaming traffic per unit time; Determine the video streaming flow rate per unit time of the stable video; Comparing the current video streaming transmission flow rate per unit time with the video streaming transmission flow rate per unit time of the stable video to determine whether the current video streaming transmission flow rate per unit time is less than the video streaming transmission flow rate per unit time of the stable video; as well as In response to determining that the current video streaming flow rate per unit time is not less than the video streaming flow rate per unit time of the stable video, it is determined that the video streaming data from the wireless camera is being correctly displayed on the vehicle display.

8. A tangible, non-transitory machine-readable medium comprising machine-readable instructions that, when executed by a processor, cause the processor to: Receive video streaming data from a wireless camera, where The wireless camera communicates wirelessly with the vehicle; identifying a stable video region in the video streaming data received from the wireless camera; detecting outdated video content in the video streaming data based on the stable video region; as well as In response to detecting the expired video content in the video streaming data, providing a notification to a vehicle occupant via a vehicle display, wherein the notification indicates that the video being played online in the vehicle display has expired.

9. The tangible, non-transitory machine-readable medium of claim 8, wherein: The tangible, non-transitory machine-readable medium further includes machine-readable instructions that, when executed by the processor, cause the processor to: Perform video frame timestamp reconstruction, wherein the video frame timestamp reconstruction includes using the following equation: VPT=1 / R to determine the video frame presentation time of the video streaming data, wherein R is the video frame rate and VPT is the video frame presentation time of the video streaming data.

10. The tangible, non-transitory machine-readable medium of claim 9, wherein: The tangible, non-transitory machine-readable medium also includes machine-readable instructions that, when executed by the processor, cause the processor to: An average value of a predetermined number of past key frame time intervals is determined.

Images