System and method for transmitting functional safety statistics via transmitted video

Abstract

The present application discloses a system and method for transmitting functional safety statistics via transmitted video. A system and method for transmitting functional safety statistics within a system is provided. A video source (16) generates a video data stream. A functional safety system driver (18) accumulates functional safety statistics from at least one system and writes the functional safety statistics to an associated system memory (20). A display subsystem driver (22) writes frames of the video data stream to the system memory (20). The display subsystem driver (12) formats the functional safety statistics as video data and appends the functional safety statistics to a portion of the frame of the video reserved for functional safety statistics. The display subsystem (12) transmits the frames of the video data stream to a host processor (24), which extracts the functional safety statistics from the video frames.

Description

[0001] This application is a divisional application of Chinese patent application 201811522234.7, entitled “System and method for sending functional safety statistics via sent video”, filed on December 13, 2018. Technical Field

[0002] The present invention relates generally to communication systems and, more particularly, to systems and methods for transmitting functional safety statistics via transmitted video. Background Art

[0003] Functional safety is part of the overall safety of a system or device and depends on the system or device operating correctly in response to its inputs, including the safe management of possible operator errors, hardware failures, and environmental changes. A key requirement of functional safety is monitoring each safety system and system-critical device to ensure they are performing correctly as expected. Functional safety is typically verified and checked on an external device acting as a master by collecting statistical information from client devices. Summary of the Invention

[0004] According to one aspect of the present invention, a system is provided. A video source generates a video data stream. A functional safety system driver accumulates functional safety statistics from at least one system and writes the functional safety statistics to an associated system memory. A display subsystem driver writes frames of the video data stream to the system memory. The display subsystem driver formats the functional safety statistics as video data and appends the functional safety statistics to a portion of the video frame reserved for the functional safety statistics. The display subsystem transmits the frames of the video data stream to a host processor, which extracts the functional safety statistics from the video frames.

[0005] According to another aspect of the present invention, a method is provided. Functional safety statistics are accumulated from at least one monitored system. The functional safety statistics are formatted as video data. The functional safety statistics are appended to a portion of a video frame reserved for the functional safety statistics. The frames of the video data stream are transmitted to a host processor via a display subsystem. The functional safety statistics are extracted from the video frames at the host processor.

[0006] According to another aspect of the present invention, a method for transmitting functional safety statistics within an automobile is provided. Functional safety statistics are accumulated from at least one automated system within the automobile. Frames of video are acquired at a camera of the automobile. The functional safety statistics are formatted as video data such that the functional safety statistics are robust to conversion of the video data from a first video format to a second video format. The functional safety statistics are appended to a portion of the video frame reserved for the functional safety statistics. At a display subsystem, frames of a video data stream are converted from the first video format to the second video format. The frames of the video data stream are transmitted to a host processor via the display subsystem. At the host processor, the functional safety statistics are extracted from the frames of the video data stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The features, objects and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0008] Figure 1 A system that uses a display subsystem to provide functional safety statistics to a host processor is described;

[0009] Figure 2 An embodiment of a system for sending functional safety statistics from a system on chip (SoC) to a host processor is described;

[0010] Figure 3 An example of a method for transmitting functional safety statistics via a display link is described; and

[0011] Figure 4 A method for transmitting functional safety statistics within a vehicle is described. DETAILED DESCRIPTION

[0012] Functional safety is about ensuring the safe operation of a system even when something goes wrong with the system. As part of functional safety, it is expected that different events within an embedded system-on-chip (SoC) or other system (which may or may not indicate the presence of a fault) are recorded and provided to a safety supervisor system, which can then determine the health of the system and ensure that it remains safe. This recorded information is referred to herein as functional safety statistics (FSS). The safety supervisor system described above can exist within the monitored system or be located external to the monitored system. In the case where the supervisor system is located external to the monitored system, the functional safety statistics can be transmitted to the safety supervisor system in a reliable and robust manner.

[0013] In some applications, such as in automotive systems, the monitored system may coexist with one or more video systems in a single embedded system. Such a system may include a display subsystem (DSS) that provides a high-bandwidth interface to system memory (e.g., double data rate (DDR) memory) and other parts of the system or a host processor external to the system via the DSS output. Because the display needs to be continuously updated, the data provided by the DSS is already a high priority at the embedded system and is controlled by a real-time software driver.

[0014] According to the present disclosure, the presence of a display subsystem can be exploited by transmitting functional safety statistics interleaved with video and graphics data. Specifically, the systems and methods presented herein reliably and robustly encode the functional safety statistics within the video frames themselves and then extract them from the video data on the receiving side at the host processor. As a result, the functional safety statistics can be conveniently provided to the host processor without significant additional overhead.

[0015] Figure 1 System 10 is illustrated, utilizing a display subsystem 12 to provide functional safety statistics to a host processor 14. A video source 16 generates a video data stream. For example, video source 16 may include a camera, a CPU / GPU within the system generating video frames, or a memory storing video content preloaded into memory via WiFi or Ethernet. A functional safety system driver 18 accumulates functional safety statistics from at least one system and writes the functional safety statistics to system memory 20. A display subsystem driver 22 reads the functional safety statistics from system memory 20 and writes frames of the video data stream to system memory 20, formatting the functional safety statistics as video data and appending the functional safety statistics to a portion of the video frame reserved for functional safety statistics. The display subsystem 12 transmits the video frames to the host processor 14, which extracts the functional safety statistics from the video frames.

[0016] In one embodiment, formatting the functional safety statistics may include altering the data so that the functional safety statistics are robust to transformations of the data. For example, video data may be initially captured in RGB format but converted to YUV format at the display subsystem 12 before being sent. This conversion may introduce quantization errors into one or more least significant bits of the video data. As a result, the functional safety statistics may be written at the system memory 20 such that no data is stored in the least significant bits. In another embodiment, the data may be converted from YUV 444 format to YUV 422 format at the display subsystem, which requires downsampling the data. To avoid data loss, the functional safety statistics may be upsampled before being appended to the video frame.

[0017] In one embodiment, frames of video are provided directly to the display 24, with the functional safety statistics still embedded within the video. It will be appreciated that this may cause a few lines of distortion within the video feed, but it allows for a simplified implementation of the host processor 14. In the illustrated embodiment, the host processor 14 extracts the functional safety statistics from the video frames to produce both a functional safety statistics stream and a video stream with the functional safety statistics removed. Thus, a clean video feed can be provided at the expense of a more complex host processor 14.

[0018] Figure 2 An embodiment of a system 50 is described that sends functional safety statistics from a system on a chip (SoC) 60 to a host processor 80. In the illustrated embodiment, the system 50 is implemented in an automobile, and the functional safety statistics may include safety parameters such as frame count, frame rate, frame latency, performance numbers, speed of a moving automobile, metadata for implementing augmented reality on a head unit, and other such values. However, it should be understood that the present invention may be applied to any of a number of systems where rapid reporting of safety statistics is desired, such as industrial systems.

[0019] Given the importance of safety statistics, it is desirable to provide them in real time to the main processor 80. Currently, in many implementations, an inter-integrated circuit interface or a serial peripheral interface is used to send functional safety statistics to external devices, but the amount of data that can be transferred is limited, the interface is typically slow, and synchronizing these interfaces using data pipes is very difficult.

[0020] Automotive chips will have a display subsystem (DSS) for displaying video and graphics data to external devices. The display subsystem provides a high-bandwidth interface to system memory and an external host via the display subsystem output. The display also needs to be updated in real time, so the display subsystem is already a high-priority master in the system-on-chip and is controlled by a real-time driver. This high-bandwidth connection can be exploited by retaining a few lines in each video frame, converting functional safety statistics to video format, embedding the functional safety statistics in the video frame itself, and extracting the functional safety statistics from the video frame at the host processor.

[0021] To this end, the system-on-chip 60 includes a functional safety statistics (FSS) driver 62 that receives functional safety statistics from multiple automated systems. A display subsystem (DSS) driver 64 receives video frames from a camera 52 associated with the vehicle. For example, the camera 52 can be implemented as a camera in the vehicle. In one embodiment, each of the display subsystem driver 64 and the functional safety statistics driver can be implemented as software or firmware instructions executed by a processor (not shown) on the system-on-chip 60. The functional safety statistics driver 62 writes the received functional safety statistics to the system memory 66. The display subsystem driver 64 retrieves the functional safety statistics from the system memory 66 and appends the functional safety statistics to the video frame. The changed video frame is then written to the system memory 66.

[0022] In the illustrated embodiment, the video frames captured at the camera 52 are captured in RGB format, but the data is converted to YUV 422 format at the display subsystem 68 for transmission. The RGB-to-YUV conversion, and the reverse YUV-to-RGB conversion at the host processor 80 used to retrieve the functional safety statistics, introduces quantization errors in the least significant bits transmitted. As a result, the display subsystem driver 64 can avoid writing the functional safety statistics in one or more of the least significant bits of each word of the video frame. The YUV 444-to-YUV 422 conversion at the display subsystem 68 results in downsampling of the input data and a corresponding loss of information. To avoid this, the display subsystem driver 64 can upsample the functional safety statistics before sending them to the display subsystem 68.

[0023] In another embodiment, the camera 52 captures video frames in YUV 422 format, but the data is converted to RGB format at the display subsystem 68 for transmission. In this embodiment, the chroma components are upsampled within the display subsystem 68 using a scaling engine before conversion to RGB. This chroma conversion can destroy the functional safety statistics, and to avoid errors caused by upsampling the chroma, only the luma component is used to transmit the functional safety statistics. The chroma components within the portion of the frame reserved for the functional safety statistics are transmitted with a fixed value of 0x80. Again, color space conversion can introduce errors, so the two least significant bits of each luma component are unused and the N most significant bits are used, where N is a positive integer greater than 1.

[0024] In yet another embodiment, the camera 52 may capture YUV 422 or YUV 420 input, and the video data may be sent as YUV 444. Thus, the display subsystem 68 upsamples the chroma components to the YUV 444 format, and the host processor 80 may downsample the chroma components to the YUV 422 format. This chroma conversion may corrupt the functional safety statistics appended to the video, so only the luma component is used to transmit the functional safety statistics, and to avoid quantization errors, the two least significant bits of each luma component are unused.

[0025] It should be understood that in some embodiments, the video format will not change and no changes to the functional safety statistics will be utilized other than the formatting necessary to fit it into the video frame. In this case, the functional safety statistics can be copied directly into the video frame and can be extracted in the host processor 80 without any conversion.

[0026] At the main processor 80, an extraction component 82 extracts functional safety statistics from the video frame and converts them from the video format. The functional safety statistics are then provided to the system control 84, which evaluates the functional safety statistics and takes remedial action if the functional safety statistics are outside of a desired range. Examples of remedial actions include providing a warning message to the driver, resetting the client system, or changing the automatic system in the vehicle or other machine (such as an industrial robot) to manual control. In the illustrated embodiment, the video is provided directly to the display 56 and the main processor 80 in parallel, and the portion of the display reserved for the functional safety statistics is retained in the video data, resulting in minor artifacts in the video. In an alternative embodiment, the main processor 80 can also extract a copy of the video frame that does not include the portion of the frame reserved for the functional safety statistics and provide this clean video for use at the display 56.

[0027] In the illustrated system 50, functional safety statistics can be transmitted in real time along with the display frame, eliminating the need for an additional data interface for functional safety statistics transmission and allowing for faster and more robust data transmission relative to existing implementations. Typically, the display size will exceed a million pixels, and the frame rate can be between thirty and sixty hertz, allowing for the transmission of large amounts of functional safety statistics by retaining only one or two lines of the video frame. If implemented in an existing system, existing dedicated interfaces previously used for functional safety statistics can be freed up for other purposes.

[0028] In view of the above Figure 1 and Figure 2 The aforementioned structural and functional features described in Figure 3 and Figure 4 , the example method will be better understood. However, for the purpose of simplifying the explanation, Figure 3 and Figure 4 The methods are shown and described as being performed serially, but it should be understood and appreciated that the invention is not limited to the order illustrated, as in other examples some actions may occur in different orders and / or concurrently with the actions illustrated and described herein.

[0029] Figure 3 An example of a method 100 for transmitting functional safety statistics via a display link is illustrated. At 102, functional safety statistics are accumulated from at least one monitored system. At 104, the functional safety statistics are formatted as video data. For example, the functional safety statistics may be written in a format compatible with video frames provided by an associated video source. At 106, the functional safety statistics are appended to a portion of a frame of video reserved for the functional safety statistics. At 108, frames of the video data stream are provided to a host processor via a display subsystem, and at 110, the functional safety statistics are extracted from the video frames at the host processor. In one example, if the functional safety statistics deviate from a desired range, remedial action may be taken at the monitored system in response to the functional safety statistics. It should be understood that the frames may be provided in parallel to the display and the host processor, or the host processor may process the received video to provide frames of clean video that do not contain the functional safety statistics and provide the clean video frames to the display.

[0030] In one embodiment, at the display subsystem, frames of a video data stream are converted from a first video format to a second video format. In this case, formatting the functional safety statistics into the video data at 104 may include formatting the functional safety statistics so that the functional safety statistics are robust to the conversion of the video data from the first video format to the second video format. For example, if the first video format is an RGB format and the second video format is a YUV 422 format, then due to quantization errors in the least significant bits that may occur during the conversion, the functional safety statistics may be written only to the N most significant bits of each word within the portion of the frame of the video reserved for the functional safety statistics, where N is a positive integer greater than 1. Similarly, the functional safety statistics may be upsampled to ensure that no data is lost when the video data is downsampled during the conversion to the YUV 422 format.

[0031] In another example, the first video format is a YUV422 format, and the second video format is one of an RGB format and a YUV444 format. In this case, formatting the functional safety statistics at 104 may include writing the functional safety statistics only in the N most significant bits of each word within the luma component of the portion of the video frame reserved for the functional safety statistics. This allows for data preservation during resampling of chroma values and avoids quantization errors in the least significant bits. It should be understood that in the event of video conversion, the extracted functional safety statistics are converted back to the original format during extraction at 110. Thus, the functional safety statistics are formatted at 104 to be robust to both the original conversion and the reverse conversion back to the original format.

[0032] Figure 4 A method 150 for transmitting functional safety statistics within an automobile is described. At 152, functional safety statistics are accumulated from at least one automated system within the automobile. At 154, frames of video are acquired from a camera of the automobile. At 156, the functional safety statistics are formatted into video data such that the functional safety statistics are robust to conversion of the video data from a first video format to a second video format. This formatting may include, for example, upsampling the functional safety statistics, writing the functional safety statistics only in the luma component within the video frame, or writing the functional safety statistics only in the N most significant bits of each word in the video frame, where N is a positive integer greater than 1.

[0033] At 158, the functional safety statistics are appended to a portion of the video frame reserved for the functional safety statistics. At 160, the frame of the video data stream is converted from the first video format to the second video format at the display subsystem. At 162, the frame of the video data stream is provided to the host processor via the display subsystem, and at 164, the functional safety statistics are extracted from the video frame at the host processor. As previously described, the extraction process may include converting the functional safety statistics back to the first video format.

[0034] The present invention has been disclosed illustratively. Therefore, the terminology used throughout this disclosure should be read in an illustrative rather than a restrictive manner. Although minor modifications of the invention will occur to those skilled in the art, it should be understood that it is intended that all such embodiments be included within the scope of the granted patent that reasonably fall within the scope of the progress contributed thereby, and that the scope should not be limited except in accordance with the appended claims and their equivalents.

Claims

1. A system for using video data, the system comprising: a first driver configured to receive a set of event data related to operation of the component; a second driver coupled to the first driver and configured to: receiving a set of video data comprising frames, wherein the set of video data is encoded in a first video format; as well as encoding the set of event data as a portion of the frame, the portion of the frame being unmodifiable during conversion of the set of video data from the first video format to a second video format; a display subsystem configured to convert the set of video data having the set of event data encoded therein from the first video format to the second video format; as well as A processor configured to: extracting the set of event data from the set of video data in the second video format; as well as Safety statistics related to the operation of the component in the set of event data are evaluated to determine whether to take safety-related measures regarding the operation of the component.

2. The system of claim 1, wherein: The first video format is RGB format; The second video format is a YUV format; and The portion of the frame in which the set of event data is encoded is a set of most significant bits of the frame.

3. The system of claim 1 , wherein: The first video format is a YUV format including a chrominance component and a luminance component; The second video format is an RGB format; and The portion of the frame in which the set of event data is encoded is within the luminance component.

4. The system of claim 1 , wherein: The first video format and the second video format are from the group consisting of YUV 422, YUV 420, and YUV 444; and The portion of the frame in which the set of event data is encoded is within a luminance component.

5. The system of claim 1 , wherein the processor is further configured to: The set of video data is provided for display without the set of event data.

6. The system of claim 1, further comprising a display coupled to the display subsystem. The system of claim 6 , wherein the display is configured to display the set of video data including the set of event data. 8 . The system of claim 1 , wherein the second driver is configured to upsample the set of event data before encoding the set of event data into the portion of the frame.

9. The system of claim 1 , further comprising a memory coupled to the first driver and the second driver, wherein: The first driver is configured to write the set of event data into the memory; The second driver is configured as: reading the set of event data from the memory; as well as The set of video data including the set of event data is written into the memory.

10. The system of claim 1, wherein: The frame includes a set of rows including rows reserved for the set of event data; The portion of the frame in which the set of event data is encoded is within the reserved rows.

11. A method for using video data, the method comprising: receiving a set of event data related to the operation of the system; receiving a set of video data in a first video format, encoding the set of event data into a portion of the set of video data, the portion of the set of video data being unmodifiable during conversion from the first video format to the second video format; converting the set of video data including the set of event data into the second video format; providing the set of video data including the set of event data in the second video format; determining whether to take safety-related measures regarding operation of the system based on safety statistics in the video data, wherein the safety statistics are related to operation of the system; as well as In response to determining that the safety-related measure is taken, the system is reset.

12. The method according to claim 11, wherein: The first video format is RGB format; The second video format is a YUV format; and The portion of the set of video data in which the set of event data is encoded is a set of most significant bits.

13. The method according to claim 11, wherein: The first video format is a YUV format including a chrominance component and a luminance component; The second video format is an RGB format; and The portion of the set of video data in which the set of event data is encoded is within the luminance component.

14. The method according to claim 11, wherein: The first video format and the second video format are from the group consisting of YUV 422, YUV 420, and YUV 444; and The portion of the set of video data in which the set of event data is encoded is within a luminance component.

15. The method of claim 11, further comprising upsampling the set of event data before encoding the set of event data into the portion of the set of video data.

16. The method of claim 11, further comprising extracting the set of event data from the set of video data in the second video format.

17. The method of claim 11, further comprising displaying the set of video data including the set of event data.

18. A system for using video data, the system comprising: Memory; a functional safety driver coupled to the memory and configured to receive a set of vehicle event data and write the set of vehicle event data to the memory, the set of vehicle event data relating to operation of an associated vehicle; A display driver is coupled to the memory and configured to: receiving a set of video data encoded in a first video format; reading the set of vehicle event data from the memory; encoding the set of vehicle event data into a portion of the set of video data such that the vehicle event data is unmodifiable during conversion from the first video format to the second video format; and writing the set of video data including the set of vehicle event data into the memory; A display subsystem is coupled to the memory and is configured to: reading the set of video data including the set of vehicle event data from the memory; converting the set of video data including the set of vehicle event data from the first video format to the second video format; as well as providing the set of video data including the set of vehicle event data in the second video format; as well as a main processor coupled to the display subsystem and configured to: extracting the set of vehicle event data from the set of video data; determining whether to take safety-related action with respect to operation of the associated vehicle based on safety statistics in the set of vehicle event data, wherein the safety statistics are related to operation of the associated vehicle; as well as In response to a decision to take the safety-related action, a system in the associated vehicle is changed from automatic control to manual control.

Images