Method for embedded video hardware decoder pre-warning

Abstract

The application discloses a fault-tolerance evaluation method and early warning method for an embedded video hardware decoder. The fault-tolerance evaluation method comprises the following steps: setting a parameterized packet loss mode for each information layer of a video bitstream, wherein the set parameters include any one of a packet loss start position, a packet loss start offset number or a packet loss number; the information layer includes a sequence parameter set layer, a picture parameter set layer, a strip information layer and a macroblock data layer; performing packet loss testing on the set parameters in the sequence parameter set layer, the picture parameter set layer, the strip information layer and the macroblock data layer respectively, recording test parameters with serious decoding errors, and obtaining a test result.

Description

Technical Field

[0001] This invention belongs to the field of video decoding technology, specifically relating to a fault tolerance assessment method and an early warning method for embedded video hardware decoders. Background Technology

[0002] With the development of emerging edge computing, more and more video analytics workloads that traditionally run on back-end servers are being placed partially or entirely on edge terminals, such as various embedded devices. The advantages of this are low data transmission overhead, low latency, and cost savings.

[0003] Low-cost video decoding chips have become an essential component of edge video analytics, directly and indirectly impacting core metrics such as the stability and cost-effectiveness of video analytics terminals. In practice, video decoder chip companies implement extensive fault-tolerance measures in their decoder hardware; however, due to cost and resource constraints, the fault tolerance capabilities of chips still lag behind those of server hardware. Front-end bitstream errors frequently lead to varying degrees of system errors, such as interrupted decoding, program crashes, driver anomalies, and system freezes. Furthermore, due to the large number of terminals, it is difficult for the central control center to detect problems and their causes, making timely warnings and preventative measures challenging. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention provides a fault tolerance evaluation method for embedded video hardware decoders, which performs targeted testing and evaluation for decoding errors that may be caused by bitstream packet loss and other reasons.

[0005] This invention also provides an early warning method for embedded video hardware decoders. The results of the fault tolerance assessment can be used for targeted early warning and prevention in production environments, thereby improving the system stability of the hardware decoder.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: This invention provides a method for evaluating the fault tolerance of an embedded video hardware decoder, comprising the following steps: The video stream is configured to simulate parameterized packet loss methods for each information layer. The parameters set include any one of the following: packet loss start position, packet loss start offset, or number of packets lost. The information layers include a sequence parameter set layer, an image parameter set layer, a strip information layer, and a macroblock data layer. The set parameters were tested for packet loss at the sequence parameter set layer, image parameter set layer, strip information layer, and macroblock data layer. Test parameters that caused serious decoding errors were recorded, and the test results were obtained.

[0007] In one possible implementation, the packet loss start position is the start position of the sequence parameter set layer, image parameter set, strip, and macroblock, used to describe which layer of information the packet loss is targeting.

[0008] In one possible implementation, the packet loss start offset is the number of bytes and bits from the start position of packet loss, used to describe how many positions the packet loss begins based on a certain layer of information.

[0009] In one possible implementation, the packet loss count is used to describe how many bytes and bits of data are discarded.

[0010] In one possible implementation, the severe decoding error includes driver crashes, system hangs and becomes unresponsive, and hardware crashes.

[0011] In another aspect, this invention provides an early warning method for embedded video hardware decoders. When the input bitstream data is a bitstream that would cause a serious decoding error after fault tolerance evaluation, the method chooses not to transmit the bitstream data to the corresponding information layer, or chooses to continue sending it but monitors the hardware status. The information layer includes a sequence parameter set layer, an image parameter set layer, a stripe information layer, and a macroblock data layer. The serious decoding error includes driver crash, system hang-up and loss of response, and hardware crash.

[0012] In one possible implementation, the fault tolerance assessment includes the following steps: Configure the simulated parameterized packet loss method for each information layer of the video bitstream. The parameters to be set include any one of the following: packet loss start position, packet loss start offset, or number of packets lost. Packet loss tests were performed on the sequence parameter set layer (SPS), image parameter set layer (PPS), slice information layer (Slice), and macroblock data layer (Macroblock) with the set parameters. Test parameters that caused serious decoding errors were recorded, and the test results were obtained.

[0013] In one possible implementation, the packet loss start position is the start position of the Sequence Parameter Set (SPS), Image Parameter Set (PPS), Slice, or Macroblock, used to describe which layer of information the packet loss is targeting.

[0014] In one possible implementation, the packet loss start offset is the number of bytes and bits from the start position of packet loss, used to describe how many positions the packet loss begins based on a certain layer of information.

[0015] In one possible implementation, the packet loss count is used to describe how many bytes and bits of data are discarded.

[0016] The present invention has the following beneficial effects: (1) Targeted testing and evaluation should be conducted to address decoding errors that may occur due to packet loss in the hardware decoder. Targeted error simulation attacks should be carried out on various information layers with different degrees of impact and severity of errors to effectively evaluate the stability and main risk points of the hardware decoder.

[0017] (2) The results of the fault tolerance assessment can be used for targeted early warning and prevention in the production environment. When an error stream occurs that will not spread, the system can continue to work and reduce unnecessary data loss. The system can focus on monitoring the bit stream layer that is prone to serious errors. If an error is found in the bit stream sent to this layer, it can be directly discarded to prevent possible fatal problems. Alternatively, it can be sent to the layer, but a fatal problem is expected. When a fatal error occurs, the cause of the error can be identified as soon as possible, and the decoding hardware, driver, or even the device can be reset immediately to prevent the problem in advance and improve the system stability of the hardware decoder. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the steps of an embedded video hardware decoder fault tolerance evaluation method according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the hierarchical structure of a video sequence; Figure 3 This is a schematic diagram of the video stream structure. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See Figure 1 The diagram shows a flowchart of the fault tolerance evaluation method for an embedded video hardware decoder according to an embodiment of the present invention, including the following steps: S10, set the simulated parameterized packet loss mode for each information layer of the video bitstream. The set parameters include any one of the following: packet loss start position, packet loss start offset, or number of packets lost. The information layer includes a sequence parameter set layer, an image parameter set layer, a strip information layer, and a macroblock data layer. S20, perform packet loss tests on the sequence parameter set layer, image parameter set layer, strip information layer, and macroblock data layer respectively with the set parameters, record the test parameters that have serious decoding errors, and obtain the test results.

[0021] The hierarchical structure of videos is basically similar across various video formats. To facilitate a better understanding of the implementation process of the embodiments of the invention by those skilled in the art, H.264 is used as an example to explain the video hierarchical structure and decoding process. See also Figure 2 The video encoding process, from top to bottom, consists of a video sequence > a frame > a slice > a macroblock. A single frame can be divided into many slices, and a macroblock is composed of one or more slices. A macroblock is the smallest processing unit in video encoding, carrying the specific YUV information of the video; a macroblock consists of one or more macroblocks.

[0022] During encoding, the video encoder writes information such as the width, height, color format, and frame rate of the video into the header information of the video bitstream (e.g., Sequence Parameter Set, SPS); it divides a video into several frames and writes the frame type and the corresponding reference frame into the header information of each frame (e.g., Picture Parameter Set, PPS); it divides each frame into several slices and writes the index information of the slice in the current frame into the header information of each slice; it divides each slice into several macroblocks and writes the encoded residuals in the image into the macroblock information.

[0023] When decoding a video, the decoder extracts information such as width and height from the Sequence Parameter Set (SPS) header information. When decoding each frame, it extracts information such as frame type and reference frame from the Picture Parameter Set (PPS) header information. When decoding each stripe, it extracts the stripe index from the stripe header information. When decoding each macroblock, it extracts the encoding residual from the macroblock bitstream and parses it into picture details, thereby presenting the original image.

[0024] Due to unavoidable reasons such as packet loss during network transmission, damage to storage media, or signal interference, the bitstream may be lost or altered at certain points. This can lead to various unrecoverable errors during the decoding process, including mosaic (macroblock error, stripe error, whole frame error, or entire sequence error), driver crashes, system hangs and becomes unresponsive, or even hardware crashes. These errors can affect the normal operation of the system and require early warning and targeted handling, such as whether the error can be ignored and tolerated, whether the decoder driver needs to be reset, whether the decoder hardware needs to be reset, or even restarting the entire device and system.

[0025] The NAL bitstream structure still uses H.264 as an example; other bitstreams may have slightly different names, but their basic structures are similar. See also... Figure 3The video stream is composed of a series of Network Abstract Layer Units (NALUs). Functionally, NALUs are divided into two layers: the Video Coding Layer (VCL) and the Network Abstract Layer (NAL). The VCL includes the core compression engine and syntax-level definitions for blocks, macroblocks, and slices, enabling efficient encoding as independently of the network as possible. The NAL is responsible for adapting the bit strings generated by the VCL to various networks and diverse environments. Before VCL data is transmitted or stored, this encoded VCL data is mapped or encapsulated into NALUs. Each NALU includes a Raw Byte Sequence Payload (RBSP) and a set of NALU header information corresponding to the video encoding. The RBSP adds a trailing bit to the raw encoded data—one "1" followed by several "0" bits—for byte alignment.

[0026] In one embodiment of the present invention, the packet loss start position is the start position of the sequence parameter set layer, image parameter set, stripe, and macroblock, used to describe which layer of information the packet loss begins with. The packet loss start offset is the number of bytes and bits from the packet loss start position, used to describe how many offsets from the start of a certain layer of information the packet loss begins. The packet loss quantity describes how many bytes and bits of data are discarded. Severe decoding errors include driver crashes, system hangs and becomes unresponsive, and hardware crashes.

[0027] Fault tolerance and stability refer to the extent to which an embedded hardware decoder is affected by bitstream errors caused by factors such as network packet loss, storage media damage, and electromagnetic interference, and its ability to continue working normally. Good fault tolerance and stability mean that even if the system encounters a decoding error after receiving an erroneous bitstream, the impact is limited, and it can recover and continue working once the normal bitstream arrives after the current error semantic range (macroblock, stripe, image, sequence). Poor fault tolerance and stability mean that after receiving an erroneous bitstream, the impact exceeds the current error semantic range, and it may even be unable to continue working normally.

[0028] In one embodiment of the present invention, parameterized packet loss tests are also performed on auxiliary data such as SEI NALU type that do not affect the main process of video decoding to obtain the fault tolerance of auxiliary data and prevent the hardware decoder from causing errors due to the poor fault tolerance of these data.

[0029] The above-described method for evaluating the fault tolerance of embedded video hardware decoders assesses the fault tolerance and stability of the embedded system's hardware decoder against bitstream errors. This involves testing and evaluating each layer of data individually. Specifically, it simulates various parameterized packet loss scenarios at the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Slice information layer, and Macroblock data layer levels to perform fault tolerance tests on the embedded system, thereby evaluating its fault tolerance to bitstream errors. This is unlike the overall evaluation of the entire decoder. This is because the amount of data and the scope of impact differ at each layer, resulting in varying degrees of damage to the hardware decoder. Generally, higher-level information has a greater impact on the bitstream and causes more damage. For example, the Sequence Parameter Set (SPS) layer affects the entire sequence until the next SPS layer appears; the Picture Parameter Set (PPS) layer affects the entire frame until the next Picture Parameter Set; Slice information affects the current slice until the next slice; and macroblock data has a smaller impact, affecting only the current macroblock.

[0030] Another embodiment of the present invention provides an early warning method for an embedded video hardware decoder. When the input bitstream data is a bitstream that would cause serious decoding errors after fault tolerance evaluation, the method chooses not to transmit the bitstream data to the corresponding information layer, or chooses to continue sending it but monitor the hardware status. If a serious hardware failure occurs, the cause of the failure can be determined immediately and dealt with promptly, such as immediately resetting the decoding hardware, resetting the driver, or even restarting the device, thus preventing problems in advance and improving the system stability of the hardware decoder. The information layer includes a sequence parameter set layer, an image parameter set layer, a stripe information layer, and a macroblock data layer. Serious decoding errors include driver crashes, system hangs and becomes unresponsive, and hardware crashes.

[0031] The fault tolerance assessment includes the following steps: setting the simulated parameterized packet loss method for each information layer of the video bitstream, and the parameters set include any one of the following: packet loss start position, packet loss start offset number, or packet loss number; Packet loss tests were performed on the sequence parameter set layer (SPS), image parameter set layer (PPS), slice information layer (Slice), and macroblock data layer (Macroblock) with the set parameters. Test parameters that caused serious decoding errors were recorded, and the test results were obtained.

[0032] In one embodiment of the present invention, the packet loss start position is the starting position of the Sequence Parameter Set (SPS), Image Parameter Set (PPS), Slice, or Macroblock, used to describe which layer of information the packet loss begins at. The packet loss start offset is the number of bytes and bits from the packet loss start position, used to describe how many positions offset from the start of a certain layer of information the packet loss begins at. The packet loss quantity is used to describe how many bytes and bits of data are discarded.

[0033] The above-described early warning method for embedded video hardware decoders allows for the continued transmission of erroneous bitstreams that will not lead to the spread and accumulation of errors in the hardware decoder, or will not affect its continued operation. This reduces unnecessary data loss. For example, it provides good protection for image parameter set (PPS), slice, and macroblock layer data. Subsequent data loss at these layers, even if resulting in decoding errors such as mosaic, can be ignored.

[0034] When encountering decoding errors that could lead to serious or unrecoverable hardware decoder failures, proactively reset the hardware decoder or driver, or even restart the system, to prevent further error accumulation and eventual system crash. For example, data loss in the Sequence Parameter Set (SPS) and Picture Parameter Set (PPS) layers is often difficult to handle. If parsing continues according to the erroneous bitstream, the error may spread and cause serious errors, requiring monitoring and timely device reset.

[0035] Testing each layer allows for the evaluation and understanding of the hardware decoder's fault tolerance and protection capabilities for each layer. For example, data protection at the Image Parameter Set (PPS), Slice, and Macroblock layers is relatively good; subsequent data loss at these layers can be ignored. However, data loss at the Sequence Parameter Set (SPS) layer is typically difficult to handle. If the hardware decoder has protection options, it can directly discard all subsequent data, which the application can ignore. If it continues to parse according to the erroneous bitstream, causing error propagation, it may lead to serious errors, requiring monitoring and timely device reset. These measures improve the system stability of the hardware decoder.

[0036] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A method for early warning in embedded video hardware decoders, characterized in that, When the input bitstream data is a bitstream that would cause a serious decoding error after fault tolerance evaluation, the system may choose not to transmit the bitstream data to the corresponding information layer, or choose to continue sending it but monitor the hardware status; the information layer includes the Sequence Parameter Set Layer (SPS), the Image Parameter Set Layer (PPS), the Slice Information Layer (Slice), and the Macroblock Data Layer (Macroblock); the serious decoding error includes driver crash, system hang-up and loss of response, or hardware crash; The fault tolerance assessment includes the following steps: The video stream is configured with simulated parameterized packet loss methods for each information layer. The parameters include any one of the following: packet loss start position, packet loss start offset, or packet loss quantity. The packet loss start position refers to the starting position of the Sequence Parameter Set (SPS), Image Parameter Set (PPS), Slice, and Macroblock layers, describing which layer the packet loss begins at. The packet loss start offset is the number of bytes and bits from the packet loss start position, describing how many offsets from a specific information layer the packet loss begins at. The packet loss quantity describes how many bytes and bits of data are discarded. Packet loss tests were performed on the Sequence Parameter Set (SPS), Image Parameter Set (PPS), Slice, and Macroblock layers with the set parameters. Test parameters that resulted in serious decoding errors were recorded, and the test results were obtained.

Images