Apparatus and method for handling video decoding errors, decoding device and chip

Abstract

This disclosure provides an apparatus and method for handling video decoding errors, a decoding device, and a chip. The apparatus includes: a time counting module configured to start counting based on the decoding action of the video bitstream, and to generate a timeout error in response to the counting time exceeding a preset time threshold; an error control module configured to receive the bitstream error generated during video bitstream decoding and the timeout error generated by the time counting module, and to issue construction instruction information; and a syntax construction module configured to construct a syntax based on the construction instruction information, the constructed syntax being used for normal decoding after a video decoding error. This disclosure can support error handling and image recovery for various video decoding formats, and can also prevent error propagation through error propagation calibration and updating.

Description

Technical Field

[0001] This disclosure pertains to the technical field of video processing, and relates to processing apparatus, particularly to apparatus and method for processing video decoding errors, as well as decoding apparatus and chips. Background Technology

[0002] During video transmission, the encoder encodes and processes the video stream to obtain a video bitstream. This bitstream is then transmitted via storage media, broadcast, or network video stream. The decoder decodes and processes the bitstream data to reconstruct the video image.

[0003] With the widespread use of the internet, video has become an indispensable part of people's daily lives. However, sometimes video parsing errors occur. When the network signal is unstable or bandwidth is insufficient, the video stream may experience packet loss, data rewriting, or data overwriting. This can lead to scrambled images, noise, and luminance / color crosstalk in the decoded video, and these issues will worsen as the video plays, resulting in a very poor viewing experience. Summary of the Invention

[0004] This disclosure provides an apparatus and method for handling video decoding errors, a decoding device and a chip, for quickly restoring normal operation after a video decoding error.

[0005] In a first aspect, embodiments of this disclosure provide an apparatus for handling video decoding errors. The apparatus includes: a time counting module configured to start counting based on the decoding action of the video bitstream, and to generate a timeout error in response to the counting time exceeding a preset time threshold; an error control module configured to receive the bitstream error generated during video bitstream decoding and the timeout error generated by the time counting module, and to issue construction instruction information; and a syntax construction module configured to construct a syntax based on the construction instruction information, the constructed syntax being used for normal decoding after a video decoding error.

[0006] In one implementation of the first aspect, the error control module is configured to: select coordinates to be constructed based on the coordinate information reported when video decoding fails, initiate a reset request after the decoding is paused, and issue the construction instruction information in response to the completion of the reset, in conjunction with the constructed coordinates.

[0007] In one implementation of the first aspect, the reported coordinates include motion vector coordinates and filtered image coordinates. The reported coordinate information includes tile information, tile count, x-coordinate, and y-coordinate corresponding to the motion vector coordinates and filtered image coordinates, respectively. The error control module is configured to: in response to the tile counts of the motion vector coordinates and the filtered image coordinates being unequal, compare the tile counts and select the coordinate with the smaller tile count to construct coordinates; in response to the tile counts of the motion vector coordinates and the filtered image coordinates being equal, first compare the y-coordinates; if the y-coordinates of the motion vector coordinates and the filtered image coordinates are unequal, then select the coordinate with the smaller y-coordinate value to construct coordinates; and if the y-coordinates of the motion vector coordinates and the filtered image coordinates are equal, then compare the x-coordinates of the motion vector coordinates and the filtered image coordinates and select the coordinate with the smaller x-coordinate value to construct coordinates.

[0008] In one implementation of the first aspect, the construction instruction information includes an error construction command, and the syntax construction module is configured to: construct stripe syntax and / or tile block syntax according to the error construction command, and then construct coding tree unit syntax and coding unit syntax; determine whether the configured mode is intra-frame mode or inter-frame mode; if it is intra-frame mode, construct transform unit syntax; and if it is inter-frame mode, perform frame end judgment and tile block end judgment; in the intra-frame mode, in response to the completion of transform unit syntax construction, perform frame end judgment and tile block end judgment; when performing frame end judgment and tile block end judgment, in response to tile block end and frame not end, return to the step of constructing tile block syntax; and in response to frame end, send a construction completion signal to the error control module.

[0009] In one implementation of the first aspect, the encoding tree unit syntax constructed by the syntax construction module includes an error identification header, which is used to identify that the construction of the current encoding tree unit block originates from the error control module, thereby preventing error propagation.

[0010] Secondly, embodiments of this disclosure provide a method for handling video decoding errors. The method includes: starting a counting process based on a video bitstream decoding action; generating a timeout error in response to the counting time exceeding a preset time threshold; receiving the bitstream error generated during video bitstream decoding and the timeout error; issuing construction instruction information; and constructing a syntax based on the construction instruction information, the constructed syntax being used for normal decoding after a video decoding error.

[0011] Thirdly, embodiments of this disclosure provide a decoding apparatus. The decoding apparatus includes: the aforementioned means for processing video decoding errors.

[0012] In one implementation of the third aspect, the decoding device further includes: an entropy decoding module, a residual buffer module, a syntax buffer module, an inverse quantization and inverse transform module, a prediction module, a filtering module, a reconstruction and restoration module, and a master control module. The master control module is configured to send decoding instructions to the entropy decoding module. The entropy decoding module is configured to start decoding the video stream based on the decoding instructions. The time counting module is configured to start counting based on the decoding actions of the video stream, generating a stream error in response to the entropy decoding module detecting an error in the video stream, or generating a timeout error if the counting time of the entire decoding process exceeds a preset time threshold. The error control module is configured to receive the stream error and the timeout error, and select the coordinate information to be constructed based on the coordinate information reported by the prediction module and the filtering module. The coordinates are determined, and after decoding is paused, a reset request is sent to the main control module. The main control module is configured to reset all modules except the device used to handle video decoding errors in response to the reset request, and to send a reset completion signal to the error control module in response to the completion of the reset of all modules except the device used to handle video decoding errors. This causes the error control module to send construction instruction information to the syntax construction module based on the constructed coordinates. The syntax construction module is configured to construct a syntax according to the construction instruction information, transmit the syntax to the syntax buffer module, and use the constructed syntax for normal decoding after a video decoding error. In response to the completion of the syntax construction, the module sends a construction completion signal to the error control module, indicating that the error handling process has ended.

[0013] In one implementation of the third aspect, the reported coordinates include motion vector coordinates and filtered image coordinates. The reported coordinate information includes tile information, tile count, x-coordinate, and y-coordinate corresponding to the motion vector coordinates and filtered image coordinates, respectively. The error control module is configured to: respond to the tile counts of the motion vector coordinates and the filtered image coordinates being unequal, compare the tile counts and select the coordinate with the smaller tile count to construct the coordinates; respond to the tile counts of the motion vector coordinates and the filtered image coordinates being equal, first compare the y-coordinates, and if the y-coordinates of the motion vector coordinates and the filtered image coordinates are unequal, select the coordinate with the smaller y-coordinate value to construct the coordinates; and if the y-coordinates of the motion vector coordinates and the filtered image coordinates are equal, compare the x-coordinates of the motion vector coordinates and the filtered image coordinates and select the coordinate with the smaller x-coordinate value to construct the coordinates.

[0014] In one implementation of the third aspect, the construction instruction information includes an error construction command, and the syntax construction module is configured to: construct stripe syntax and / or tile block syntax according to the error construction command, and then construct coding tree unit syntax and coding unit syntax; determine whether the configured mode is intra-frame mode or inter-frame mode; if it is intra-frame mode, construct transform unit syntax; and if it is inter-frame mode, perform frame end judgment and tile block end judgment; in the intra-frame mode, in response to the completion of transform unit syntax construction, perform frame end judgment and tile block end judgment; when performing frame end judgment and tile block end judgment, in response to tile block end and frame not end, return to the step of constructing tile block syntax; and in response to frame end, send a construction completion signal to the error control module.

[0015] In one implementation of the third aspect, the encoding tree unit syntax constructed by the syntax construction module includes an error identification header, which is used to identify that the construction of the current encoding tree unit block originates from the error control module, thereby preventing the spread of errors.

[0016] In one implementation of the third aspect, the header of the syntax input selection includes either the header parsed by entropy decoding or the error identification header. In response to the header of the syntax input selection being the error identification header, the error identification header is sent to the syntax buffer module. The prediction module is configured to access the syntax buffer module, parse the current coding tree unit block constructed by the syntax construction module, and configure the prediction module to set the error propagation calibration. Then, the error propagation calibration value and the assist frame motion vector information are saved together to the storage module to complete the error calibration. In response to the header of the syntax input selection being the header parsed by entropy decoding, the prediction module is configured to reset the error propagation calibration, and then the error propagation calibration value and the assist frame motion vector information are saved together to the storage module.

[0017] In one implementation of the third aspect, a tile block of the current frame in the decoding process corresponds to a reference tile block at a reference position in a reference frame. The prediction module is configured to obtain the error propagation calibration value of the reference tile block, determine the error propagation calibration value of the current tile block of the current frame based on the error propagation calibration value of the reference tile block, save the error propagation calibration value of the current tile block of the current frame to the storage module and transmit it to the reconstruction and restoration module. The reconstruction and restoration module is configured to determine whether the current tile block has accumulated residuals normally based on the error propagation calibration value.

[0018] In one implementation of the third aspect, the master control module is configured to send decoding instructions to the entropy decoding module. The entropy decoding module is configured to parse the original residual and syntax from the video bitstream, transmit the original residual to the residual cache module, and transmit the syntax to the syntax cache module. The syntax cache module is configured to be accessed in turn by the inverse quantization and inverse transform module, the prediction module, and the filtering module. Each storage address of the syntax cache module is only released after all three modules have finished accessing it. The inverse quantization and inverse transform module is configured to perform inverse quantization and inverse transform calculations based on the original residual and the syntax, and output the image residual. The prediction module is configured to output intra-frame or inter-frame prediction values ​​based on the syntax, using intra-frame prediction and inter-frame prediction. The reconstruction and restoration module is configured to accumulate the image residual and the prediction values ​​to obtain a reconstructed image, output the reconstructed image to the filtering module for image deblocking and pixel value correction adjustment, and output a frame image to the storage module.

[0019] Fourthly, embodiments of this disclosure provide a chip. The chip includes: a storage module; and the aforementioned decoding device.

[0020] The apparatus, method, decoding device, and chip for handling video decoding errors described in this disclosure can support error handling after decoding timeout errors and bitstream errors, complete the erroneous frame image, and allow subsequent frames to continue normal decoding. This disclosure supports error propagation prevention through error propagation calibration and updating, and supports error handling for formats such as HEVC, H264, AVS2, VP9, ​​and AV1. This disclosure supports erroneous frame image completion through intra-frame or inter-frame construction.

[0021] This disclosure allows the device to take over the syntax construction work after a decoding error occurs, and to input the constructed syntax into the syntax buffer module, thus avoiding video stuttering and discontinuous playback caused by ignoring error frames. This disclosure also handles decoding timeout errors and bitstream errors, preventing subsequent decoding from continuously encountering errors. Furthermore, this disclosure prevents error propagation by marking and updating error propagation, thereby preventing subsequent frames from being unaware of the current frame's error and ensuring image quality. Attached Figure Description

[0022] Figure 1 The diagram shown is a schematic representation of the structure and principle of the apparatus for processing video decoding errors according to an embodiment of this disclosure.

[0023] Figure 2 This diagram shows the internal architecture of an apparatus for handling video decoding errors according to an embodiment of this disclosure.

[0024] Figure 3 This is shown as an error control flowchart of the apparatus for handling video decoding errors according to an embodiment of this disclosure.

[0025] Figure 4 The diagram shows a coordinate comparison of the apparatus for processing video decoding errors according to an embodiment of this disclosure.

[0026] Figure 5 The diagram shown is a flowchart illustrating the error construction of an apparatus for processing video decoding errors according to an embodiment of this disclosure.

[0027] Figure 6 The diagram shows an error configuration coordinate jump of the apparatus for processing video decoding errors according to an embodiment of this disclosure.

[0028] Figure 7 This diagram shows the reference relationship between the current frame and the reference frame of the apparatus for processing video decoding errors according to an embodiment of this disclosure.

[0029] Figure 8 The diagram shown is a flowchart illustrating the principle of a method for handling video decoding errors according to an embodiment of this disclosure.

[0030] Figure 9 The diagram shown is a structural schematic of the decoding device described in an embodiment of this disclosure.

[0031] Figure 10 The diagram shown is a schematic diagram of the decoding device described in an embodiment of this disclosure.

[0032] Figure 11 The diagram shows the error propagation calibration and update of the decoding device according to an embodiment of this disclosure.

[0033] Figure 12 The diagram shown is a schematic representation of the chip described in an embodiment of this disclosure. Detailed Implementation

[0034] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0035] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. Therefore, the illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0036] Currently, for the issues mentioned in the background technology, one approach is to directly exit error frame parsing without displaying the error frame image. The decoder continues parsing the video after the error frame until no more error frames are detected before displaying the video, resulting in video stuttering and discontinuous playback. Another approach is to copy the image from the previous frame to the current error frame after detecting an error, thus completing the error frame image. This error handling method has several drawbacks: first, it only completes the image without completing other data, potentially leading to continuous decoding errors in subsequent frames; second, it may fail to detect bitstream errors, causing decoding timeouts and preventing the decoder from completing the image; and third, subsequent frames are unaware of the current frame's error, potentially causing the error to propagate and deteriorate the image quality, resulting in ineffective error handling.

[0037] To address at least the aforementioned technical problems, this disclosure provides a hardware-based solution for handling video decoding errors and preventing error propagation. According to embodiments of this disclosure, the current erroneous frame is processed, and error calibration information is saved to an external storage module. Subsequent frames read and use this calibration information for updates. Furthermore, based on the error calibration information, residuals are not accumulated, thereby preventing error propagation and improving the effectiveness of video decoding error recovery.

[0038] According to embodiments of this disclosure, if an error exists in the video stream, the error handling module takes over the input of the syntax buffer and constructs a syntax so that subsequent modules can continue decoding normally. This construction is divided into intra-frame mode and inter-frame mode. Intra-frame mode configures the intra-prediction mode and TU syntax information in the syntax. Inter-frame mode requires configuring the backup reference frame address and predicted motion vectors, among other information. Simultaneously, the module constructs a special CTU header and stores it in the syntax buffer, allowing the prediction module to identify whether the current CTU block is incorrectly constructed. If it is incorrectly constructed, the current CTU block's `error_ctu_flag` is set to 1 for error marking, and then `error_ctu_flag` and `colmv` information are stored together in an external storage module. When the next decoded frame references the CTU block constructed from this incorrect frame, the prediction module can obtain the corresponding `error_ctu_flag` information. At this point, it is necessary to update the error_ctu_flag of the current frame CTU and output the updated value along with the colmv information to the external storage module. On the other hand, the error_ctu_flag is sent to the reconstruction and restoration module so that the reconstruction and restoration module can choose whether to accumulate the image residual sent by the inverse quantization and inverse transform module according to the error_ctu_flag, thereby preventing the spread of errors.

[0039] The technical solutions in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0040] Definitions or explanations of the technical terms used in this disclosure:

[0041] CTU: Coding Tree Unit;

[0042] CU: Coding Unit;

[0043] TU: Transform Unit;

[0044] colmv: Assisted frame motion vector;

[0045] TILE: tile;

[0046] slice: strip;

[0047] error_ctu_flag: CTU error propagation calibration.

[0048] Please see Figure 1 The diagram shows the structural principle of the apparatus for processing video decoding errors according to an embodiment of this disclosure. Figure 1As shown, this embodiment provides a device 1 for handling video decoding errors, supporting error handling for formats such as HEVC, H264, AVS2, VP9 and AV1, specifically including: a time counting module 11, an error control module 12 and a syntax construction module 13.

[0049] The time counting module 11 is configured to start counting based on the decoding action of the video bitstream, and generate a timeout error in response to the counting time exceeding a preset time threshold.

[0050] The error control module 12 is connected to the time counting module 11. The error control module 12 is configured to receive bitstream errors generated during video bitstream decoding and timeout errors generated by the time counting module, and issue construction instruction information.

[0051] The syntax construction module 13 is connected to the error control module 12. The syntax construction module 13 is configured to construct a syntax according to the construction instruction information, and the constructed syntax is used for normal decoding after a video decoding error.

[0052] Please see Figure 2 This is a diagram showing the internal architecture of an apparatus for handling video decoding errors according to an embodiment of this disclosure. Figure 2 As shown, the error control module 12 is configured to select the coordinates to be constructed based on the coordinate information reported when a video decoding error occurs, and initiate a reset request after the decoding is paused. Furthermore, the error control module 12 is configured to issue the construction instruction information in response to the reset completion, based on the constructed coordinates, and the syntax construction module 13 inputs the constructed syntax into the syntax buffer.

[0053] Please see Figure 3 This is shown as an error control flowchart of the apparatus for handling video decoding errors according to an embodiment of this disclosure. Figure 3The diagram illustrates the processing flow of the internal state machine of the device. By default, the state machine remains in step 1, waiting for error handling. If an error message is received from the entropy decoding bitstream, the state machine jumps to step 2, waiting for the decoder to stop. Because when an error is detected in the bitstream, subsequent modules such as the inverse quantization / inverse transform module, prediction module, filtering module, and reconstruction / restoration module may still be working, a direct reset is not possible. Instead, the machine waits for all modules to provide an idle_flag (1 indicates the module has stopped working, 0 indicates it is working). When the idle_flag of the entropy decoding module, inverse quantization / inverse transform module, prediction module, filtering module, and reconstruction / restoration module are all 1, and the duration reaches a threshold (assuming the threshold is 50, configurable in practical applications), the decoder can be considered to have stopped. After the decoder stops, the state machine jumps to step 3, where the error control module sends a reset request to the main control module. After the reset request is received, the machine jumps to step 4, waiting for the reset to complete. The main control module resets all modules except the device itself. After the reset is complete, it sends a signal to the error control module to indicate that the reset is complete. After the reset is complete, the state machine jumps to step 5, initiating a start construction command. The error control module sends TILE information, construction start coordinates, and other information to the error construction module to begin constructing the syntax. After receiving the construction command, the error construction module's state machine jumps to step 6, waiting for construction to complete. Once the error construction module completes construction, it sends a construction completion command to the error control module, and the state machine jumps back to step 1 from step 6, waiting for the next error handling process. If the time counter module's count exceeds the threshold, a decoding timeout error occurs, and the state machine jumps directly from step 1 to step 3, skipping step 2. Because a decoding timeout has occurred, it may not be possible to wait for the idle_flag of each module to reach 1, therefore step 2 is skipped. The other steps are consistent with the bitstream error process.

[0054] In one embodiment, the reported coordinates include motion vector coordinates and filtered image coordinates. The reported coordinate information includes tile information, tile count, x-coordinate, and y-coordinate corresponding to the motion vector coordinates and filtered image coordinates, respectively. The error control module is configured to: in response to the tile counts of the motion vector coordinates and the filtered image coordinates being unequal, compare the tile counts and select the coordinate with the smaller tile count to construct coordinates; in response to the tile counts of the motion vector coordinates and the filtered image coordinates being equal, first compare the y-coordinates; if the y-coordinates of the motion vector coordinates and the filtered image coordinates are unequal, then select the coordinate with the smaller y-coordinate value to construct coordinates; and if the y-coordinates of the motion vector coordinates and the filtered image coordinates are equal, then compare the x-coordinates of the motion vector coordinates and the filtered image coordinates and select the coordinate with the smaller x-coordinate value to construct coordinates.

[0055] Combination Figure 2The prediction and filtering modules report coordinates separately. The prediction module reports the coordinates output by colmv, while the filtering module reports the coordinates of the filtered image data. There is no fixed order for the two modules to report coordinates. The reported coordinate information includes TILE information, TILE count (tile_cnt), x-coordinate, and y-coordinate.

[0056] Please see Figure 4 This is a schematic diagram showing the coordinates of the apparatus for processing video decoding errors according to an embodiment of this disclosure. Figure 4 As shown, the `tile_cnt` count is first compared; the smaller the value, the earlier the coordinate appears. Therefore, if the two are not equal, the coordinate with the smaller `tile_cnt` value is directly selected as the starting coordinate for construction. If the `tile_cnt` values ​​are equal, the y-coordinates are compared first. If they are not equal, the coordinate with the smaller y-coordinate value is directly selected for construction. If the y-coordinates are equal, the x-coordinates are compared, and the coordinate with the smaller x-coordinate value is selected as the starting coordinate for construction. The TILE information mainly contains the coordinates of the top-left and bottom-right corners of the TILE.

[0057] In one embodiment, the construction instruction information includes an error construction command. The syntax construction module is configured to: construct stripe syntax and / or tile block syntax according to the error construction command, and then construct coding tree unit syntax and coding unit syntax; determine whether the configured mode is intra-frame mode or inter-frame mode; if it is intra-frame mode, construct transform unit syntax; and if it is inter-frame mode, perform frame end judgment and tile block end judgment; in the intra-frame mode, in response to the completion of transform unit syntax construction, perform frame end judgment and tile block end judgment; during frame end judgment and tile block end judgment, in response to tile block end and frame not end, return to the step of constructing tile block syntax; in response to frame end, send a construction completion signal to the error control module.

[0058] Please see Figure 5 This is a flowchart illustrating the error construction of an apparatus for handling video decoding errors according to an embodiment of this disclosure. Figure 5The diagram illustrates the construction flow of the internal state machine for error construction. Step 1 waits for error construction. Upon receiving an error construction command from the error control module, the process jumps to Step 2 to construct the slice syntax, including the slice header, slice type, number of reference frames, inverse quantization correlation coefficient, filter correlation coefficient, and assist frame correlation coefficient. For formats without slice syntax (e.g., AV1 / VP9), the state machine idles and does not send any syntax to the syntax buffer. After the slice syntax is received, the state machine jumps to Step 3 to construct the TILE syntax. The TILE syntax mainly includes the TILE header and the coordinates of the top-left and bottom-right corners of the TILE. For formats without TILE (H264 / AVS2), the top-left and bottom-right corners of the image are sent. Then, the state machine jumps to Step 4 to construct the CTU / CU syntax, including the error CTU header (which differs from the normal CTU header for easy identification and error calibration by subsequent modules), CTU block coordinates, prediction mode (intra-frame or inter-frame), prediction block size, and prediction block coordinates. The prediction mode can be configured as intra-frame or inter-frame. If configured as intra-frame, the intra-frame prediction mode syntax needs to be provided; if configured as inter-frame, the motion vector syntax needs to be provided. The state machine jumps to step 5, where it needs to determine whether the construction is intra-frame or inter-frame. If the construction is intra-frame, it jumps to step 6 to construct the TU syntax information. After the TU syntax is constructed, it jumps to step 7. If the construction is inter-frame, it jumps directly from step 5 to step 7. Step 7 determines whether the current CTU is TILE or the frame has ended. If it is TILE and not the frame has ended, it jumps to step 3 and continues the above process. If it is the frame has ended, it jumps to step 1 and sends a construction completion signal to the error control module. Construction continues from the construction start coordinates until the end of a frame.

[0059] See Figure 6 This is a schematic diagram showing the error construction coordinate jump of the apparatus for processing video decoding errors according to an embodiment of this disclosure. Figure 6 As shown, assuming the starting coordinates of the construction are P0, within TILE0, the construction begins with a raster scan from P0 to the right of the TILE. Then, starting from the next row after P0, the raster scan continues to the left of TILE0, until TILE0 construction ends. After TILE0 ends, the raster scan continues from P1 (left of TILE1) to TILEn (right side of the image), and then CTUs are used to construct line by line until TILE1 to TILEn are completed. If the image has more than one row of TILEs, the next row of TILEs is from TILEn+1 to TILE2n, and the raster scan construction begins from P2. This method avoids the need to store a large amount of TILE information for different formats, because the AV1 format can have a maximum of 64 TILEs horizontally and 64 TILEs vertically.

[0060] In one embodiment, the encoding tree unit syntax constructed by the syntax construction module includes an error identification header, which is used to identify that the construction of the current encoding tree unit block originates from the error control module, thereby preventing the spread of errors.

[0061] Please see Figure 7 This is a reference relationship diagram between the current frame and a reference frame of the apparatus for processing video decoding errors according to embodiments of this disclosure. Figure 7 As shown, assume video frame F0 is the reference frame for the current frame F1 and is stored in the external storage module. The current frame F1 is the frame being decoded by the decoder. Assume the error propagation flag (error_ctu_flag) of block E0 is 1 and is stored in the external storage module. Since block D0 of the current frame F1 references block E0, block D0 of the current frame F1 retrieves the error propagation flag (error_ctu_flag) of block E0 from the external storage module and updates its error_ctu_flag value to 1. Simultaneously, the error propagation flag (error_ctu_flag) of block D0 is saved to the external storage module and sent to the reconstruction and restoration module. The reconstruction and restoration module checks the error propagation flag (error_ctu_flag) value of each CTU. A value of 1 indicates that no image residual is accumulated; a value of 0 indicates that the residual is accumulated normally. If the error_ctu_flag value of block E0 is 0, then the error_ctu_flag value of block D0 is also 0, and the reconstruction module accumulates the residual normally, consistent with the normal decoding process, with no difference. The reference frame F0 can be an erroneous frame or a normal frame that references an erroneous frame.

[0062] Please see Figure 8 The diagram shows a flowchart illustrating the principle of a method for handling video decoding errors as described in an embodiment of this disclosure. Figure 8 As shown, this embodiment provides a method for handling video decoding errors, specifically including the following steps S11 to S13.

[0063] S11, the decoding action based on the video bitstream starts counting, and a timeout error is generated in response to the counting time exceeding the preset time threshold.

[0064] S12, receive the bitstream error and timeout error generated during video bitstream decoding, and issue construction instruction information.

[0065] S13, construct a syntax according to the constructed instruction information, and the constructed syntax is used for normal decoding after a video decoding error.

[0066] Please see Figure 9 The diagram shows a schematic representation of the decoding device described in an embodiment of this disclosure. Figure 9As shown, the decoding device provided in this embodiment includes the device 1 described above for processing video decoding errors.

[0067] The device for handling video decoding errors includes a time counting module, an error control module, and a syntax construction module. The time counting module is configured to start counting based on the decoding action of the video bitstream, and generate a timeout error in response to the counting time exceeding a preset time threshold. The error control module is configured to receive bitstream errors generated during video bitstream decoding and timeout errors generated by the time counting module, and issue construction instruction information. The syntax construction module is configured to construct a syntax based on the construction instruction information; the constructed syntax is used for normal decoding after a video decoding error.

[0068] Please see Figure 10 The diagram shows the structural principle of the decoding device described in the embodiments of this disclosure. Figure 10 As shown, the decoding device, in addition to the device 1 for handling video decoding errors, also includes: an entropy decoding module 3, a residual buffer module 4, a syntax buffer module 5, an inverse quantization and inverse transform module 6, a prediction module 7, a filtering module 8, a reconstruction and restoration module 9, and a master control module 10. The video stream decoding error handling process in the decoding device includes the following:

[0069] (1) The master control module 10 is configured to send a decoding instruction to the entropy decoding module 3, the entropy decoding module 3 is configured to start decoding the video stream based on the decoding instruction, and the time counting module 13 is configured to start counting based on the decoding action of the video stream.

[0070] (2) In response to the video bitstream being found to have an error by the entropy decoding module 3, a bitstream error is generated or the counting time of the entire decoding process exceeds a preset time threshold, a timeout error is generated; the error control module 12 is configured to receive the bitstream error and the timeout error, select the coordinates to be constructed according to the coordinate information reported by the prediction module 7 and the filtering module 8, and after waiting for the decoding to pause, initiate a reset request to the master control module 10.

[0071] (3) The master control module 10 is configured to reset all modules except the device for handling video decoding errors in response to the reset request, and to send a reset completion signal to the error control module 12 in response to the completion of the reset of all modules except the device for handling video decoding errors, so that the error control module 12 sends construction instruction information to the syntax construction module 13 in combination with the constructed coordinates.

[0072] (4) The syntax construction module 13 is configured to construct a syntax according to the construction instruction information, transmit the syntax to the syntax buffer module 5, and use the constructed syntax for normal decoding after a video decoding error; and in response to the completion of the construction by the syntax construction module 13, send a construction completion signal to the error control module 12 to indicate that the error handling process has ended.

[0073] In one embodiment, the reported coordinates include motion vector coordinates and filtered image coordinates. The reported coordinate information includes tile information, tile count, x-coordinate, and y-coordinate corresponding to the motion vector coordinates and filtered image coordinates, respectively. The error control module 12 is configured to: in response to the tile counts of the motion vector coordinates and the filtered image coordinates being unequal, compare the tile counts and select the coordinate with the smaller tile count to construct the coordinates;

[0074] In response to the equal tile counts of the motion vector coordinates and the filtered image coordinates, the ordinates are first compared. If the ordinates of the motion vector coordinates and the filtered image coordinates are not equal, the coordinate with the smaller ordinate value is selected to construct the coordinates. If the ordinates of the motion vector coordinates and the filtered image coordinates are equal, the abscissas of the motion vector coordinates and the filtered image coordinates are compared, and the coordinate with the smaller abscissa value is selected to construct the coordinates.

[0075] In one embodiment, the construction instruction information includes an error construction command. The syntax construction module is configured to: construct stripe syntax and / or tile block syntax according to the error construction command, and then construct coding tree unit syntax and coding unit syntax; determine whether the configuration is intra-frame mode or inter-frame mode; if it is intra-frame mode, construct transform unit syntax; if it is inter-frame mode, perform frame end judgment and tile block end judgment; in the intra-frame mode, in response to the completion of transform unit syntax construction, perform frame end judgment and tile block end judgment; during frame end judgment and tile block end judgment, in response to tile block end and frame not end, return to the step of constructing tile block syntax; and in response to frame end, send a construction completion signal to the error control module.

[0076] In one embodiment, the encoding tree unit syntax constructed by the syntax construction module includes an error identification header, which is used to identify that the construction of the current encoding tree unit block originates from the error control module, thereby preventing the spread of errors.

[0077] In one embodiment, the header for syntax input selection includes either the header for entropy decoding parsing or the error identification header.

[0078] In response to the information header selected by the syntax input, the error identification information header is sent to the syntax buffer module. The prediction module is configured to access the syntax buffer module, parse the current coding tree unit block constructed by the syntax construction module, and the prediction module is configured to set the error propagation calibration bit. Then, the error propagation calibration value and the assist frame motion vector information are saved together to the storage module to complete the error calibration.

[0079] In response to the information header selected by the syntax input being the information header parsed by entropy decoding, the prediction module is configured to perform an error propagation calibration reset, and then save the error propagation calibration value and the assist frame motion vector information together to the storage module.

[0080] Please see Figure 11 This is a schematic diagram illustrating the error propagation calibration and update of the decoding device described in an embodiment of this disclosure. Combined with... Figure 7 and Figure 11 The diagram illustrates the error propagation labeling and update process. Assuming the syntax input is selected, the header of the device constructing the erroneous CTU (position D0) is chosen. After this header is sent to the syntax buffer, the prediction module accesses the syntax buffer and can parse that the current CTU block is incorrectly constructed. At this point, the prediction module performs error propagation labeling on D0 (setting `error_ctu_flag` to 1). Then, the prediction module saves the error propagation label (`error_ctu_flag`) and the `colmv` information together to the external storage module, thus completing the error labeling. Both the error propagation label (`error_ctu_flag`) and the `colmv` information are stored in CTU units. If the syntax input is selected, and D0 is the CTU header parsed by the normal entropy decoding module, then the error propagation label (`error_ctu_flag`) value is 0. In this case, the error propagation label (`error_ctu_flag`) still needs to be saved to the external storage module and sent to the reconstruction and restoration module.

[0081] In one embodiment, a tile block of the current frame in the decoding process corresponds to a reference tile block at a reference position in a reference frame. The prediction module is configured to obtain the error propagation calibration value of the reference tile block, determine the error propagation calibration value of the current tile block of the current frame based on the error propagation calibration value of the reference tile block, save the error propagation calibration value of the current tile block of the current frame to the storage module and transmit it to the reconstruction and restoration module. The reconstruction and restoration module is configured to determine whether the current tile block has accumulated residuals normally based on the error propagation calibration value.

[0082] Please continue reading. Figure 7 ,like Figure 7As shown, assume that video frame F0 is the reference frame for the current frame F1 and is stored in the external storage module. The current frame F1 is the frame that the decoder is currently decoding. Assume that the error propagation flag (error_ctu_flag) of block E0 is 1 and is stored in the external storage module. Figure 1-7 and Figure 1-8 As shown, since the D0 block of the current frame F1 references the E0 block, the D0 block of the current frame F1 will retrieve the error propagation flag (error_ctu_flag) of the E0 block from the external storage module, and then update the error_ctu_flag value of the D0 block to 1. Simultaneously, the error propagation flag (error_ctu_flag) of the D0 block will be saved to the external storage module and sent to the reconstruction and restoration module. The reconstruction and restoration module determines the error propagation flag (error_ctu_flag) value of each CTU; a value of 1 indicates that no image residual is accumulated, while a value of 0 indicates that the residual is accumulated normally. If the error_ctu_flag value of the E0 block is 0, then the error_ctu_flag value of the D0 block is also 0, and the reconstruction module accumulates the residual normally, consistent with the normal decoding process, with no difference. The reference frame F0 can be an erroneous frame or a normal frame that references an erroneous frame.

[0083] In one embodiment, the normal decoding process of the video stream in the decoding device includes: the main control module is configured to send decoding instructions to the entropy decoding module; the entropy decoding module is configured to parse the original residual and syntax from the video stream, transmit the original residual to the residual cache module, and transmit the syntax to the syntax cache module; the syntax cache module is configured to be accessed in turn by the inverse quantization and inverse transform module, the prediction module, and the filtering module, and each storage address of the syntax cache module is only released after all three modules have finished accessing it; the inverse quantization and inverse transform module is configured to perform inverse quantization and inverse transform calculations based on the original residual and the syntax, and output the image residual; the prediction module is configured to output intra-frame or inter-frame prediction values ​​based on the syntax, using intra-frame prediction and inter-frame prediction; the reconstruction and restoration module is configured to accumulate the image residual and the prediction values ​​to obtain a reconstructed image, output the reconstructed image to the filtering module for image deblocking and pixel value correction adjustment, and output a frame image to the storage module.

[0084] Please see Figure 12 The image shows a schematic diagram of the chip structure according to an embodiment of this disclosure. The chip provided in this embodiment includes: a storage module 2 and the decoding device described above. The decoding device includes a device 1 for processing video decoding errors.

[0085] Combination Figure 7 , Figure 10and Figure 11 The storage module 2 is an external storage module 2.

[0086] On one hand, during the error propagation calibration and update process, assuming the syntax input is selected, the header of the device-constructed error CTU (D0 position) is chosen. After this header is sent to the syntax buffer, the prediction module accesses the syntax buffer and can parse that the current CTU block is incorrectly constructed. At this time, the prediction module will perform error propagation calibration on D0 (set error_ctu_flag to 1). Then, the prediction module will save the error propagation calibration (error_ctu_flag) and colmv information together to the external storage module, thus completing the error calibration. Both the error propagation calibration (error_ctu_flag) and colmv information are stored in units of CTU. If the syntax input is selected, and D0 is the CTU header parsed by the normal entropy decoding module, then the error propagation calibration (error_ctu_flag) value is 0. In this case, the error propagation calibration (error_ctu_flag) still needs to be saved to the external storage module and sent to the reconstruction and restoration module.

[0087] On the other hand, during the decoding process of the current frame, when error propagation calibration is applied, it is assumed that video frame F0 is the reference frame for the current frame F1 and is stored in the external storage module. The current frame F1 is the frame that the decoder is currently decoding. It is also assumed that the error propagation calibration (error_ctu_flag) value of block E0 is 1 and is stored in the external storage module. Figure 1-7 and Figure 1-8 As shown, since the D0 block of the current frame F1 references the E0 block, the D0 block of the current frame F1 will retrieve the error propagation flag (error_ctu_flag) of the E0 block from the external storage module, and then update the error_ctu_flag value of the D0 block to 1. Simultaneously, the error propagation flag (error_ctu_flag) of the D0 block will be saved to the external storage module and sent to the reconstruction and restoration module. The reconstruction and restoration module determines the error propagation flag (error_ctu_flag) value of each CTU; a value of 1 indicates that no image residual is accumulated, while a value of 0 indicates that the residual is accumulated normally. If the error_ctu_flag value of the E0 block is 0, then the error_ctu_flag value of the D0 block is also 0, and the reconstruction module accumulates the residual normally, consistent with the normal decoding process, with no difference. The reference frame F0 can be an erroneous frame or a normal frame that references an erroneous frame.

[0088] The scope of protection of the method for handling video decoding errors described in this disclosure is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this disclosure is included within the scope of protection of this disclosure.

[0089] This disclosure also provides an apparatus, a decoding device, and a chip for processing video decoding errors, which can implement the method for processing video decoding errors described in this disclosure. However, the apparatus for implementing the method for processing video decoding errors described in this disclosure includes, but is not limited to, the structures of the apparatus, decoding device, and chip for processing video decoding errors listed in this embodiment. Any structural modifications and substitutions of the prior art made based on the principles of this disclosure are included within the protection scope of this disclosure.

[0090] In the several embodiments provided in this disclosure, it should be understood that the disclosed apparatus or method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0091] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this disclosure, depending on actual needs. For example, the functional modules / units in the various embodiments of this disclosure may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0092] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0093] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0094] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. An apparatus for handling video decoding errors, the apparatus comprising: comprises: a time counting module configured to start counting based on a decoding action of a video code stream, and generate a timeout error in response to a counting time exceeding a preset time threshold; an error control module configured to receive a code stream error generated during decoding of the video code stream and the timeout error generated by the time counting module, select a coordinate to be constructed according to tile information, tile count, horizontal coordinate and vertical coordinate corresponding to a motion vector coordinate and tile information, tile count, horizontal coordinate and vertical coordinate corresponding to a filter image coordinate reported when video decoding fails, and initiate a reset request after decoding is paused; and in response to completion of the reset, issue a construction instruction information in combination with the constructed coordinate; and a syntax construction module configured to construct a syntax according to the construction instruction information, and the constructed syntax is used for normal decoding after video decoding fails.

2. The apparatus of claim 1, wherein, The error control module is configured to: in response to tile counts of the motion vector coordinate and the filter image coordinate being not equal, compare the tile counts and select a coordinate with a smaller tile count to construct a coordinate; in response to the tile counts of the motion vector coordinate and the filter image coordinate being equal, first compare vertical coordinates, and if the vertical coordinates of the motion vector coordinate and the filter image coordinate are not equal, select a coordinate with a smaller vertical coordinate value to construct a coordinate; and if the vertical coordinates of the motion vector coordinate and the filter image coordinate are equal, compare horizontal coordinates of the motion vector coordinate and the filter image coordinate, and select a coordinate with a smaller horizontal coordinate value to construct a coordinate.

3. The apparatus of claim 1, wherein, The construction instruction information comprises an error construction command, and the syntax construction module is configured to: construct a slice syntax and / or a tile syntax according to the error construction command, and then construct a coding tree unit syntax and a coding unit syntax; determine whether an intra mode or an inter mode is configured, if the intra mode is configured, construct a transform unit syntax, and if the inter mode is configured, perform frame end determination and tile end determination; in the intra mode, in response to completion of construction of the transform unit syntax, perform frame end determination and tile end determination; in performing the frame end determination and the tile end determination, in response to the tile end and the frame not ending, return to a step of constructing the tile syntax; and in response to the frame ending, send a construction completion signal to the error control module.

4. The apparatus of claim 3, wherein: the coding tree unit syntax constructed by the syntax construction module comprises an error identification information header, and the error identification information header is used to identify that the construction of a current coding tree unit block is derived from the error control module, so as to prevent error propagation.

5. A method for handling video decoding errors, the method comprising: comprises: start counting based on a decoding action of a video code stream, and generate a timeout error in response to a counting time exceeding a preset time threshold; receive a code stream error generated during decoding of the video code stream and the timeout error, select a coordinate to be constructed according to tile information, tile count, horizontal coordinate and vertical coordinate corresponding to a motion vector coordinate and tile information, tile count, horizontal coordinate and vertical coordinate corresponding to a filter image coordinate reported when video decoding fails, and initiate a reset request after decoding is paused; and in response to completion of the reset, the syntax construction module sends construction instruction information in combination with the constructed coordinates; and According to the construction instruction information, the syntax is constructed, and the constructed syntax is used for normal decoding after video decoding error.

6. A decoding apparatus, characterized by comprising: Comprise: The apparatus for processing video decoding error according to any one of claims 1 to 4.

7. The decoding apparatus of claim 6, wherein, Also include: Entropy decoding module, residual buffer module, syntax buffer module, inverse quantization and inverse transform module, prediction module, filtering module, reconstruction module and general control module, The general control module is configured to send decoding instructions to the entropy decoding module; The entropy decoding module is configured to start decoding the video code stream based on the decoding instructions; The time counting module is configured to start counting based on the decoding action of the video code stream, generate a code stream error in response to the video code stream being found to have an error by the entropy decoding module, or generate a timeout error when the counting time of the entire decoding process exceeds a preset time threshold; The error control module is configured to receive the code stream error and the timeout error, select the coordinates to be constructed according to the coordinate information reported by the prediction module and the filtering module, and initiate a reset request to the general control module after decoding is paused; The general control module is configured to reset all modules except the apparatus for processing video decoding error in response to the reset request, and send a reset completion signal to the error control module in response to the completion of the reset of all modules except the apparatus for processing video decoding error, so that the error control module sends construction instruction information to the syntax construction module in combination with the constructed coordinates; The syntax construction module is configured to construct syntax according to the construction instruction information, transmit the syntax to the syntax buffer module, and use the constructed syntax for normal decoding after video decoding error, and in response to the completion of the construction of the syntax construction module, send a construction completion signal to the error control module to indicate the end of the error processing process.

8. The decoding apparatus of claim 7, wherein, The reported coordinates include motion vector coordinates and filtering image coordinates, and the reported coordinate information includes tile information, tile count, horizontal coordinates and vertical coordinates corresponding to the motion vector coordinates and the filtering image coordinates, respectively. The error control module is configured to: In response to the tile counts of the motion vector coordinates and the filtering image coordinates being not equal, compare the tile counts and select the coordinates with smaller tile count to construct coordinates; In response to the tile counts of the motion vector coordinates and the filtering image coordinates being equal, first compare the vertical coordinates, and if the vertical coordinates of the motion vector coordinates and the filtering image coordinates are not equal, select the coordinates with smaller vertical coordinate value to construct coordinates; And If the vertical coordinates of the motion vector coordinates and the filtering image coordinates are equal, compare the horizontal coordinates of the motion vector coordinates and the filtering image coordinates, and select the coordinates with smaller horizontal coordinate value to construct coordinates.

9. The decoding apparatus of claim 7, wherein, The construction instruction information includes an error construction command, and the syntax construction module is configured to: According to the error construction command, construct slice syntax and / or tile syntax, and then construct coding tree unit syntax and coding unit syntax; determining whether the configuration is an intra mode or an inter mode, constructing a transform unit syntax if the configuration is the intra mode, and performing a frame end determination and a tile end determination if the configuration is the inter mode; in the intra mode, performing the frame end determination and the tile end determination in response to completion of the transform unit syntax construction; in the performing of the frame end determination and the tile end determination, returning to the step of constructing the tile syntax in response to the tile end and the frame not being ended; and sending a construction completion signal to the error control module in response to the frame being ended.

10. The decoding apparatus of claim 9, wherein: the syntax construction module constructs the coding tree unit syntax including an error identification information header, the error identification information header being used to identify that the current coding tree unit block is constructed from the error control module to prevent error propagation.

11. The decoding apparatus of claim 10, wherein, the information header selected by the syntax input includes an entropy decoding parsed information header or the error identification information header, in response to the information header selected by the syntax input being the error identification information header, the error identification information header is sent to the syntax buffer module, the prediction module is configured to access the syntax buffer module to parse that the current coding tree unit block is constructed by the syntax construction module, the prediction module is configured to perform error propagation flag setting, and then save the error propagation flag value and the helper frame motion vector information to the storage module to complete error flagging; in response to the information header selected by the syntax input being the entropy decoding parsed information header, the prediction module is configured to perform error propagation flag resetting, and then save the error propagation flag value and the helper frame motion vector information to the storage module.

12. The decoding apparatus of claim 11, wherein, a reference tile corresponding to a reference position in a reference frame for a current tile of a current frame in a decoding process, the prediction module is configured to obtain an error propagation flag value of the reference tile, determine an error propagation flag value of the current tile of the current frame according to the error propagation flag value of the reference tile, save the error propagation flag value of the current tile of the current frame to the storage module and transmit to the reconstruction module; the reconstruction module is configured to determine whether the current tile is normally accumulated with a residual according to the error propagation flag value.

13. The decoding apparatus of claim 7, wherein: the total control module is configured to send a decoding instruction to the entropy decoding module, and the entropy decoding module is configured to parse an original residual and a syntax from a video bitstream, transmit the original residual to the residual buffer module, and transmit the syntax to the syntax buffer module; the syntax buffer module is configured to be accessed by the inverse quantization and inverse transform module, the prediction module and the filtering module in turn, and each storage address of the syntax buffer module is released only after the inverse quantization and inverse transform module, the prediction module and the filtering module all end the access; the inverse quantization and inverse transform module is configured to perform inverse quantization and inverse transform calculation according to the original residual and the syntax, and output an image residual; the prediction module is configured to output an intra or inter prediction value according to the syntax based on intra prediction and inter prediction; and the reconstruction module is configured to reconstruct the image residual according to the output of the prediction module. The reconstruction reduction module is configured to accumulate the image residual and the prediction value to obtain a reconstructed image, output the reconstructed image to the filtering module for image blocking effect and pixel value correction and adjustment, and output a frame image to the storage module.

14. A chip, characterized by Comprising: a storage module; and The decoding device according to any one of claims 6 to 13.

Images