Code stream processing method, decoding device, chip and electronic equipment

Abstract

The invention relates to the technical field of video decoding, and discloses a code stream processing method, a decoding device, a chip and electronic equipment. According to the method, a storage unit is additionally arranged in the decoding device, recovery information of a video frame which is decoded in an interrupted mode when the decoding device is suspended is recorded through the storage unit, and decoding of the video frame which is decoded in the interrupted mode is recovered based on the recorded recovery information when the decoding device is restarted. Specifically, when the decoding device is suspended, the storage unit can store first recovery information corresponding to a first bit for entropy decoding interruption, such as position information of the first bit and a corresponding entropy context, and store second recovery information corresponding to a first macro block for decoding after interruption, such as position information of the first macro block and corresponding state information. Therefore, when the decoding device is restarted, entropy decoding and post-decoding can be recovered, and image missing or pixel missing of the decoded video is avoided.

Description

Technical Field

[0001] The present application relates to the field of video decoding technology, and in particular to a code stream processing method, a decoding device, a chip and an electronic device. Background Art

[0002] With the continuous advancement of video compression technology, the continuous evolution of encoding tools in video encoding and decoding scenarios has also led to the continuous updating of corresponding decoders. The enhancement of encoding tools aims to improve video compression efficiency and video quality, which is usually accompanied by an increase in the hardware complexity of the corresponding decoder, such as the increase in the complexity of register transfer level (RTL) design at the hardware implementation level. As the complexity of RTL design increases, new encoding tools may encounter more corner cases during operation, causing hardware failures of the decoder, such as a hardware failure of the encoding tool causing a kernel hang during the decoding process.

[0003] Specifically, during the decoding of the video code stream, if the kernel hangs, the decoding of the video code stream will be suspended until the decoder is restarted. The decoding process generally performs entropy decoding on each video frame in the video code stream according to the entropy context and stores the generated decoding information such as motion vector difference, and then calls the decoding information of each video frame according to the current frame number and other status information to perform motion compensation and other post-decoding on each video frame, thereby generating image pixels of each video frame.

[0004] However, when the decoding process is suspended, the decoding information of some currently stored video frames has not been called, and these decoding information may not be recognized when the decoder is restarted later due to the loss of state information such as frame numbers, resulting in the failure of post-decoding of video frames in the video bitstream that rely on these decoding information for post-decoding processing. As a result, the final decoded video has missing images or missing image pixels. Summary of the invention

[0005] The embodiments of the present application disclose a code stream processing method, a decoding device, a chip and an electronic device.

[0006] In a first aspect, an embodiment of the present application provides a code stream processing method, which is applied to a decoding device including a first storage unit, the method including: entropy decoding and post-decoding the code stream, the code stream including a first data stream and a first macroblock row; in the process of entropy decoding the first bit of the first data stream and post-decoding the first macroblock in the first macroblock row, it is detected that a hardware failure occurs in the decoding device, and first recovery information and second recovery information are stored in the first storage unit, wherein the first recovery information includes position information of the first bit and an entropy context corresponding to the first bit, and the second recovery information is used to indicate the position information of the first macroblock and the status information corresponding to the first macroblock; when it is detected that the decoding device has completed restarting, the first recovery information and the second recovery information are obtained from the first storage unit, and the first data stream where the first bit in the code stream is located is continued to be entropy decoded according to the first recovery information, and the first macroblock row where the first macroblock in the code stream is located is continued to be post-decoded according to the second recovery information. In this way, the present application uses the recovery information recorded in the newly added storage unit to resume the entropy decoding of the video frame where the first bit is located and the post-decoding of the video frame where the first macroblock is located after the subsequent decoding device is restarted, and the data of the two video frames will not be discarded, so that the decoded video contains the image pixels of the two video frames. Therefore, the situation of image loss or image pixel loss in the decoded video is improved, and the video quality is improved.

[0007] In a possible implementation of the first aspect, the first bit in the code stream is determined according to the position information in the first recovery information, and the first data stream where the first bit in the code stream is located is further entropy decoded according to the entropy context in the first recovery information. In this way, the code stream can be accurately entropy decoded from the first bit in the first data stream through the position information of the first bit in the first recovery information, without re-entropy decoding the data before the first bit in the video frame where the first bit is located. The rapidity of recovery decoding is improved, computing resources are saved, and the occupied system bandwidth is reduced.

[0008] In a possible implementation of the first aspect above, the decoding device also includes: a second storage unit, the second storage unit is used to store the decoding information of each macroblock in the code stream, and the method also includes: corresponding to the decoding information of each macroblock in the first data stream generated by entropy decoding, storing the decoding information of each macroblock in the first data stream in the second storage unit; wherein the decoding information of the macroblock includes at least one of the following: the syntax element of the macroblock and the auxiliary information of the macroblock. Then, the code stream can be accurately continued to be post-decoded starting from the macroblock row where the first macroblock is located through the position information of the first macroblock in the second recovery information, without the need to re-post-decode other macroblock rows before the current macroblock row in the video frame where the first macroblock is located. The rapidity of recovery decoding is further improved, computing resources are saved, and the occupied system bandwidth is reduced.

[0009] In a possible implementation manner of the first aspect above, the status information of the first macroblock includes: a frame number of a video frame to which the first macroblock belongs, a coding technology corresponding to the first macroblock, and a filtering control parameter corresponding to the first macroblock.

[0010] In a possible implementation manner of the first aspect above, the second restoration information is set at a frame level, a slice level, a block level, or a stripe level.

[0011] In a possible implementation of the first aspect, the code stream is post-decoded starting from the first macroblock row where the first macroblock is located according to the second recovery information, including: obtaining the decoding information of each third macroblock before the first macroblock in the first macroblock row from the second storage unit according to the position information in the second recovery information, and post-decoding the corresponding third macroblock according to the decoding information of each third macroblock and the status information in the second recovery information; post-decoding the first macroblock in the first macroblock row and the fourth macroblock after the first macroblock according to the error repair method according to the position information in the second recovery information. It can be understood that when the decoding device restarts and resumes decoding, the macroblock row where the macroblock that had an error during suspension is located can be error-repaired to avoid errors in the macroblock row when resuming decoding.

[0012] In a possible implementation of the first aspect, the code stream is further post-decoded starting from the macroblock row where the first macroblock is located according to the second recovery information, including: obtaining the decoding information of each third macroblock before the first macroblock in the first macroblock row from the second storage unit according to the position information in the second recovery information, and post-decoding the corresponding third macroblock according to the decoding information of each third macroblock and the status information in the second recovery information; post-decoding the first macroblock in an error repair manner according to the position information in the second recovery information; obtaining the decoding information of each fifth macroblock after the first macroblock in the first macroblock row from the second storage unit according to the position information in the second recovery information, and post-decoding the corresponding fifth macroblock according to the decoding information of each fifth macroblock and the status information in the second recovery information. It can be understood that since the sizes of macroblocks in a macroblock row may be different, if an error occurs in a macroblock, the decoding device may not be able to determine the size of the macroblock. Therefore, error repair is performed on the macroblock where the error occurs, specifically, error repair is performed on all data starting from the macroblock in the current macroblock row. In this way, it can be ensured that the decoding process after the first macroblock row is restored will not cause errors, such as causing the kernel to hang.

[0013] In a possible implementation of the first aspect above, the process of post-decoding a macroblock in an error correction manner includes: using preset pixel data as pixel data for the macroblock, or using pixel data of a macroblock having the same position as the macroblock in a previous video frame adjacent to the video frame where the macroblock is located as pixel data for the first macroblock.

[0014] In a possible implementation of the first aspect above, the decoding device also includes: a third storage unit, the third storage unit is used to store pixel data of each macroblock in the code stream, and the method also includes: when completing the post-decoding of the first macroblock and generating the pixel data of the first macroblock, storing the pixel data of the first macroblock in the third storage unit.

[0015] In a possible implementation manner of the first aspect above, entropy decoding of the code stream and post-decoding of the code stream are performed asynchronously or synchronously.

[0016] In a possible implementation manner of the first aspect above, the hardware failure of the decoding device includes: a failure caused by an error in post-decoding of the first macroblock.

[0017] In a possible implementation of the first aspect above, the decoding device includes multiple cores, and performs entropy decoding and post-decoding on the code stream, including: entropy decoding each bit stream in the code stream in parallel through the multiple cores, and post-decoding each macroblock in the code stream in parallel through the multiple cores.

[0018] In a possible implementation manner of the first aspect above, the first storage unit includes multiple sub-storage units, and each sub-storage unit corresponds to one core among the multiple cores.

[0019] In a possible implementation manner of the first aspect above, the second storage unit and the third storage unit are arranged in a double data rate synchronous dynamic random access memory DDR in the decoding device.

[0020] In a second aspect, an embodiment of the present application provides a decoding device, which is used to execute the code stream processing method in the above-mentioned first aspect and any possible implementation manner thereof.

[0021] In a third aspect, an embodiment of the present application provides a chip, including a circuit, wherein the circuit is used to execute the bit stream processing method in the above-mentioned first aspect and any possible implementation manner thereof.

[0022] In a fourth aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes the chip as described in the third aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1According to some embodiments of the present application, a schematic diagram of a video encoding and decoding scenario is shown;

[0024] Figure 2A According to some embodiments of the present application, a schematic diagram of the structure of a video code stream is shown;

[0025] Figure 2B According to some embodiments of the present application, a schematic diagram of inputs and outputs of various hardware units of a decoding device 100A is shown;

[0026] Figure 2C According to some embodiments of the present application, a schematic diagram of a process of a decoding device 100A performing code stream processing based on each hardware unit is shown;

[0027] Figure 3 According to some embodiments of the present application, a timing diagram of an image frame decoded by a decoding device 100A is shown;

[0028] Figure 4 According to some embodiments of the present application, a schematic diagram of a decoding device 100A restoring a decoded video frame is shown;

[0029] Figure 5A According to some embodiments of the present application, a schematic diagram of input and output of each hardware unit of a decoding device 100B is shown;

[0030] Figure 5B According to some embodiments of the present application, a schematic diagram of a process of a decoding device 100B performing code stream processing based on each hardware unit is shown;

[0031] Figure 6 According to some embodiments of the present application, a schematic diagram of a decoding device 100A restoring a decoded video frame is shown;

[0032] Figure 7 According to some embodiments of the present application, a schematic flow chart of a code stream processing method is shown;

[0033] Figure 8 According to some embodiments of the present application, a schematic diagram of a process of performing error repair on post-decoding of a macroblock row where an interrupted macroblock is located is shown;

[0034] Fig. 9 According to some embodiments of the present application, a schematic diagram of a process of performing error repair on post-decoding of a macroblock row where an interrupted macroblock is located is shown;

[0035] Fig.10 According to some embodiments of the present application, a structural schematic diagram of an electronic device is shown;

[0036] Fig.11 According to some embodiments of the present application, a block diagram of a system on chip (SoC) is shown. DETAILED DESCRIPTION

[0037] The illustrative embodiments of the present application include but are not limited to a code stream processing method, a medium, and an electronic device.

[0038] The code stream processing method provided in this application can be applied to the fields of automobiles, embedded devices, video surveillance, smart homes, medical treatment, and the Internet of Things. Specifically, the code stream processing method can be applied to some full HD video transmission scenarios, such as decoding and processing videos involved in virtual reality (VR), augmented reality (AR), mixed reality (MR), 1080P, 4K and other video transmission scenarios. The video compression standards involved in this application include but are not limited to the following standards: High efficiency video coding (HEVC), advanced video coding (AVC) and source coding standard (AVS).

[0039] Figure 1 According to an embodiment of the present application, a video encoding and decoding scene graph is shown. Figure 1 The scenario in FIG. 1 includes a first electronic device 10 and a second electronic device 20 connected via one or more networks.

[0040] In some embodiments, the device types of the first electronic device 10 and the second electronic device 20 include but are not limited to: smart phones, vehicle-mounted devices, personal computers, artificial intelligence devices, tablet computers, personal digital assistants, smart wearable devices (such as smart watches or bracelets, smart glasses), smart TVs (or called smart big screens, smart screens, or large-screen TVs, etc.), virtual reality / mixed reality / augmented display devices, servers, etc.

[0041] In some embodiments of the present application, when the second electronic device 20 (e.g., a server) transmits video data to the first electronic device 10 (e.g., a smart TV) based on a communication method such as Bluetooth, a wireless network, or a mobile network, the second electronic device 20 can encode the source video to obtain a video bit stream; the second electronic device 20 can send the video bit stream to the first electronic device 10; the first electronic device 10 decodes the video bit stream to obtain decoded video data.

[0042] The decoding of the code stream is entropy decoding and post-decoding in sequence. Post-decoding may include: inverse quantization, inverse transformation, intra-frame / inter-frame prediction, motion compensation, image reconstruction, loop filtering, pixel output and other processing. Of course, the above-mentioned post-decoding includes but is not limited to the above-mentioned examples. For example, before the pixel output of the image, the post-decoding may also include other processing such as color space conversion and frame rate conversion.

[0043] The decoding device for executing the code stream processing method in the present application may be a module or unit for executing video decoding in the first electronic device 10, for example, the decoding device may be referred to as a decoder. Specifically, the decoding device may be arranged in a system on chip (SoC) in the first electronic device 10, such as a video processing unit (VPU) in the first electronic device 10. As an example, the decoding device is an intellectual property unit (IP) in the VPU, i.e., an IP core.

[0044] Taking the H.264 video standard as an example, refer to Figure 2A The structure diagram of the video code stream in the figure explains the structure of the video code stream. Figure 2A As shown, the video bitstream can be divided into multiple sequences. A sequence is a data stream after a segment of image is encoded. A sequence can include multiple frames of images. The first I frame in a sequence is an instantaneous decoding refresh (IDR) image, and a sequence can include P frames and B frames. It can be understood that an I frame is an intra-picture, which is a complete compressed image. A P frame is a predictive-picture, which records the motion vector of the previous frame. A B frame is a bi-directional interpolated prediction frame, which records a motion vector based on the next frame and the previous frame.

[0045] Furthermore, an image frame (e.g., a B frame) can be divided into multiple slices, a slice includes multiple macroblocks (MB), and a macroblock can include multiple sub-macroblocks. For example, a macroblock includes 8×8, 4×8, or 8×4 pixel sub-macroblocks. In addition, the difference between a block and a slice is that a block must be rectangular. Furthermore, a stripe can include one or more macroblock rows.

[0046] Based on the above Figure 2AThe structure shown in FIG. 1 illustrates the decoding process.

[0047] In some embodiments, the present application may decode the bitstream of the video according to the image unit classification. For example, the image unit classification may be frame, slice, tile, stripe, CTU / MBline, macroblock, etc. Macroblocks may have different names in different video standards. For example, in some video standards, macroblocks may be represented as CTUs, i.e., coding tree units.

[0048] As described in the background technology above, when the first electronic device 10 is suspended during the decoding process, the first data stream in the first video frame currently performing entropy decoding will interrupt the entropy decoding, and the first macroblock in the second video frame currently performing post-decoding will interrupt the post-decoding. At this time, the first electronic device 10 will delete the entropy context of the first video stream and delete the state information such as the frame number corresponding to the first macroblock. Then, when the first electronic device 10 is restarted, due to the lack of the entropy context related to the first video frame that interrupted the entropy decoding and the state information of the first macroblock related to the second video frame that interrupted the post-decoding, the decoding information of the image frame between the two frames of images will not be recognized, resulting in the video frames in the video code stream that rely on these decoding information for post-decoding processing cannot be normally post-decoded. Furthermore, the first electronic device 10 will delete the first video frame and the second video frame that interrupted the entropy decoding, as well as the video frame between the two frames of images, resulting in image missing and pixel missing in the final decoded video. For example, Figure 2A The decoding information of the first video frame, namely the I frame, in an image frame sequence shown cannot be recognized and cannot be decoded normally, which will cause the subsequent B frames and P frames in the image frame sequence to be unable to be decoded normally, so that the image frame sequence is missing in the decoded video frames.

[0049] In order to solve the problem of missing images or pixels in decoded videos, an embodiment of the present application provides a code stream processing method. The method adds a storage unit in a decoding device, records the recovery information of the video frames whose decoding is interrupted when the decoding device is suspended, and resumes decoding of the interrupted video frames based on the recorded recovery information when the decoding device is restarted.

[0050] Specifically, as mentioned above, the decoding unit of entropy decoding is a bit, and the decoding unit of post-decoding is a macroblock. It is assumed that when the decoding device is suspended, the first bit in the decoded bit stream is currently being entropy decoded, and the first bit belongs to the first bit stream. The first macroblock is currently being post-decoded, and the first macroblock belongs to the first macroblock row. At this time, the decoding device can store the first recovery information corresponding to the first bit, such as the position information of the first bit and the corresponding entropy context, in the storage unit, and store the second recovery information corresponding to the first macroblock, such as the position information of the first macroblock and the corresponding state information, in the storage unit. Subsequently, when it is detected that the decoding device has completed restarting, the first recovery information and the second recovery information are obtained from the above storage unit, and the first data stream in the code stream continues to be entropy decoded from the first bit according to the first recovery information, and the code stream continues to be post-decoded from the first macroblock row where the first macroblock is located according to the second recovery information, so as to restore the entropy decoding and post-decoding of the code stream.

[0051] In this way, the present application uses the recovery information recorded in the newly added storage unit to resume the entropy decoding of the video frame where the first bit is located and the post-decoding of the video frame where the first macroblock is located after the subsequent decoding device is restarted, and the data of the two video frames will not be discarded, so that the decoded video contains the image pixels of the two video frames. Therefore, the situation of image loss or image pixel loss in the decoded video is improved, and the video quality is improved.

[0052] In addition, when the decoding device is restarted, the present application can accurately continue to entropy decode the code stream from the first bit in the first data stream through the position information of the first bit in the first recovery information, without the need to re-entropy decode the data before the first bit in the video frame where the first bit is located. And, through the position information of the first macroblock in the second recovery information, the code stream can be accurately continued to be post-decoded from the macroblock row where the first macroblock is located, without the need to re-post-decode other macroblock rows before the current macroblock row in the video frame where the first macroblock is located. In this way, the rapidity of recovery decoding is improved, computing resources are saved, and the occupied system bandwidth is reduced.

[0053] In some embodiments, the newly added storage unit may be a buffer in the decoding device. For the convenience of description, the storage unit may be referred to as a first storage unit or a cache unit in the present application. In addition, the decoding device may store data such as a code stream of the video through other storage units different from the first storage unit, such as a memory such as a double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR).

[0054] Next, combine Figure 2B , Figure 2C , Figure 3and Figure 4 The decoding process of a decoding device is described. Figure 2B The diagram shows the hardware units included in the decoding apparatus 100A and the input and output of each hardware unit. Figure 2C The flowchart of the decoding apparatus 100A performing code stream processing based on each hardware unit is shown. Figure 3 The timing chart of the image frame decoded by the decoding device 100A is shown. Figure 4 FIG. 1 is a schematic diagram showing a decoding apparatus 100A restoring a decoded video frame.

[0055] Figure 2B The decoding device 100A shown includes the following hardware units: one or more cores (also called cores) 101 (such as core 101a or core 101b), a distributor 102, a core scheduler 103 and a double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR) 104.

[0056] The distributor 102 is used to obtain a code stream from the DDR 104, identify the starting position of the image unit classification in the code stream, such as a start code, and send data streams (steams) of different images in the code stream to different cores 101 for decoding.

[0057] The kernel scheduler 103, also called the arbitration unit, is used to schedule the behavior of each kernel 101, such as starting or stopping the kernel. Specifically, after the kernel scheduler 103 starts the kernel 101, it can receive a response message returned by the kernel 101, such as a response message that the kernel 101 completes decoding of a frame of image.

[0058] DDR 104 is used to store the bitstream of the video, the decoding information (denoted as decinfo) obtained by entropy decoding the bitstream, and the decoded pixels (dec pel) obtained by post-decoding the bitstream, etc. The bitstream, decoding information and decoded pixels can be stored in the same or different storage units in DDR 104.

[0059] Specifically, Figure 2B The illustrated core 101 includes a core 101a and a core 101b. Each core 101 includes a parsing engine (PE) and a decoding engine (DE). The PE is used to entropy decode the bitstream, and the DE is used to post-decode the bitstream. The PE outputs the decoded information of the entropy decoding to the DDR 104, and the PE and the DE can be decoupled. The PE and the DE in the core can run synchronously or asynchronously. As an example, Figure 2BThe encoding device 100A in the embodiment is a dual-core asynchronous system, the two PEs PE 01 and PE 02 can process different images simultaneously, and the two DEs DE 01 and DE 02 can process other images simultaneously, for example, the two DEs can process different macroblock rows in the same image simultaneously.

[0060] like Figure 2C As shown in Figure 2B The flowchart of a code stream processing method in an asynchronous decoding scenario shown in FIG. 1 is executed by a decoding device 100A, and includes the following steps:

[0061] S201: Use the distributor 102 to obtain the video code stream from the DDR 104.

[0062] S202: Use the distributor 102 to divide the code stream into various data streams.

[0063] For example, data stream A1 and data stream A2 are divided from the code stream.

[0064] S203 : The core 101 a is used to obtain the data stream A1 from the distributor 102 .

[0065] S204 : Use the core 101 b to obtain the data stream A2 from the distributor 102 .

[0066] The distributor 102 can schedule the core 101a through the core scheduler 103, and the core 101a obtains different data streams.

[0067] For example, data stream A1 and data stream A2 are different slices in different images.

[0068] S205 : Use PE 01 in the core 101 a to perform entropy decoding on the data stream A1 according to the entropy context of the data stream A1 , and store the generated decoding information B1 of each macroblock in the data stream A1 into the DDR 104 .

[0069] In entropy decoding, the entropy context is mainly used to guide the decoding process and ensure that the decoder can correctly parse the syntax elements in the compressed bitstream.

[0070] In some embodiments, a data stream may be entropy decoded bit by bit.

[0071] In some embodiments, the decoding information of a macroblock may include: syntax elements and auxiliary information.

[0072] The syntax elements of a macroblock include at least one of the following information: macroblock type (MB type), prediction mode (prediction mode), motion vectors (MVs), coded block pattern (CBP), and quantized transform coefficients (quantized transform coefficients).

[0073] The auxiliary information of a macroblock includes at least one of the following information: a quantizer parameter (QP) and a slice type.

[0074] S206 : Use PE 02 in the core 101 b to perform entropy decoding on the data stream A2 according to the entropy context of the data stream A2 , and store the generated decoding information B1 of each macroblock in the data stream A2 in the DDR 104 in sequence.

[0075] The data stream A1 and the data stream A2 may be entropy decoded in parallel.

[0076] S207: Use DE01 in the core 101a to obtain the decoding information C1 of the macroblock 1 in the macroblock row S1 from the DDR 104.

[0077] S208: Use DE01 in the core 101a to post-decode macroblock 1 according to the status information of macroblock row S1 and the decoding information C1 of macroblock 1.

[0078] Assuming that the post-decoding of the macroblock 1 is completed normally, the PE 01 may generate the decoded pixel D1 of the macroblock 1 and store the decoded pixel D1 in the DDR 104 .

[0079] In some embodiments, the state information of a macroblock may be set at the level of frame, slice, block, stripe or macroblock row, etc. For example, in the embodiment of the application, the state information of the macroblock is mainly set at the level of macroblock row.

[0080] In some embodiments, the status information of a macroblock is used to guide how to post-decode the macroblock row where the macroblock is located. For example, the status information of a macroblock row includes at least one of the following: the frame number of the video frame to which the macroblock row belongs, the encoding technology corresponding to the macroblock row, and the filtering control parameters corresponding to the macroblock row. The above-mentioned encoding and decoding technology can be fine granularity scalable (FGS) technology, etc., indicating that the macroblocks in the current video frame are decoded according to the encoding technology. The above-mentioned filtering control parameters may include α and β, which are used to adjust the performance of the filter to meet specific image processing requirements. Among them, in the α-β filter, α controls the filtering effect of the low-pass filter channel (α channel) and affects the smoothness of the filtered signal. In the α-β filter, β is used to adjust the filtering effect of the high-pass filter channel (β channel), which affects the response speed of the filtered signal to high-frequency components.

[0081] S209: Use DE 02 in the core 101b to obtain decoding information C2 of macroblock 2 in macroblock row S2 from DDR 104.

[0082] S210: Use DE 02 in core 101b to post-decode macroblock 2 according to the status information of macroblock row S2 and decoding information C2 of macroblock 2.

[0083] S211: When bit 1 in data stream A1 is entropy decoded and macroblock 1 is post-decoded, kernel 101a is used to detect that a kernel hang has occurred and delete the entropy context of data stream A1 and the status information of macroblock row S1.

[0084] In some embodiments, the decoding device may have a kernel hang during the decoding process of the code stream due to high RTL complexity.

[0085] At this time, PE 01 in the core 101a interrupts the entropy decoding of the data stream A1, and DE 01 interrupts the post-decoding of the macroblock row S1.

[0086] Assuming that the entropy decoding of data stream A1 is completed normally, PE 01 can store the decoding information B1 of each macroblock in data stream A1 to DDR 104. Assuming that the post-decoding of macroblock 1 is completed normally, DE 01 can generate the decoded pixel D1 of macroblock 1 and store the decoded pixel D1 to DDR 104.

[0087] S212: When bit 2 in data stream A2 is entropy decoded and macroblock 2 is post-decoded, kernel 101b detects that a kernel hang has occurred and deletes the entropy context of data stream A2 and the status information of macroblock row S2.

[0088] At this time, PE 02 in core 101b interrupts the entropy decoding of data stream A1, and DE 02 interrupts the post-decoding of macroblock row S2.

[0089] Assuming that the entropy decoding of data stream A2 is completed normally, PE 02 can store the decoding information B2 of each macroblock in data stream A2 to DDR 104. Assuming that the post-decoding of macroblock 2 is completed normally, DE 02 can generate the decoded pixel D2 of macroblock 2 and store the decoded pixel D2 to DDR 104.

[0090] In some embodiments, each core deletes the corresponding entropy context and state information when the decoding device is shut down.

[0091] S213: When the decoding device 100A is restarted, the video frame where the data stream A1 is located is discarded to the video frame where the macroblock 2 is located.

[0092] Figure 3 The processing status of each macroblock in the 3rd to 8th frame images in a bitstream is shown. The processing status of the macroblock includes: entropy decoding completed, post-decoding available, decoding completed, and undecoded. For example, Figure 3 In the 3rd frame, all macroblocks are decoded. In the 4th frame, all macroblocks in the 1st row, macroblocks before macroblock 1 in the 2nd row (macroblock row S1), and macroblocks before macroblock 2 in the 3rd row (macroblock row S3) are decoded. Macroblocks 1 and 2 in the 4th frame can be decoded later, and all other macroblocks are entropy decoded. All macroblocks in the 5th frame are decoded. In the 6th frame, all macroblocks before macroblock 3 are entropy decoded, while all other macroblocks are not decoded. In the 7th frame, all macroblocks before macroblock 4 are entropy decoded, while all other macroblocks are not decoded. All macroblocks in the 8th frame are not decoded.

[0093] It can be understood that under normal bit rates, the speed of entropy decoding is usually higher than the speed of post-decoding. Figure 2C The process shown in Figure 3 In the video frame decoding sequence shown, the decoding device 100A performs entropy decoding on the 6th and 7th frames of the code stream in parallel through PE 01 and PE02, and performs post-decoding on the macroblock row S1 and macroblock row S2 in the 4th frame of the code stream in parallel through DE 01 and DE 02. At this time, the entropy decoding of the 7th frame of the image has begun, and the 4th frame of the image is still being post-decoded. In addition, there is a dependency relationship between the post-decoding of macroblocks, that is, some post-decoding processing of a macroblock in a frame of an image (such as intra-frame / inter-frame prediction) may depend on its adjacent decoded macroblocks, and may even depend on the decoded macroblocks at the corresponding position in the adjacent image frame. Figure 3 As shown, the block decoding of macroblock 1 in macroblock row S1 depends on the decoded data of the four adjacent macroblocks on the left, upper left, upper side, and upper right.

[0094] However, when the kernel of the decoding device 100A is suspended due to the high RTL complexity, the entropy context of the data stream A1 in the 6th frame image where entropy decoding is interrupted and the entropy context of the data stream A2 in the 7th frame image will be discarded, and the status information of the macroblock row S1 and the status information of the macroblock row S2 in the 4th frame image where post-decoding is interrupted will also be lost, which will cause the images between the 4th frame image to the 7th frame image to be unable to be decoded. Furthermore, when the decoding device is restarted, the decoding information of each macroblock in the 4th frame image to the 7th frame image has not been used, and this decoding information may not be used due to reasons such as the lack of status information of the macroblock row in these video frames, thereby causing the images between the 4th frame image to the 7th frame image to be unable to be decoded. In this way, when the decoding device is restarted, the decoding device will delete the video frame where entropy decoding is interrupted, the video frame where post-decoding is interrupted, and the video frame that has started entropy decoding but has not started post-decoding, that is, delete the 4th frame image to the 7th frame image, resulting in missing video frames and missing pixels in the decoded video.

[0095] Specifically, Figure 4 As shown, when the decoding device 100A restarts and resumes decoding, after deleting the 4th to 7th frame images, entropy decoding and post-decoding will be resumed from the 8th frame image, that is, the 8th frame image is a new starting point for PE and DE. For example, for the 8th frame image, entropy decoding can be performed by PE 01 in the core 101a, and post-decoding of different macroblock rows can be performed in parallel by DE01 in the core 101a and DE 02 in the core 101b. In this way, the data of the 5th to 7th frame images may be missing in the decoded video, that is, the video has more image missing.

[0096] Next, in Figure 3 Based on the video frame processing flow shown, combined with Figure 5A , Figure 5B and Figure 6 The decoding process of the decoding device with a newly added cache unit is described. Figure 5A The hardware units included in the decoding device 100B and the input and output of each hardware unit are shown. Figure 5B The flowchart of code stream processing performed by the decoding device 100B using each hardware unit is shown. Figure 6 FIG. 4 is a schematic diagram showing a decoding apparatus 100B restoring a decoded video frame.

[0097] Specifically, Figure 5A The decoding device 100B shown is Figure 2BThe difference between the decoding device 100A shown is that a cache unit 105 is added to the decoding device 100B. In addition, in the embodiment of the present application, the similarities between the decoding device 100B and the decoding device 100A are not described in detail.

[0098] The cache unit 105 is used to store the recovery information of the interrupt decoding of each core 101, such as the recovery information of entropy decoding and the recovery information of post decoding. In addition, the cache unit 105 can be divided into different storage areas for different cores 101. For example, the core 101a and the core 101b store corresponding recovery information in different storage areas respectively.

[0099] Specific, combined Figure 5A The scene shown, Figure 5B The code stream processing flow shown includes:

[0100] S501-S510, wherein these steps are Figure 2C S101-S110 are the same as those in FIG. 1 and are not described in detail here.

[0101] S511: When bit 1 in data stream A1 is entropy decoded and macroblock 1 is post-decoded, kernel 101a is used to detect that a kernel hang has occurred, and the position information of bit 1, the entropy context corresponding to bit 1, the position information of macroblock 1, and the status information corresponding to macroblock 1 are stored in cache unit 105.

[0102] For example, the state information corresponding to macroblock 1 may be the state information of macroblock row S1, and the entropy context corresponding to bit 1 may be the entropy context of data stream A1.

[0103] S512: When bit 2 in data stream A2 is entropy decoded and macroblock 2 is post-decoded, kernel 101b is used to detect that a kernel hang has occurred, and the position information of bit 2, the entropy context corresponding to bit 2, the position information of macroblock 2, and the status information corresponding to macroblock 2 are stored in cache unit 105.

[0104] For example, the state information corresponding to macroblock 2 may be the state information of macroblock row S2, and the entropy context corresponding to bit 2 may be the entropy context of data stream A2.

[0105] At this time, DDR104 has stored the decoding information of the macroblocks corresponding to the bits before bit 1 in data stream A1, the decoding information of the macroblocks corresponding to the bits before bit 2 in data stream A2, the decoded pixels of the previous macroblock in macroblock row S1 where macroblock 1 is located, and the decoded pixels of the previous macroblock in macroblock row S2 where macroblock 2 is located.

[0106] S513: When the decoding device 100B is restarted, the kernel 101a is used to obtain the position information of bit 1, the entropy context corresponding to bit 1, the position information of macroblock 1 and the status information corresponding to macroblock 2 from the cache unit 105.

[0107] S514 : Use PE 01 in the core 101 a to continue entropy decoding the data stream A1 starting from bit 1 according to the position information of bit 1 and the entropy context corresponding to bit 1 , and store the generated decoding information B1 of each macroblock in the data stream A1 into the DDR 104 .

[0108] S515: Using DE 01 in the core 101a, obtain the decoding information C1 of the macroblock in the macroblock row S1 from the DDR 104 according to the position information of the macroblock 1.

[0109] S516: Use DE 01 in core 101a to recover and post-decode each macroblock in macroblock row S1 according to the position information of macroblock 1, the status information corresponding to macroblock 1 and the decoding information C1 of the macroblock in macroblock row S1, and store the generated decoded pixels D1 of each macroblock in macroblock row S1 into DDR 104.

[0110] S517: When the decoding device 100B is restarted, the kernel 101b is used to obtain the position information of bit 2, the entropy context corresponding to bit 2, the position information of macroblock 2 and the status information corresponding to macroblock 2 from the cache unit 105.

[0111] S518: Use PE02 in the core 101b to continue entropy decoding of the data stream A1 starting from bit 1 according to the position information of bit 2 and the entropy context corresponding to bit 2, and store the generated decoding information B1 of each macroblock in the data stream A1 into DDR104.

[0112] S519: Use DE02 in the core 101b to obtain the decoding information C2 of the macroblock in the macroblock row S2 from DDR104 according to the position information of the macroblock 2.

[0113] S520: Use DE02 in core 101b to recover and post-decode each macroblock in macroblock row S2 according to the position information of macroblock 2, the status information corresponding to macroblock 2 and the decoding information C2 of the macroblocks in macroblock row S2, and store the decoded pixels D2 of each macroblock in the generated macroblock row S2 to DDR 104.

[0114] It is understandable that the above Figure 5A The order of the steps in the embodiment is only an example and does not limit the execution order of the bitstream processing. The steps may also be in other orders. For example, S511 and S512 may be executed in parallel, S513 and S517 may be executed in parallel, and so on.

[0115] Combination Figure 3 , Figure 5A and Figure 5B In the example of, when the kernel of the decoding device 100B is suspended due to high RTL complexity, the kernel 101a stores the recovery information of bit 1 and the recovery information of macroblock 1 to the DDR 104, and the kernel 101b stores the recovery information of bit 2 and the recovery information of macroblock 2 to the DDR 104. For example, the recovery information of macroblock 1 includes the following information: the row and column number of macroblock 1 in the 4th frame image is such as the 2nd row and the 5th column, indicating that macroblock 1 is the 5th macroblock in the 2nd macroblock row in the 4th frame image. The frame number of the video frame where macroblock 1 is located is such as 4, the corresponding encoding technology is the FGS technology, and the corresponding filtering control parameters include α and β.

[0116] Then, when the decoding device 100B is restarted, the recovery information of bit 1, the recovery information of bit 2, the recovery information of macroblock 1 and the recovery information of macroblock 2 can be obtained from the cache unit 105. Figure 3 and Figure 6 For example, when the decoding device 100B resumes decoding, DE01 in the core 101a will resume decoding from macroblock 1 in macroblock row S1 in the 4th frame image according to the recovery information of macroblock 1, and DE02 in the core 101b will resume decoding from macroblock 2 in macroblock row S2 in the 4th frame image according to the recovery information of macroblock 2. Moreover, PE01 in the core 101a will resume entropy decoding for the 6th frame image from bit 1 according to the recovery information of bit 1, and PE02 in the core 101b will resume entropy decoding for the 7th frame image from bit 1 according to the recovery information of bit 2. Then, after completing post-decoding of the 4th frame image, DE01 and DE02 will post-decode the next frame (i.e., the 5th frame). Moreover, after completing entropy decoding of the 6th and 7th frames image, PE01 may continue to perform entropy decoding on the 8th frame image. After completing decoding of the 4th frame image, DE01 and DE02 may continue to perform entropy decoding on the 5th frame image. At this time, the decoding device 100B will not delete the 4th to 7th frame images, which can avoid the decoded video from lacking video frames, and is beneficial to improving the smoothness of the decoded video.

[0117] Understandably, Figure 5A The structure of the decoding device 100B shown is only an example, and the actual decoding device may also include more or fewer components. Figure 5A Not shown, in practice, the decoding device 100B may also include other units such as a static random access memory (SRAM), which is not limited in this embodiment of the present application.

[0118] As an example, the SRAM memory in the decoding device 100B can store a flag for indicating whether the core is suspended, such as a flag value of 1 indicates that the core is suspended, and a flag value of 0 indicates that the core is not suspended (i.e., the core is normal). Specifically, when a core in the decoding device 100B has an error and hangs, the core can set the above flag in the SRAM memory to 1. Moreover, when the flag in the SRAM memory is 0, the decoding device 100B can broadcast to each core so that each core detects that the core is suspended.

[0119] In some embodiments, the number of cores in the decoding device 100B is not limited to the above example, and may include more or fewer cores, such as only one core, which is not specifically limited in the present embodiment. In addition, each core 101 may also include firmware ( Figure 5A The firmware in the kernel 101 can control the input and output operations of the PE and DE therein, and detect whether the kernel is suspended.

[0120] In some embodiments, DDR 104 may store the bitstream, decoded information, and decoded pixels in different storage units ( Figure 5A (not shown). For the convenience of distinction, the storage unit storing the decoded information in the DDR 104 may be referred to as a second storage unit, and the storage unit storing the decoded pixels may be referred to as a third storage unit.

[0121] In addition, the kernel hang of the decoding device 100 may be caused by an error in a macroblock located at a previous position. For example, an error occurs when DE01 in the kernel 101a of the decoding device 100B processes macroblock 1, causing the kernel 101a to hang.

[0122] In some embodiments, when the decoding device is restarted and decoding is resumed, the macroblock row where the error occurred during suspension can be repaired to prevent the error from occurring again when the decoding of the macroblock row is resumed. For example, the core 101a repairs the error of the macroblock row S1 where the macroblock 1 is located. In addition, when the decoding device is restarted and decoding is resumed, normal decoding can be resumed for other macroblock rows where decoding was normally interrupted.

[0123] In some embodiments, since the sizes of macroblocks in a macroblock row may be different, if an error occurs in a macroblock, the decoding device may not be able to determine the size of the macroblock. Therefore, the error is repaired for the macroblock in which the error occurs, specifically, the error is repaired for all data starting from the macroblock in the current macroblock row. For example, DE01 in the core 101a of the decoding device 100B can perform error repair on all data starting from macroblock 1 in the macroblock row S1.

[0124] Furthermore, in another embodiment, DE01 in the core 101a of the decoding device 100B may perform error repair on macroblock 1, and perform normal post-decoding on other macroblocks except macroblock 1 in the macroblock row S1.

[0125] For example, when an error occurs in macroblock 1 and causes the kernel 101a to hang, when the decoding device restarts and resumes decoding, it first obtains the state information and position information in the recovery information of macroblock 1, and obtains the decoding information of each macroblock in the current macroblock row S1 according to the position information, and then post-decodes the macroblock according to the state information and the decoding information of the macroblock, until the macroblock 1 indicated by the position information. Subsequently, the data segments starting from macroblock 1 in the macroblock row S1 are all error-repaired, such as setting the decoded pixels of the data segment to preset pixels, or setting the decoded pixels of the data segment to the pixels of the data segment at the same position in the adjacent image frame. At this time, in the process of error repairing the data segment starting from macroblock 1 in the macroblock row S1 of the present application, there is no need to obtain and use the decoding information of each macroblock in the data segment. In this way, it can be ensured that the decoding process of the macroblock row S1 after recovery does not cause errors, such as not causing the kernel to hang.

[0126] Next, taking a decoding device with a single core as an example, the decoding device using a single core to perform code stream processing based on a newly added storage unit is described as an example. Figure 5A The decoding device 100B is shown and the code stream processing flow is described.

[0127] Reference Figure 7 , shows a schematic diagram of a flow chart of a code stream processing method provided by an embodiment of the present application, and the execution subject of the flow can be Figure 5A In the illustrated decoding device 100B, the core 100a and the cache unit 105 and other units, the process includes the following steps:

[0128] S701: Obtain a data stream in a video bitstream through a first kernel.

[0129] For example, combined with Figure 5A As shown in the code stream processing architecture diagram, the decoding device 100B can obtain the video code stream from the DDR 104 through the distributor 102, and divide the code stream into multiple data streams according to the image unit level, such as dividing it into multiple data streams according to slices or macroblock rows. In addition, the decoding device 100B can schedule the first kernel to obtain the data stream in the code stream through the kernel scheduler 103.

[0130] In some embodiments, the decoding device may be a single-core decoding device or a multi-core decoding device. The first core mentioned above is a core in the decoding device. This embodiment mainly describes the decoding process of one core.

[0131] In some embodiments, the present application performs entropy decoding on each bit stream in the code stream in parallel through multiple cores in the decoding device, and performs post-decoding on each macroblock in the code stream in parallel through multiple cores. As an example, in this embodiment, the entropy decoding of the code stream and the post-decoding of the code stream are performed asynchronously or synchronously.

[0132] S702: Perform entropy decoding on the corresponding data stream in the code stream according to the entropy context of each data stream through the PE in the first core, generate decoding information of each macroblock in each data stream, and store the generated decoding information of each macroblock in the DDR.

[0133] For example, the first kernel is Figure 5A In the illustrated kernel 101a, the PE in the first kernel is PE01.

[0134] S703: Post-decoding each macroblock to generate decoded pixels of each macroblock according to the state information and decoding information corresponding to each macroblock through the DE in the first kernel, and storing the generated decoded pixels of each macroblock into the DDR.

[0135] For example, the first kernel is Figure 5A In the illustrated kernel 101a, the DE in the first kernel is DE01.

[0136] S704: When entropy decoding is performed on the first bit in the first data stream and the first macroblock in the first macroblock row is post-decoded, it is detected that a core hang occurs in the decoding device, and first recovery information of the first bit and second recovery information of the first macroblock are stored in the first storage unit.

[0137] The first recovery information of the first bit includes the entropy context corresponding to the first bit and the position information of the first bit (i.e., the bit position of the first bit). At this time, the entropy context corresponding to the first bit is the entropy context of the first data stream where the first bit is located. The position information of the first bit is the position of the first bit in the first data stream (i.e., the bit position).

[0138] The second recovery information of the first macroblock includes state information such as a frame number corresponding to the first macroblock and position information of the first macroblock (i.e., the macroblock position of the first macroblock). At this time, the state information corresponding to the first macroblock is the state information of the first macroblock row, and the position information of the first macroblock can indicate the position of the first macroblock in the first macroblock row.

[0139] In some embodiments, the first core decodes using asynchronous decoding, and the processing speed of the PE is higher than the processing speed of the DE. Figure 3 and Figure 5AIn the example, when the first kernel is kernel 101a, the first data stream is data stream A1 in the 6th frame image in the code stream, the first bit is a bit in the data stream A1, the first macroblock is macroblock 1 in the 4th frame image, and the first storage unit is cache unit 105.

[0140] In some embodiments, if a hardware failure causes the first core of the decoding device to hang, the first core may store an identifier indicating the core hang in the decoding device, and then broadcast the core hang information to each core in the decoding device, so that each core detects that the core of the decoding device has hung.

[0141] S705: When it is detected that the decoding device is restarted, the first recovery information of the first bit and the second recovery information of the first macroblock are obtained from the first storage unit through the first core.

[0142] Combination Figure 3 , Figure 5A and Figure 6 For example, when the first kernel is kernel 101a, the first bit may be bit 1 in the sixth frame image. For example, the recovery information of bit 1 includes the following information: the bit position of bit 1 in the bitstream, specifically the bit position of bit 1 in the sixth frame image; and the entropy context corresponding to bit 1.

[0143] S706: Determine, through the PE in the first kernel, a first bit in the bitstream according to the bit position in the first recovery information, and continue to perform entropy decoding on the bitstream starting from the first bit according to the entropy context in the first recovery information.

[0144] It can be understood that the position information of the first bit in the first recovery information can accurately continue to entropy decode the code stream from the first bit, without the need to re-entropy decode the data before the first bit in the video frame where the first bit is located, thereby improving the rapidity of recovery decoding. For example, when the decoding device 100B is restarted, bit 1 can be identified and located from the 6th frame image according to the bit position of bit 1, and the entropy decoding of the 6th frame image can be restored from bit 1 according to the corresponding entropy context. At this time, there is no need to repeat entropy decoding of other bits before bit 1 in the 6th frame image.

[0145] S707: The DE in the first kernel starts to continue post-decoding the code stream for the first macroblock row where the first macroblock is located according to the second recovery information.

[0146] In some embodiments, when the decoding device 100B is restarted, the macroblock 1 in the macroblock row S1 can be identified and located from the fourth frame image according to the position information in the recovery information of the macroblock 1, and the decoding information corresponding to the macroblock 1 is obtained from the DDR 104 through the core 101a, and the post-decoding of the fourth frame image is resumed from the macroblock row S1 according to the decoding information and the status information in the recovery information. At this time, the status information of the first macroblock can be the status information of the first macroblock row.

[0147] It can be understood that the position information of the first macroblock in the second recovery information can accurately start the post-decoding of the code stream from the macroblock row where the first macroblock is located, without re-decoding other macroblock rows before the current macroblock row in the video frame where the first macroblock is located. In this way, the speed of recovery decoding is improved, computing resources are saved, and the occupied system bandwidth is reduced. Figure 3 , Figure 5A and Figure 6 In the example, when the decoding device 100B resumes decoding, DE01 in the core 101a will locate macroblock 1 according to the recovery information of macroblock 1, and resume decoding from macroblock 1 in macroblock row S1 in the 4th frame image according to the status information corresponding to macroblock 1.

[0148] S708: Store the decoding information of each macroblock in the first data stream in the second storage unit in accordance with the decoding information of the first data stream where the first bit generated by entropy decoding is located.

[0149] The decoding information of the macroblock includes at least one of the following: a syntax element of the macroblock and auxiliary information of the macroblock.

[0150] S709: When the post-decoding of the first macroblock is completed and the pixel data of the first macroblock is generated, the pixel data of the first macroblock is stored in the third storage unit.

[0151] For example, the second storage unit and the third storage unit are arranged in a DDR in the decoding device.

[0152] Then, when the present application restores the entropy decoding of the first data stream where the first bit is located and restores the post-decoding of the first macroblock row where the first macroblock is located, the decoding information obtained by entropy decoding and the decoded pixels (i.e., pixel data) obtained by post-decoding can be stored normally to ensure that subsequent data in the entire code stream can be decoded normally.

[0153] In this way, the decoding device in the present application can store the recovery information of the bits where entropy decoding is interrupted and the recovery information of the macroblocks where post-decoding is interrupted through the newly added first storage unit when the decoding process is interrupted due to kernel hang. In this way, when the decoding device is restarted, entropy decoding can be restored through the recovery information of the bits in the first storage unit, and post-decoding can be restored through the recovery information of the macroblocks in the first storage unit. In this way, the present application can restore entropy decoding for video frames where entropy decoding is interrupted, and restore post-decoding for video frames where post-decoding is interrupted, avoiding the situation where the entire video frame is deleted or a large number of image pixels are deleted when resuming decoding, which is conducive to ensuring the smoothness of the decoded video.

[0154] In some embodiments, the hardware failure of the decoding device may be caused by an error in post-decoding of the first macroblock. Then, in order to avoid post-decoding errors when resuming decoding, post-decoding errors of the macroblock row where the interrupted macroblock is located may be repaired.

[0155] Reference Figure 8 As shown above Figure 7 The illustrated S707 may include the following steps:

[0156] S707A: According to the macroblock position in the second recovery information, the decoding information of each third macroblock before the first macroblock in the first macroblock row is obtained from the second storage unit, and each third macroblock is post-decoded according to the information in the second recovery information except the macroblock position and the corresponding decoding information.

[0157] It can be understood that the second storage unit, such as DDR, has stored the decoding information of each third macroblock before the first macroblock in the first macroblock row. Furthermore, after post-decoding each third macroblock, the decoded pixels of each third macroblock can be stored in the second storage unit.

[0158] S707B: According to the macroblock position in the second recovery information, post-decode the first macroblock and each fourth macroblock after the first macroblock in the first macroblock row in an error repair manner.

[0159] In some embodiments, the present application may not be able to distinguish the macroblocks after the first macroblock in the first macroblock row, so the first macroblock in the first macroblock row and the data after the first macroblock (ie, the fourth macroblock) may both be error corrected.

[0160] In some embodiments, the process of post-decoding the macroblock in an error repair manner may be to use the preset pixel data as the pixel data of the macroblock. For example, the present application may set the decoded pixels of the first macroblock and each fourth macroblock as the preset pixels, and store the decoded pixels of these macroblocks in the second storage unit.

[0161] In some embodiments, the above process of post-decoding a macroblock in an error repair manner may be: pixel data of a macroblock in the previous video frame adjacent to the video frame where the macroblock is located and having the same position as the macroblock is used as pixel data of the first macroblock. For example, when the first macroblock row belongs to the fourth frame image in the bitstream, the present application may set the decoded pixels of the first macroblock and each fourth macroblock to the decoded pixels of each macroblock in the same position in the third frame image in the bitstream, and store the decoded pixels of these macroblocks in the second storage unit.

[0162] In this way, interruption of post-decoding of the first macroblock and subsequent macroblocks by the first macroblock row can be avoided, thereby ensuring normal decoding of the code stream.

[0163] Reference Fig. 9 As shown above Figure 7 The illustrated S707 may include the following steps:

[0164] S707C: According to the macroblock position in the second recovery information, the decoding information of each third macroblock before the first macroblock in the first macroblock row is obtained from the second storage unit, and each third macroblock is post-decoded according to the information in the second recovery information except the macroblock position and the corresponding decoding information.

[0165] S707D: Post-decode the first macroblock in the first macroblock row according to the macroblock position in the second recovery information in an error repair manner.

[0166] In some embodiments, the present application can pre-divide each macroblock in the first macroblock row, such as dividing each fifth macroblock after the first macroblock in the first macroblock row, so that the error of the first macroblock row in the first macroblock row can be repaired.

[0167] For example, the present application may set the decoded pixels of the first macroblock as preset pixels, and store the decoded pixels in the second storage unit. For another example, when the first macroblock row belongs to the fourth frame image in the code stream, the present application may set the decoded pixels of the first macroblock as the decoded pixels of the macroblock at the same position in the third frame image in the code stream, and store the decoded pixels of the macroblock in the second storage unit.

[0168] S707E: According to the macroblock position in the second recovery information, obtain the decoding information of each fifth macroblock after the first macroblock in the first macroblock row from the second storage unit, and post-decode each fifth macroblock according to the information in the second recovery information except the macroblock position and the corresponding decoding information.

[0169] It can be understood that the second storage unit, such as DDR, has stored the decoding information of the first macroblock and each fifth macroblock in the first macroblock row. Furthermore, after post-decoding each fifth macroblock, the decoded pixels of each fifth macroblock can be stored in the second storage unit.

[0170] In this way, the first macroblock row can avoid interruption of post-decoding of the first macroblock and subsequent macroblocks, ensuring normal decoding of the code stream and ensuring that all data in the code stream can be decoded normally to ensure the smoothness of the decoded video.

[0171] Next, the hardware structure of the electronic device that executes the code stream processing method in the present application is described.

[0172] Fig.10 FIG. 1 is a block diagram of a first electronic device 10 according to an embodiment of the present application. Fig.10 An example first electronic device 10 according to various embodiments is schematically shown. In one embodiment, the first electronic device 10 may include one or more processors 1604, a system control logic 1608 connected to at least one of the processors 1604, a system memory 1612 connected to the system control logic 1608, a non-volatile memory (NVM) 1616 connected to the system control logic 1608, and a network interface 1620 connected to the system control logic 1608.

[0173] In some embodiments, the processor 1604 may include one or more single-core or multi-core processors. In some embodiments, the processor 1604 may include any combination of a general-purpose processor and a dedicated processor (e.g., a graphics processor, an application processor, a baseband processor, etc.). In an embodiment where the first electronic device 10 uses an enhanced base station (evolved node B, eNB) or a radio access network (radio access network, RAN) controller, the processor 1604 may be configured to execute various embodiments that comply with the present invention, for example, Figures 7 to 9 The embodiment shown.

[0174] In some embodiments, system control logic 1608 may include any suitable interface controller to provide any suitable interface to at least one of processors 1604 and / or any suitable device or component in communication with system control logic 1608 .

[0175] In some embodiments, the system control logic 1608 may include one or more memory controllers to provide an interface to the system memory 1612. The system memory 1612 may be used to load and store data and / or instructions. In some embodiments, the memory 1612 of the system 1600 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0176] NVM 1616 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM 1616 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a hard disk drive (HDD), a compact disc (CD) drive, and a digital versatile disc (DVD) drive.

[0177] The NVM 1616 may include a portion of storage resources on the device on which the first electronic device 10 is installed, or it may be accessible by the device but is not necessarily a portion of the device. For example, the NVM 1616 may be accessed over a network via the network interface 1620 .

[0178] In particular, the system memory 1612 and the NVM 1616 may include a temporary copy and a permanent copy of the instruction 1624, respectively. The instruction 1624 may include: when executed by at least one of the processors 1604, causing the first electronic device 10 to implement the following Figures 7 to 9 In some embodiments, instructions 1624, hardware, firmware, and / or software components thereof may additionally / alternatively be placed in system control logic 1608, network interface 1620, and / or processor 1604.

[0179] The network interface 1620 may include a transceiver for providing a radio interface for the first electronic device 10, thereby communicating with any other suitable device (such as a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 1620 may be integrated with other components of the first electronic device 10. For example, the network interface 1620 may be integrated with at least one of the processor 1604, the system memory 1612, the NVM 1616, and a firmware device (not shown) having instructions. When at least one of the processors 1604 executes the instructions, the first electronic device 10 implements the following. Figures 7 to 9 The method shown.

[0180] The network interface 1620 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 1620 may be a network adapter, a wireless network adapter, a telephone modem and / or a wireless modem.

[0181] In one embodiment, at least one of the processors 1604 may be packaged together with logic for one or more controllers of the system control logic 1608 to form a system in package (SiP). In one embodiment, at least one of the processors 1604 may be integrated on the same die with logic for one or more controllers of the system control logic 1608 to form a system on a chip (SoC).

[0182] The first electronic device 10 may further include an input / output (I / O) device 1632 .

[0183] According to an embodiment of the present application, Fig.11 1 shows a block diagram of a system on chip (SoC) 1700. The SoC 1700 is provided in the first electronic device 10. Fig.11 In the drawings, similar parts have the same reference numerals. Figure 8 In the embodiment, the SoC 1700 includes: an interconnect unit 1750, which is coupled to the application processor 1710; a system agent unit 1770; a bus controller unit 1780; an integrated memory controller unit 1740; a video processor 1720; a static random access memory (SRAM) unit 1730; and a direct memory access (DMA) unit 1760. In one embodiment, the SoC 1700 may also include, for example, a network or communication processor, a compression engine, a GPU, a high throughput MIC processor, or an embedded processor. For example, the above-mentioned decoding device 100B may be set in the video processor 1720.

[0184] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0185] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0186] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0187] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0188] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required, but rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0189] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0190] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0191] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A code stream processing method, characterized in that: Applied to a decoding device including a first storage unit, the method includes: Performing entropy decoding and post-decoding on a bitstream, wherein the bitstream includes a first data stream and a first macroblock row; In the process of entropy decoding the first bit of the first data stream and post-decoding the first macroblock in the first macroblock row, if a hardware failure of the decoding device is detected, first recovery information and second recovery information are stored in the first storage unit, wherein the first recovery information includes position information of the first bit and an entropy context corresponding to the first bit, and the second recovery information is used to indicate position information of the first macroblock and status information corresponding to the first macroblock; When it is detected that the decoding device has completed restarting, the first recovery information and the second recovery information are obtained from the first storage unit, and the first data stream where the first bit in the code stream is located is continued to be entropy decoded according to the first recovery information, and the first macroblock row where the first macroblock in the code stream is located is continued to be post-decoded according to the second recovery information.

2. The method according to claim 1, characterized in that The continuing to perform entropy decoding on the first data stream where the first bit in the code stream is located according to the first recovery information includes: The first bit in the code stream is determined according to the position information in the first recovery information, and entropy decoding is continued on the first data stream where the first bit in the code stream is located according to the entropy context in the first recovery information.

3. The method according to claim 2, characterized in that The decoding device further includes: a second storage unit, the second storage unit is used to store decoding information of each macroblock in the code stream, and The method further comprises: storing the decoding information of each macroblock in the first data stream in the second storage unit in response to the decoding information of each macroblock in the first data stream generated by entropy decoding; The decoding information of the macroblock includes at least one of the following: syntax elements of the macroblock and auxiliary information of the macroblock.

4. The method according to claim 3, characterized in that: The status information of the first macroblock includes: a frame number of a video frame to which the first macroblock belongs, a coding technology corresponding to the first macroblock, and a filtering control parameter corresponding to the first macroblock.

5. The method according to claim 4, characterized in that The second restoration information is set at a frame level, a slice level, a block level or a stripe level.

6. The method according to claim 5, characterized in that The step of continuing to post-decode the code stream starting from the first macroblock row where the first macroblock is located according to the second recovery information includes: According to the position information in the second restored information, the decoding information of each third macroblock before the first macroblock in the first macroblock row is obtained from the second storage unit, and the corresponding third macroblock is post-decoded according to the decoding information of each third macroblock and the state information in the second restored information; The first macroblock and a fourth macroblock after the first macroblock in the first macroblock row are post-decoded according to the position information in the second recovery information in an error repair manner.

7. The method according to claim 5, characterized in that The step of continuing to perform post-decoding processing on the code stream starting from the macroblock row where the first macroblock is located according to the second recovery information includes: According to the position information in the second restored information, the decoding information of each third macroblock before the first macroblock in the first macroblock row is obtained from the second storage unit, and the corresponding third macroblock is post-decoded according to the decoding information of each third macroblock and the state information in the second restored information; post-decoding the first macroblock in an error repair manner according to the position information in the second recovery information; According to the position information in the second recovery information, the decoding information of each fifth macroblock after the first macroblock in the first macroblock row is obtained from the second storage unit, and the corresponding fifth macroblock is post-decoded according to the decoding information of each fifth macroblock and the status information in the second recovery information.

8. The method according to claim 6 or 7, characterized in that: The process of post-decoding a macroblock in an error-correcting manner includes: Using the preset pixel data as the pixel data for the macroblock, or, The pixel data of a macroblock having the same position as the macroblock in a previous video frame adjacent to the video frame where the macroblock is located is used as the pixel data of the first macroblock.

9. The method according to claim 8, characterized in that The decoding device further includes: a third storage unit, the third storage unit is used to store pixel data of each macroblock in the code stream, and the method further includes: When the post-decoding of the first macroblock is completed and the pixel data of the first macroblock is generated, the pixel data of the first macroblock is stored in the third storage unit.

10. The method according to claim 9, characterized in that The entropy decoding of the code stream and the post-decoding of the code stream are performed asynchronously or synchronously.

11. The method according to claim 10, characterized in that The hardware failure of the decoding device includes: a failure caused by an error in post-decoding of the first macroblock.

12. The method according to claim 10 or 11, characterized in that: The decoding device comprises a plurality of cores, and The entropy decoding and post-decoding of the code stream includes: The plurality of cores are used to perform entropy decoding on each bit stream in the code stream in parallel, and the plurality of cores are used to perform post-decoding on each macroblock in the code stream in parallel.

13. The method according to claim 12, characterized in that The first storage unit includes a plurality of sub-storage units, each of which corresponds to one of the plurality of cores.

14. The method according to claim 13, characterized in that The second storage unit and the third storage unit are arranged in a double data rate synchronous dynamic random access memory DDR in the decoding device.

15. A decoding device, characterized in that: The decoding device is used to execute the code stream processing method according to any one of claims 1 to 14.

16. A chip, characterized in that: The invention comprises a circuit, wherein the circuit is used to execute the code stream processing method according to any one of claims 1 to 14.

17. An electronic device, characterized in that: The electronic device comprises the chip as claimed in claim 16.

Images