Image block recovery method, device, computer equipment, and storage medium

Abstract

The present application relates to an image block recovery method, device, computer equipment, and storage medium. The method includes: in the process of video decoding, obtaining a target image block with decoding errors, and obtaining corresponding information of the target image block; determining the target component type based on the component information, and determining the target prediction type based on the prediction type information; obtaining a target image block recovery network corresponding to the target image block according to the target component type and the target prediction type; obtaining surrounding image blocks corresponding to the target image block, and using the target image block and the surrounding image blocks as input image blocks; inputting the input image block into the target image block recovery network to obtain an image block recovery result of the target image block. The method can automatically call the image block recovery network for different error conditions when video decoding errors occur, and recover the image blocks with decoding errors to ensure normal video playback.

Description

Technical Field

[0001] The present application relates to the field of video decoding technology, and in particular to an image block recovery method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] Videos are having an increasing impact on people's daily lives. With the advent of the short video era in recent years, the proportion of video data may have far exceeded expectations. At the same time, as video data is transmitted at the network layer, it may be affected by network fluctuations or noise interference, resulting in packet loss or data errors. These errors may affect subsequent image decoding or even make decoding impossible, which poses a great challenge to the video decoding end.

[0003] At present, the mainstream video coding and decoding standards mainly use the traditional hybrid coding framework. The encoded bit stream output under this coding framework is the code stream, and the decoding process of the code stream is basically similar to the encoding process. The syntax elements in the code stream will be entropy encoded according to the standard. Basically, each syntax element has a certain practical meaning, so it has a value range, which makes it convenient to detect errors in the current syntax element during the decoding process. At the same time, the inter-frame prediction mode under this coding framework will pass on the information of the current error block, resulting in a decrease in the overall decoding quality. Therefore, error handling in the video decoding process is very necessary, and the quality of the handling method directly affects the overall decoding quality.

[0004] During the video decoding process, after a decoding error is detected, the traditional error handling method is simple and does not provide a good user experience. For example, some of the current mainstream video mobile applications will display green blocks and obvious freezes when encountering a bitstream error, and will not decode and display normally until the next immediate refresh frame (IDR frame). Although this method can quickly handle the error area / block so that the video playback will not be obviously freezed, it also has obvious shortcomings, that is, the boundary of the error area / block and the surrounding image are not continuous, there is a serious block effect or even obvious offset.

[0005] The current method of handling video decoding errors has a problem of significantly affecting normal video playback. Summary of the invention

[0006] Based on this, it is necessary to provide an image block recovery method, device, computer equipment, computer-readable storage medium and computer program product that can ensure normal video playback when video decoding errors occur in response to the above technical problems.

[0007] In a first aspect, the present application provides an image block restoration method. The method comprises:

[0008] In the video decoding process, a target image block with a decoding error is obtained, and corresponding information of the target image block is obtained; the corresponding information includes component information and prediction type information;

[0009] Determine a target component type based on the component information, and determine a target prediction type based on the prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction;

[0010] According to the target component type and the target prediction type, a target image block restoration network corresponding to the target image block is obtained;

[0011] Acquire surrounding image blocks corresponding to the target image block, and use the target image block and the surrounding image blocks as input image blocks; the syntax element levels of the surrounding image blocks and the target image block in the video decoding process are the same;

[0012] The input image block is input into the target image block restoration network to obtain the image block restoration result of the target image block.

[0013] In one of the embodiments, during a video decoding process, obtaining a target image block with a decoding error includes:

[0014] In the video decoding process, each syntax element in the video bitstream is parsed in descending order of syntax element levels to obtain syntax element information of each syntax element;

[0015] Acquire a target syntax element requiring error detection according to syntax element information of each syntax element;

[0016] Performing error detection on a target syntax element, and determining an error level of the target syntax element according to a syntax element level of the target syntax element;

[0017] If the error level meets the preset condition, the target image block with decoding error is obtained according to the target syntax element and the syntax element level at which the target syntax element is located.

[0018] In one of the embodiments, obtaining a target syntax element requiring error detection according to syntax element information of each syntax element includes:

[0019] Matching the syntax element information of each syntax element with each preset detection tag;

[0020] If the syntax element information of the syntax element can match the preset detection tag, the syntax element with successful matching is used as the target syntax element that needs to be error detected.

[0021] In one of the embodiments, obtaining a target image block restoration network corresponding to the target image block according to the target component type and the target prediction type includes:

[0022] If the component type is brightness component restoration and the prediction type is intra-frame prediction, obtain the intra-frame prediction brightness component restoration network as the target image block restoration network;

[0023] If the component type is chroma component restoration and the prediction type is intra-frame prediction, obtain the intra-frame prediction chroma component restoration network as the target image block restoration network;

[0024] If the component type is brightness component restoration and the prediction type is inter-frame prediction, obtain the inter-frame prediction brightness component restoration network as the target image block restoration network;

[0025] If the component type is chroma component restoration and the prediction type is inter-frame prediction, the inter-frame prediction chroma component restoration network is obtained as the target image block restoration network.

[0026] In one embodiment, the target image block restoration network is an intra-frame prediction brightness component restoration network, and the input image block is input into the target image block restoration network to obtain an image block restoration result of the target image block, including:

[0027] Input the input image block into the intra-frame prediction brightness component recovery network to obtain the intra-frame prediction brightness component recovery result of the target image block;

[0028] Obtain the chrominance component image block corresponding to the input image block;

[0029] According to the intra-frame prediction luminance component restoration result and the chrominance component image block, an image block restoration result of the target image block is obtained.

[0030] In one embodiment, the target image block restoration network is an intra-frame prediction chrominance component restoration network, and the input image block is input into the target image block restoration network to obtain an image block restoration result of the target image block, including:

[0031] Obtain a brightness component image block corresponding to the input image block;

[0032] Input the input image block and the luminance component image block into the intra-frame prediction chrominance component restoration network to obtain the intra-frame prediction chrominance component restoration result of the target image block;

[0033] An image block restoration result of a target image block is obtained according to the intra-frame predicted chrominance component restoration result and the luminance component image block.

[0034] In one embodiment, the target image block restoration network is an inter-frame prediction brightness component restoration network, and the input image block is input into the target image block restoration network to obtain an image block restoration result of the target image block, including:

[0035] Obtain the brightness image block of the previous frame corresponding to the input image block;

[0036] Input the input image block and the brightness image block of the previous frame into the inter-frame prediction brightness component recovery network to obtain the inter-frame prediction brightness component recovery result of the target image block;

[0037] Obtain the chrominance component image block corresponding to the input image block;

[0038] According to the inter-frame prediction luminance component restoration result and the chrominance component image block, an image block restoration result of the target image block is obtained.

[0039] In one embodiment, the target image block restoration network is an inter-frame prediction chrominance component restoration network, and the input image block is input into the target image block restoration network to obtain an image block restoration result of the target image block, including:

[0040] Obtain a brightness component image block corresponding to the input image block;

[0041] Obtain the chrominance image block of the previous frame corresponding to the input image block;

[0042] Input the input image block, the chroma image block of the previous frame and the luminance component image block into the inter-frame prediction chroma component restoration network to obtain the inter-frame prediction chroma component restoration result of the target image block;

[0043] According to the inter-frame prediction chrominance component restoration result and the luminance component image block, an image block restoration result of the target image block is obtained.

[0044] In a second aspect, the present application further provides an image block restoration device. The device comprises:

[0045] An acquisition module, used to acquire a target image block with a decoding error during video decoding, and acquire corresponding information of the target image block; the corresponding information includes component information and prediction type information;

[0046] A classification module, used to determine a target component type based on component information, and to determine a target prediction type based on prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction;

[0047] A matching module, used for acquiring a target image block restoration network corresponding to the target image block according to the target component type and the target prediction type;

[0048] An input module, used for acquiring surrounding image blocks corresponding to a target image block, and taking the target image block and the surrounding image blocks as input image blocks; the surrounding image blocks and the target image block have the same syntax element level in the video decoding process;

[0049] The restoration module is used to input the input image block into the target image block restoration network to obtain the image block restoration result of the target image block.

[0050] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0051] In the video decoding process, a target image block with a decoding error is obtained, and corresponding information of the target image block is obtained; the corresponding information includes component information and prediction type information;

[0052] Determine a target component type based on the component information, and determine a target prediction type based on the prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction;

[0053] According to the target component type and the target prediction type, a target image block restoration network corresponding to the target image block is obtained;

[0054] Acquire surrounding image blocks corresponding to the target image block, and use the target image block and the surrounding image blocks as input image blocks; the syntax element levels of the surrounding image blocks and the target image block in the video decoding process are the same;

[0055] The input image block is input into the target image block restoration network to obtain the image block restoration result of the target image block.

[0056] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0057] In the video decoding process, a target image block with a decoding error is obtained, and corresponding information of the target image block is obtained; the corresponding information includes component information and prediction type information;

[0058] Determine a target component type based on the component information, and determine a target prediction type based on the prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction;

[0059] According to the target component type and the target prediction type, a target image block restoration network corresponding to the target image block is obtained;

[0060] Acquire surrounding image blocks corresponding to the target image block, and use the target image block and the surrounding image blocks as input image blocks; the syntax element levels of the surrounding image blocks and the target image block in the video decoding process are the same;

[0061] The input image block is input into the target image block restoration network to obtain the image block restoration result of the target image block.

[0062] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0063] In the video decoding process, a target image block with a decoding error is obtained, and corresponding information of the target image block is obtained; the corresponding information includes component information and prediction type information;

[0064] Determine a target component type based on the component information, and determine a target prediction type based on the prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction;

[0065] According to the target component type and the target prediction type, a target image block restoration network corresponding to the target image block is obtained;

[0066] Acquire surrounding image blocks corresponding to the target image block, and use the target image block and the surrounding image blocks as input image blocks; the syntax element levels of the surrounding image blocks and the target image block in the video decoding process are the same;

[0067] The input image block is input into the target image block restoration network to obtain the image block restoration result of the target image block.

[0068] The above-mentioned image block recovery method, device, computer equipment, storage medium and computer program product, in the process of video decoding, first obtains the target image block with decoding error, and obtains the corresponding information of the target image block, wherein the corresponding information includes component information and prediction type information. Then, the target component type is determined based on the component information, and the target prediction type is determined based on the prediction type information, wherein the target component type includes luminance component recovery and chrominance component recovery, and the target prediction type is divided into intra-frame prediction and inter-frame prediction. In this way, a target image block recovery network matching the target image block can be selected according to the target component type and the target prediction type, and different image recovery networks are selected for image blocks of different prediction types and different components, which is conducive to improving the image block recovery quality, thereby improving the overall decoding quality. Obtain surrounding image blocks with the same syntax element level as the target image block in the video decoding process, and input the target image block and the surrounding image blocks as input image blocks into the target image block recovery network, so as to be processed by the selected target image block recovery network matching the corresponding information of the target image block, thereby obtaining the image block recovery result of the target image block. In this way, the present application can detect and process the error information in the code stream in a timely manner when a video decoding error occurs. According to different error situations, the image block recovery network is automatically called to recover the image blocks with decoding errors, thereby avoiding greater impact on subsequent decoding and ensuring normal video playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of an image block recovery method in one embodiment;

[0070] Figure 2 is a schematic diagram of a flow chart of error detection in one embodiment;

[0071] Figure 3 A schematic diagram of a residual network structure in one embodiment;

[0072] Figure 4 A schematic diagram of an image block recovery process of an intra-frame prediction brightness component recovery network in one embodiment;

[0073] Figure 5 A schematic diagram of an image block recovery process of an intra-frame prediction chrominance component recovery network in one embodiment;

[0074] Figure 6 A schematic diagram of an image block recovery process of an inter-frame prediction brightness component recovery network in one embodiment;

[0075] Figure 7 A schematic diagram of an image block restoration process of an inter-frame prediction chrominance component restoration network in one embodiment;

[0076] Figure 8is a structural block diagram of an image block restoration device in one embodiment;

[0077] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0079] In one embodiment, Figure 1 As shown, an image block recovery method is provided. This embodiment uses the method applied to a computer device as an example for illustration. It can be understood that the computer device can specifically be a terminal or a server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices, portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart medical devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps:

[0080] Step 102: during the video decoding process, a target image block with a decoding error is obtained, and corresponding information of the target image block is obtained; the corresponding information includes component information and prediction type information.

[0081] Among them, the component information is used to characterize whether the target image block has a decoding error in the luminance component (also known as Luma block) or a decoding error in the chrominance component (also known as Chroma block); the prediction type information is used to characterize whether the decoding error of the target image block requires intra-frame recovery or inter-frame recovery.

[0082] Optionally, during the video decoding process, the computer device parses the syntax elements (Syntax) in the video bitstream layer by layer in the order of the syntax element level from high to low, obtains the syntax element information of each syntax element, and selects the target syntax element that needs to be error detected; then, error detection is performed on the target syntax element, and the error level of the target syntax element is determined according to the syntax element level of the target syntax element; if the error level belongs to the range that requires image block recovery, the target image block corresponding to the target syntax element is found according to the target syntax element and the syntax element level of the target syntax element, and the corresponding information in the target syntax element is recorded. The syntax element levels from high to low include Frame, Slice, CTU (coding tree unit), CU (coding unit), etc.

[0083] In a feasible implementation, the target image block may have decoding errors in both the luminance component and the chrominance component. In this case, image block recovery will be performed on both the luminance component and the chrominance component of the target image block, with image block recovery performed on the luminance component first and then on the chrominance component.

[0084] Step 104, determining a target component type based on the component information, and determining a target prediction type based on the prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction.

[0085] Among them, the target component type is used to characterize the component (luminance component or chrominance component) that the image block restoration network can process, and the target prediction type is used to characterize the prediction type (intra-frame prediction or inter-frame prediction) when the image block restoration network restores a certain component.

[0086] Optionally, the computer device determines a target component type of the image block restoration network to be used according to component information in the target syntax element, and determines a target prediction type of the image block restoration network to be used according to prediction type information in the target syntax element.

[0087] Step 106: Acquire a target image block restoration network corresponding to the target image block according to the target component type and the target prediction type.

[0088] Optionally, the computer device matches a plurality of pre-trained image block restoration networks according to the determined target component type and target prediction type, and selects an image block restoration network that matches both the target component type and the target prediction type as the target image block restoration network.

[0089] Step 108, obtaining surrounding image blocks corresponding to the target image block, and taking the target image block and the surrounding image blocks together as input image blocks; the syntax element levels of the surrounding image blocks and the target image block in the video decoding process are the same.

[0090] The surrounding image blocks refer to the upper image block, the left image block and the upper-left image block adjacent to the target image block, and each of the surrounding image blocks has the same syntax element level as the target image block during the video decoding process.

[0091] Optionally, the computer device performs input acquisition corresponding to the network each time according to the selected target image block recovery network. Different image block recovery networks require different input data, but all types of image block recovery networks require input of the target image block and the corresponding three surrounding image blocks (adjacent upper image blocks, left image blocks and upper left image blocks). These four image blocks together constitute the input image block. Generally, if the target image block recovery network belongs to the chrominance component recovery network, then it is also necessary to input the normal luminance component image blocks with the same size and position as the input image block into the network; if the target image block recovery network belongs to the inter-frame prediction recovery network, then it is also necessary to input the image blocks with the same size and position in the previous frame of the input image block on the same component into the network; if a target image block has decoding errors in both the luminance component and the chrominance component, the luminance component is first image-blocked, and then the chrominance component is image-blocked.

[0092] Step 110: input the input image block into the target image block restoration network to obtain an image block restoration result of the target image block.

[0093] Optionally, the computer device obtains different network input data according to the type of the matched target image block recovery network to perform image block recovery of the target image block. Specifically, it can be divided into the following four cases:

[0094] In the first case, when the computer device is matched to the intra-frame prediction brightness component recovery network, the input image block is input into the intra-frame prediction brightness component recovery network to obtain the intra-frame prediction brightness component recovery result of the target image block; the chrominance component image block corresponding to the input image block is obtained; and the image block recovery result of the target image block is obtained according to the intra-frame prediction brightness component recovery result and the chrominance component image block.

[0095] In the second case, the computer device is matched to the intra-frame prediction chroma component recovery network to obtain the luminance component image block corresponding to the input image block; the input image block and the luminance component image block are input into the intra-frame prediction chroma component recovery network to obtain the intra-frame prediction chroma component recovery result of the target image block; based on the intra-frame prediction chroma component recovery result and the luminance component image block, the image block recovery result of the target image block is obtained.

[0096] In the third case, the computer device is matched to the inter-frame prediction brightness component recovery network to obtain the brightness image block of the previous frame corresponding to the input image block; the input image block and the brightness image block of the previous frame are input into the inter-frame prediction brightness component recovery network to obtain the inter-frame prediction brightness component recovery result of the target image block; the chrominance component image block corresponding to the input image block is obtained; and the image block recovery result of the target image block is obtained according to the inter-frame prediction brightness component recovery result and the chrominance component image block.

[0097] In a fourth case, the computer device is matched to the inter-frame prediction chroma component recovery network, and the input image block is input into the target image block recovery network to obtain the image block recovery result of the target image block, including: obtaining the luminance component image block corresponding to the input image block; obtaining the previous frame chroma image block corresponding to the input image block; inputting the input image block, the previous frame chroma image block and the luminance component image block into the inter-frame prediction chroma component recovery network to obtain the inter-frame prediction chroma component recovery result of the target image block; obtaining the image block recovery result of the target image block according to the inter-frame prediction chroma component recovery result and the luminance component image block.

[0098] In the above-mentioned image block recovery method, during the video decoding process, the target image block with decoding error is obtained, and the corresponding information of the target image block is obtained; the corresponding information includes component information and prediction type information; the target component type is determined based on the component information, and the target prediction type is determined based on the prediction type information; the target component type includes brightness component recovery and chrominance component recovery, and the target prediction type is divided into intra-frame prediction and inter-frame prediction; according to the target component type and the target prediction type, a target image block recovery network corresponding to the target image block is obtained; the surrounding image blocks corresponding to the target image block are obtained, and the target image block and the surrounding image blocks are used as input image blocks together; the syntax element level of the surrounding image blocks and the target image blocks in the video decoding process is the same; the input image block is input into the target image block recovery network to obtain the image block recovery result of the target image block. When the video decoding error occurs, the error information in the code stream can be detected and processed in time, and the image block recovery network can be automatically called for different error situations to recover the image block with decoding error, so as to avoid a greater impact on subsequent decoding and ensure normal video playback.

[0099] In one embodiment, during a video decoding process, a target image block with a decoding error is obtained, including: during the video decoding process, each syntax element in a video bitstream is parsed in descending order of syntax element levels to obtain syntax element information of each syntax element; based on the syntax element information of each syntax element, a target syntax element that needs to be error detected is obtained; error detection is performed on the target syntax element, and an error level of the target syntax element is determined based on the syntax element level of the target syntax element; if the error level meets a preset condition, based on the target syntax element and the syntax element level at which the target syntax element is located, a target image block with a decoding error is obtained.

[0100] Optional, such as Figure 2As shown, the syntax elements in the code stream are parsed in order, and the outermost (High level) syntax is generally parsed first; when parsing each syntax, it is determined whether error detection is currently required. The error detection function can be enabled by setting in the configuration file of the decoder. If the current Syntax does not require error detection, execute step h, otherwise execute the next step; look up the table to obtain the value range of the current Syntax, and at the same time obtain the error level to determine whether image block recovery is required; determine whether the value of the current Syntax is legal, if the current Syntax is legal, determine whether the decoding of the current Slice is completed, otherwise it means that the Syntax is illegal, and execute the next step; determine whether the current error level is Skip Slice (skip the entire slice), if so, stop decoding the current Slice, and determine whether the decoding of the current Slice is completed, if not, execute the next step; determine whether the current error level is Skip Frame (skip the entire frame), if so, end the decoding of the current Frame, if not, execute the next step; handle other error levels, the specific processing method can be pre-configured; determine whether the decoding of the current Slice is completed, if not, continue decoding the Syntax in the current Slice, otherwise execute the next step; determine whether the decoding of the current Frame is completed, if not, continue decoding the next Slice, otherwise end the decoding of the current Frame. It should be noted that if the process jumps to this step because a bitstream error is detected (the corresponding error level here is Skip Slice), image block recovery processing is still required.

[0101] Specifically, the error levels involved in this embodiment are divided into Warning, Skip Slice, Skip Frame, CTUBack and CTU Forward. The basis for this classification is that different Syntax errors have different effects on the overall decoding. In practical applications, the error level can be increased or decreased according to the specific situation of the Codec.

[0102] Warning: This type of error level applies to those syntaxes that will not affect other processes. For example, the syntax "end_of_slice_segment_flag" in H.265 appears at the end of the current slice. If the previous decoding process does not detect an error and the information of the current CU can be used to determine that the last CU has been decoded, then "end_of_slice_segment_flag" will not affect the previously decoded or later decoded syntaxes, so it does not matter even if it is decoded incorrectly.

[0103] Skip Slice: This type of error level applies to syntaxes that will affect the current slice. For example, the syntax "MVD_X" and "MVD_Y" in H.265 are directly related to the reconstruction effect of the current image block, and if the subsequent frames also refer to this image block, the error will be passed on, causing the overall decoding quality to deteriorate. In this case, the decoding of the current slice needs to be terminated immediately. It is assumed here that the slices are independent of each other. Then some methods need to be used to restore the image information of the current slice. In this embodiment, a neural network can be used to restore the image block.

[0104] Skip Frame: This error level is applicable when decoding the Syntax at the Frame level. If the current Syntax is decoded incorrectly, it will directly affect the overall reconstruction quality of the current frame. For example, the Syntax "log2_sao_offset_scale_luma" in H.265 should have a value range of Max(0,BitDepthY-10). If the decoded result is not within this range due to a bitstream error, it will affect the subsequent SAO filtering process, thereby reducing the decoding quality of the reconstructed image.

[0105] CTU Back: This type of error occurs when decoding the current CTU. The current CTU has been decoded through auxiliary information such as coordinate information or CTU counter. This type of error may be caused by repeated transmission of network packets. In this case, you need to update the relevant register information in the decoder (such as CTU counter) and then decode the CTU again.

[0106] CTU Forward: This type of error is the opposite of the above error. It is due to the error of the bitstream or packet loss that causes the decoding of the current slice to end prematurely. When decoding the next slice, it is necessary to determine whether the last CTU decoded in the previous slice is the lower right corner CTU. If not, it means that there is a CTU missing and image block recovery processing is required. Similar to the processing method of Skip Slice, a neural network can be used for image block recovery in this embodiment.

[0107] When the error level is Skip Slice and CTU Forward, the computer device considers that the error level meets the preset condition, and obtains the target image block with decoding error according to the target syntax element and the syntax element level at which the target syntax element is located.

[0108] In a feasible implementation, based on the syntax element information of each syntax element, a target syntax element that needs to be error detected is obtained, including: matching the syntax element information of each syntax element with each preset detection tag; if the syntax element information of the syntax element can match the preset detection tag, the syntax element with the successful match is used as the target syntax element that needs to be error detected.

[0109] In this embodiment, in the video decoding process, each syntax element in the video bitstream is parsed in the order of syntax element levels from high to low, and syntax element information of each syntax element is obtained; according to the syntax element information of each syntax element, the target syntax element that needs to be error detected is obtained; error detection is performed on the target syntax element, and the error level of the target syntax element is determined according to the syntax element level of the target syntax element; if the error level meets the preset conditions, the target image block with decoding error is obtained according to the target syntax element and the syntax element level where the target syntax element is located. The target syntax element that needs to be error detected can be found first, and then the image block can be restored or other processing can be performed according to the error level of the target syntax element, and the bitstream error can be processed in a hierarchical manner, and the software and hardware resources can be used reasonably to avoid resource waste.

[0110] In one embodiment, according to the target component type and the target prediction type, a target image block recovery network corresponding to the target image block is obtained, including: if the component type is luminance component recovery and the prediction type is intra-frame prediction, an intra-frame prediction luminance component recovery network is obtained as the target image block recovery network; if the component type is chrominance component recovery and the prediction type is intra-frame prediction, an intra-frame prediction chrominance component recovery network is obtained as the target image block recovery network; if the component type is luminance component recovery and the prediction type is inter-frame prediction, an inter-frame prediction luminance component recovery network is obtained as the target image block recovery network; if the component type is chrominance component recovery and the prediction type is inter-frame prediction, an inter-frame prediction chrominance component recovery network is obtained as the target image block recovery network.

[0111] Optionally, the residual network Resnet can be selected as the recovery network for each initial image block. The activation layer of the residual network Resnet uses the relu function; the convolution kernel size is 3*3. Compared with the 5*5 and 7*7 convolution kernels, the 3*3 convolution kernel has fewer parameters for the same receptive field (requiring multiple 3*3 convolution layers), which can prevent overfitting. At the same time, the multi-layer relu activation increases the nonlinearity in the convolution process and increases the network's expressive power. The four neural network structures all have skip connections, that is, the input information is directly added to the output of the second-to-last layer of the entire network to obtain the final output result. The advantage of this is that shallow features can be used to accelerate network learning and improve network performance. The four neural network structures correspond to Luma and Chroma in the intra-frame prediction mode, and Luma and Chroma in the inter-frame prediction mode (the computer device selects at least one to use based on the component information and prediction type information). The network structure of the basic unit Res-block of Res-Net is involved as follows Figure 3 It should be noted that the neural network involved in this embodiment is not limited to this network structure, and can also be deformed on this basis or replaced by other neural networks with the same characteristics.

[0112] In a feasible implementation, the neural network model is trained using the public data set Vimeo-90k, which contains a large number of high-quality videos of different resolutions and various scenes, and is very suitable for video-related neural network training. The network model training steps in this patent are as follows: randomly select several clips from the data set, then encode these clips with HM16.20, and then decode them. In the decoding process, unfiltered reconstructed frames are obtained as sample sets. The I frame in the sample is cut at a random position to obtain a subsample with a size of 192x192, and then the copy of the subsample is denoised to obtain an input sample. The specific method is to set the middle 64x64 area to black. Use the same method to make several input samples to form an intra-frame prediction model sample set for training and testing the intra-frame prediction model. Randomly extract two adjacent frames from the sample, block the two frames to obtain a subsample pair, both with a size of 192x192, and then add noise to the sample copy of the latter frame and form an input sample with the sample of the previous frame. The denoising method is consistent with the above. Use the same method to make several sub-sample pairs to form an inter-frame prediction model sample set for training and testing the inter-frame prediction model. During the training process, Luma and Chroma need to be trained separately. The Adam optimization algorithm is used during training, with an initial learning rate of 1e-4, and a decay coefficient α=0.9 multiplied by the learning rate every 10 epochs. BatchSize is set to 256, and a total of 70 epochs are trained. It should be noted that the above training method is only for illustration, and the actual application is not limited to this training method and data set.

[0113] In this embodiment, if the component type is luminance component restoration and the prediction type is intra-frame prediction, the intra-frame prediction luminance component restoration network is obtained as the target image block restoration network; if the component type is chrominance component restoration and the prediction type is intra-frame prediction, the intra-frame prediction chrominance component restoration network is obtained as the target image block restoration network; if the component type is luminance component restoration and the prediction type is inter-frame prediction, the inter-frame prediction luminance component restoration network is obtained as the target image block restoration network; if the component type is chrominance component restoration and the prediction type is inter-frame prediction, the inter-frame prediction chrominance component restoration network is obtained as the target image block restoration network. It is possible to have specific image restoration methods for image blocks of different prediction types and different components, which is conducive to improving the image block restoration quality, thereby improving the overall decoding quality.

[0114] In one embodiment, the target image block recovery network is an intra-frame prediction luminance component recovery network, and the input image block is input into the target image block recovery network to obtain an image block recovery result of the target image block, including: inputting the input image block into the intra-frame prediction luminance component recovery network to obtain an intra-frame prediction luminance component recovery result of the target image block; obtaining a chrominance component image block corresponding to the input image block; and obtaining an image block recovery result of the target image block according to the intra-frame prediction luminance component recovery result and the chrominance component image block.

[0115] Optional, such as Figure 4 As shown in FIG. 1 , the intra-frame prediction luminance component recovery network is used to process the image recovery of the Luma block in the intra-frame prediction mode. t_error It represents the error block (the black block in the figure) and the surrounding blocks (Top-Left block, Top block and Left block) in the image at time t. No filtering operation is performed in this area. t Indicates P t_error The corresponding correct block information has been reconstructed and unfiltered. The number of residual blocks in Res-Net is 20.

[0116] In this embodiment, there can be specific image restoration methods for image blocks of different prediction types and different components, which is beneficial to improving the image block restoration quality, thereby improving the overall decoding quality.

[0117] In one embodiment, the target image block recovery network is an intra-frame prediction chroma component recovery network, and the input image block is input into the target image block recovery network to obtain an image block recovery result of the target image block, including: obtaining a luminance component image block corresponding to the input image block; inputting the input image block and the luminance component image block into the intra-frame prediction chroma component recovery network to obtain an intra-frame prediction chroma component recovery result of the target image block; and obtaining an image block recovery result of the target image block according to the intra-frame prediction chroma component recovery result and the luminance component image block.

[0118] Optional, such as Figure 5 As shown in Figure 1, this network is used to process the image restoration of the Chroma block in the intra-frame prediction mode. t_error It represents the error block (the black block in the figure) and the surrounding blocks (Top-Left block, Top block and Left block) in the image at time t. No filtering operation is performed in this area. t Indicates P t_error The corresponding correct block information that has been reconstructed and not filtered, P t_luma Indicates P t_error The corresponding unfiltered Luma block information has been reconstructed. This block needs to be downsampled through the max pool layer to get the same size as the Chroma block. The number of residual blocks in Res-Net is 25.

[0119] In this embodiment, there can be specific image restoration methods for image blocks of different prediction types and different components, which is beneficial to improving the image block restoration quality, thereby improving the overall decoding quality.

[0120] In one embodiment, the target image block recovery network is an inter-frame prediction luminance component recovery network, and the input image block is input into the target image block recovery network to obtain an image block recovery result of the target image block, including: obtaining a previous frame luminance image block corresponding to the input image block; inputting the input image block and the previous frame luminance image block into the inter-frame prediction luminance component recovery network to obtain an inter-frame prediction luminance component recovery result of the target image block; obtaining a chrominance component image block corresponding to the input image block; and obtaining an image block recovery result of the target image block according to the inter-frame prediction luminance component recovery result and the chrominance component image block.

[0121] Optional, such as Figure 6 As shown in Figure 1, this network is used to process the image restoration of the Luma block in the inter-frame prediction mode. t_error It represents the error block (the black block in the figure) and the surrounding blocks (Top-Left block, Top block and Left block) in the image at time t. No filtering operation is performed in this area. t Indicates P t_error The corresponding correct block information that has been reconstructed and not filtered, P t_1 Indicates the unfiltered Luma block information of the corresponding position of the forward adjacent reference frame that has been reconstructed. The number of residual blocks in Res-Net is 20.

[0122] In this embodiment, there can be specific image restoration methods for image blocks of different prediction types and different components, which is beneficial to improving the image block restoration quality, thereby improving the overall decoding quality.

[0123] In one embodiment, the target image block recovery network is an inter-frame prediction chroma component recovery network, and the input image block is input into the target image block recovery network to obtain an image block recovery result of the target image block, including: obtaining a luminance component image block corresponding to the input image block; obtaining a previous frame chroma image block corresponding to the input image block; inputting the input image block, the previous frame chroma image block and the luminance component image block into the inter-frame prediction chroma component recovery network to obtain an inter-frame prediction chroma component recovery result of the target image block; and obtaining an image block recovery result of the target image block according to the inter-frame prediction chroma component recovery result and the luminance component image block.

[0124] Optional, such as Figure 7 As shown in Figure 1, this network is used to process the image restoration of the Chroma block in the inter-frame prediction mode. t_errorIt represents the error block (the black block in the figure) and the surrounding blocks (Top-Left block, Top block and Left block) in the image at time t. No filtering operation is performed in this area. t Indicates P t_error The corresponding correct block information that has been reconstructed and not filtered, P t_1 Indicates the reconstructed unfiltered Chroma block information of the corresponding position of the forward adjacent reference frame, P t_luma Indicates the reconstructed unfiltered Luma block information at the corresponding position of the current frame. This block needs to be downsampled through the max pool layer to get the same size as the Chroma block. The number of residual blocks in Res-Net is 25.

[0125] In this embodiment, there can be specific image restoration methods for image blocks of different prediction types and different components, which is beneficial to improving the image block restoration quality, thereby improving the overall decoding quality.

[0126] In one embodiment, a method for restoring an image block includes:

[0127] In the video decoding process, each syntax element in the video bitstream is parsed in the order of syntax element levels from high to low, and the syntax element information of each syntax element is obtained; the syntax element information of each syntax element is matched with each preset detection label; if the syntax element information of the syntax element can match the preset detection label, the syntax element that successfully matches is used as the target syntax element that needs to be error detected; error detection is performed on the target syntax element, and the error level of the target syntax element is determined according to the syntax element level of the target syntax element; if the error level meets the preset conditions, the target image block with decoding error is obtained according to the target syntax element and the syntax element level where the target syntax element is located. And the corresponding information of the target image block is obtained; the corresponding information includes component information and prediction type information.

[0128] The target component type is determined based on the component information, and the target prediction type is determined based on the prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction.

[0129] If the component type is luminance component restoration and the prediction type is intra-frame prediction, the intra-frame prediction luminance component restoration network is obtained as the target image block restoration network; if the component type is chrominance component restoration and the prediction type is intra-frame prediction, the intra-frame prediction chrominance component restoration network is obtained as the target image block restoration network; if the component type is luminance component restoration and the prediction type is inter-frame prediction, the inter-frame prediction luminance component restoration network is obtained as the target image block restoration network; if the component type is chrominance component restoration and the prediction type is inter-frame prediction, the inter-frame prediction chrominance component restoration network is obtained as the target image block restoration network.

[0130] A surrounding image block corresponding to the target image block is obtained, and the target image block and the surrounding image block are used as input image blocks together; the syntax element level of the surrounding image block and the target image block in the video decoding process is the same.

[0131] The target image block restoration network is an intra-frame prediction brightness component restoration network. The input image block is input into the intra-frame prediction brightness component restoration network to obtain the intra-frame prediction brightness component restoration result of the target image block; the chrominance component image block corresponding to the input image block is obtained; and the image block restoration result of the target image block is obtained according to the intra-frame prediction brightness component restoration result and the chrominance component image block.

[0132] The target image block restoration network is an intra-frame prediction chroma component restoration network, which obtains a luminance component image block corresponding to the input image block; the input image block and the luminance component image block are input into the intra-frame prediction chroma component restoration network to obtain an intra-frame prediction chroma component restoration result of the target image block; according to the intra-frame prediction chroma component restoration result and the luminance component image block, an image block restoration result of the target image block is obtained.

[0133] The target image block restoration network is an inter-frame prediction brightness component restoration network, which obtains the brightness image block of the previous frame corresponding to the input image block; the input image block and the brightness image block of the previous frame are input into the inter-frame prediction brightness component restoration network to obtain the inter-frame prediction brightness component restoration result of the target image block; the chrominance component image block corresponding to the input image block is obtained; and the image block restoration result of the target image block is obtained according to the inter-frame prediction brightness component restoration result and the chrominance component image block.

[0134] The target image block restoration network is an inter-frame prediction chroma component restoration network, which obtains a luminance component image block corresponding to an input image block; obtains a previous frame chroma image block corresponding to the input image block; inputs the input image block, the previous frame chroma image block and the luminance component image block into the inter-frame prediction chroma component restoration network to obtain an inter-frame prediction chroma component restoration result of the target image block; obtains an image block restoration result of the target image block based on the inter-frame prediction chroma component restoration result and the luminance component image block.

[0135] In this embodiment, the error information in the code stream can be detected and processed in time to avoid a greater impact on subsequent decoding; the code stream errors can be processed in a hierarchical manner, and software and hardware resources can be used reasonably to avoid resource waste; there are specific image recovery methods for image blocks of different prediction types and different components, which is conducive to improving the image block recovery quality, thereby improving the overall decoding quality; the decoding speed of the decoder is accelerated, and there will be no obvious freezes due to code stream errors, while optimizing the subjective visual effect, which will bring about an improvement in user experience. It can also enhance the fault tolerance of the decoder, improve the overall video decoding quality, and help improve the market image of the product, and is applicable to existing or future video codec standards based on a hybrid coding framework.

[0136] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0137] Based on the same inventive concept, the embodiment of the present application also provides an image block restoration device for implementing the image block restoration method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more image block restoration device embodiments provided below can refer to the limitations of the image block restoration method above, and will not be repeated here.

[0138] In one embodiment, Figure 8 As shown, an image block restoration device 800 is provided, comprising: an acquisition module 801, a classification module 802, a matching module 803, an input module 804 and a restoration module 805, wherein:

[0139] The acquisition module 801 is used to acquire a target image block with a decoding error and acquire corresponding information of the target image block during the video decoding process; the corresponding information includes component information and prediction type information.

[0140] The classification module 802 is used to determine the target component type based on the component information and determine the target prediction type based on the prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction.

[0141] The matching module 803 is used to obtain a target image block restoration network corresponding to the target image block according to the target component type and the target prediction type.

[0142] The input module 804 is used to obtain surrounding image blocks corresponding to the target image block, and use the target image block and the surrounding image blocks as input image blocks; the surrounding image blocks and the target image block have the same syntax element level during the video decoding process.

[0143] The restoration module 805 is used to input the input image block into the target image block restoration network to obtain an image block restoration result of the target image block.

[0144] In one embodiment, the acquisition module 801 is also used to parse each syntax element in the video bitstream in the order of syntax element levels from high to low during the video decoding process, and obtain syntax element information of each syntax element; obtain the target syntax element that needs to be error detected based on the syntax element information of each syntax element; perform error detection on the target syntax element, and determine the error level of the target syntax element based on the syntax element level of the target syntax element; if the error level meets the preset conditions, obtain the target image block with decoding error based on the target syntax element and the syntax element level where the target syntax element is located.

[0145] In one embodiment, the acquisition module 801 is further used to match the syntax element information of each syntax element with each preset detection tag; if the syntax element information of the syntax element can match the preset detection tag, the successfully matched syntax element is used as the target syntax element that needs to be error detected.

[0146] In one embodiment, the matching module 803 is also used to obtain an intra-frame prediction luminance component recovery network as a target image block recovery network if the component type is luminance component recovery and the prediction type is intra-frame prediction; obtain an intra-frame prediction chroma component recovery network as a target image block recovery network if the component type is luminance component recovery and the prediction type is intra-frame prediction; obtain an inter-frame prediction luminance component recovery network as a target image block recovery network if the component type is luminance component recovery and the prediction type is inter-frame prediction; obtain an inter-frame prediction chroma component recovery network as a target image block recovery network if the component type is chroma component recovery and the prediction type is inter-frame prediction.

[0147] In one embodiment, the target image block recovery network is an intra-frame prediction luminance component recovery network, and the recovery module 805 is further used to input the input image block into the intra-frame prediction luminance component recovery network to obtain the intra-frame prediction luminance component recovery result of the target image block; obtain the chrominance component image block corresponding to the input image block; and obtain the image block recovery result of the target image block according to the intra-frame prediction luminance component recovery result and the chrominance component image block.

[0148] In one embodiment, the target image block recovery network is an intra-frame prediction chroma component recovery network, and the recovery module 805 is further used to obtain a luminance component image block corresponding to the input image block; the input image block and the luminance component image block are input into the intra-frame prediction chroma component recovery network to obtain an intra-frame prediction chroma component recovery result of the target image block; and the image block recovery result of the target image block is obtained according to the intra-frame prediction chroma component recovery result and the luminance component image block.

[0149] In one embodiment, the target image block recovery network is an inter-frame prediction luminance component recovery network, and the recovery module 805 is further used to obtain a previous frame luminance image block corresponding to the input image block; input the input image block and the previous frame luminance image block into the inter-frame prediction luminance component recovery network to obtain an inter-frame prediction luminance component recovery result of the target image block; obtain a chrominance component image block corresponding to the input image block; and obtain an image block recovery result of the target image block based on the inter-frame prediction luminance component recovery result and the chrominance component image block.

[0150] In one embodiment, the target image block recovery network is an inter-frame prediction chroma component recovery network, and the recovery module 805 is further used to obtain a luminance component image block corresponding to the input image block; obtain a previous frame chroma image block corresponding to the input image block; input the input image block, the previous frame chroma image block and the luminance component image block into the inter-frame prediction chroma component recovery network to obtain an inter-frame prediction chroma component recovery result of the target image block; obtain an image block recovery result of the target image block based on the inter-frame prediction chroma component recovery result and the luminance component image block.

[0151] Each module in the above-mentioned image block restoration device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0152] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image block recovery network data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image block recovery method is implemented.

[0153] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0154] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0155] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0156] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0158] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image block restoration method, characterized in that: The method comprises: In the video decoding process, a target image block with a decoding error is obtained, and corresponding information of the target image block is obtained; the corresponding information includes component information and prediction type information; In the video decoding process, obtaining a target image block with a decoding error includes: In the video decoding process, each syntax element in the video bitstream is parsed in the order of syntax element levels from high to low, and syntax element information of each syntax element is obtained; the syntax element information includes the syntax element level, and the syntax element level includes Frame, Slice, CTU and CU from high to low; according to the syntax element information of each syntax element, a target syntax element that needs to be error detected is obtained; error detection is performed on the target syntax element, and the error level of the target syntax element is determined according to the syntax element level of the target syntax element; the error level is divided into Warning, Skip Slice, Skip Frame, CTU Back and CTU Forward; if the error level meets the preset condition, the target image block with decoding error is obtained according to the target syntax element and the syntax element level where the target syntax element is located; The step of obtaining a target syntax element requiring error detection according to syntax element information of each syntax element includes: Matching the syntax element information of each syntax element with each preset detection tag; if the syntax element information of the syntax element can match the preset detection tag, the syntax element that successfully matches is used as the target syntax element that needs to be error detected; Determine a target component type based on the component information, and determine a target prediction type based on the prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction; Acquire a target image block restoration network corresponding to the target image block according to the target component type and the target prediction type; Acquire surrounding image blocks corresponding to the target image block, and use the target image block and the surrounding image blocks together as input image blocks; the surrounding image blocks and the target image block have the same syntax element level in the video decoding process; The input image block is input into the target image block restoration network to obtain an image block restoration result of the target image block.

2. The method according to claim 1, characterized in that The step of acquiring a target image block restoration network corresponding to the target image block according to the target component type and the target prediction type comprises: If the component type is the brightness component restoration, and the prediction type is the intra-frame prediction, obtaining an intra-frame prediction brightness component restoration network as the target image block restoration network; If the component type is the chroma component restoration, and the prediction type is the intra-frame prediction, obtaining an intra-frame prediction chroma component restoration network as the target image block restoration network; If the component type is the brightness component restoration, and the prediction type is the inter-frame prediction, obtaining an inter-frame prediction brightness component restoration network as the target image block restoration network; If the component type is the chroma component restoration, and the prediction type is the inter-frame prediction, an inter-frame prediction chroma component restoration network is obtained as the target image block restoration network.

3. The method according to claim 2, characterized in that The target image block restoration network is the intra-frame prediction brightness component restoration network, and the step of inputting the input image block into the target image block restoration network to obtain an image block restoration result of the target image block includes: Inputting the input image block into the intra-frame prediction brightness component recovery network to obtain an intra-frame prediction brightness component recovery result of the target image block; Obtaining a chrominance component image block corresponding to the input image block; An image block restoration result of the target image block is obtained according to the intra-frame prediction luminance component restoration result and the chrominance component image block.

4. The method according to claim 2, characterized in that: The target image block restoration network is the intra-frame prediction chrominance component restoration network, and the step of inputting the input image block into the target image block restoration network to obtain an image block restoration result of the target image block includes: Obtaining a brightness component image block corresponding to the input image block; Inputting the input image block and the luminance component image block into the intra-frame prediction chrominance component restoration network to obtain an intra-frame prediction chrominance component restoration result of the target image block; An image block restoration result of the target image block is obtained according to the intra-frame prediction chrominance component restoration result and the luminance component image block.

5. The method according to claim 2, characterized in that: The target image block restoration network is the inter-frame prediction brightness component restoration network, and the inputting the input image block into the target image block restoration network to obtain an image block restoration result of the target image block includes: Obtaining a brightness image block of a previous frame corresponding to the input image block; Inputting the input image block and the previous frame brightness image block into the inter-frame prediction brightness component recovery network to obtain the inter-frame prediction brightness component recovery result of the target image block; Obtaining a chrominance component image block corresponding to the input image block; An image block restoration result of the target image block is obtained according to the inter-frame prediction luminance component restoration result and the chrominance component image block.

6. The method according to claim 2, characterized in that The target image block restoration network is the inter-frame prediction chrominance component restoration network, and the step of inputting the input image block into the target image block restoration network to obtain an image block restoration result of the target image block includes: Obtaining a brightness component image block corresponding to the input image block; Obtaining a chrominance image block of a previous frame corresponding to the input image block; Inputting the input image block, the previous frame chrominance image block and the luminance component image block into the inter-frame prediction chrominance component restoration network to obtain an inter-frame prediction chrominance component restoration result of the target image block; An image block restoration result of the target image block is obtained according to the inter-frame prediction chrominance component restoration result and the luminance component image block.

7. An image block restoration device, characterized in that: The device comprises: An acquisition module, used to acquire a target image block with a decoding error during video decoding, and acquire corresponding information of the target image block; the corresponding information includes component information and prediction type information; The acquisition module is also used to parse each syntax element in the video bitstream in the order of syntax element levels from high to low during the video decoding process, and obtain syntax element information of each syntax element; the syntax element information includes the syntax element level, and the syntax element level includes Frame, Slice, CTU and CU from high to low; according to the syntax element information of each syntax element, obtain the target syntax element that needs to be error detected; perform error detection on the target syntax element, and determine the error level of the target syntax element according to the syntax element level of the target syntax element; the error level is divided into Warning, Skip Slice, Skip Frame, CTU Back and CTU Forward; if the error level meets the preset condition, according to the target syntax element and the syntax element level where the target syntax element is located, obtain the target image block with decoding error; The acquisition module is further used to match the syntax element information of each syntax element with each preset detection tag; if the syntax element information of the syntax element can match the preset detection tag, the syntax element that successfully matches is used as the target syntax element that needs to be error detected; A classification module, used to determine a target component type based on the component information, and determine a target prediction type based on the prediction type information; the target component type includes luminance component restoration and chrominance component restoration, and the target prediction type is divided into intra-frame prediction and inter-frame prediction; A matching module, configured to obtain a target image block restoration network corresponding to the target image block according to the target component type and the target prediction type; An input module, used for acquiring surrounding image blocks corresponding to the target image block, and taking the target image block and the surrounding image blocks as input image blocks; the surrounding image blocks and the target image block have the same syntax element level in the video decoding process; The restoration module is used to input the input image block into the target image block restoration network to obtain an image block restoration result of the target image block.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images