Methods, devices, electronic devices and media for concealing video frame errors

Abstract

This application discloses a video frame error hiding method, apparatus, electronic device, and medium, belonging to the field of communication technology. The method includes: determining a damaged video frame in a video coded bitstream; if the damaged video frame is intra-coded, error hiding the damaged macroblock according to its macroblock type, including flat blocks, edge blocks, and texture blocks; if the damaged video frame is inter-coded, error hiding the damaged macroblock according to its macroblock state; the target macroblock includes any one of the following: macroblocks adjacent to the damaged macroblock in the damaged video frame; macroblocks adjacent to the damaged macroblock in the damaged video frame; and macroblocks in a reference frame of the damaged video frame.

Description

Technical Field

[0001] This application belongs to the field of communication technology, and specifically relates to a method, apparatus, electronic device and medium for hiding video frame errors. Background Technology

[0002] With the development of communication technology, electronic devices are becoming increasingly feature-rich. For example, when a video frame in a video encoding stream is damaged, the electronic device can use an error-hiding algorithm to hide the damaged macroblocks in the damaged video frame, thereby improving the display effect of the decoded video frame.

[0003] Specifically, if the damaged video frame is intra-frame coded, the electronic device can use the bilinear interpolation (BI) method to erroneously hide the damaged macroblocks in the damaged video frame; if the damaged video frame is inter-frame coded, the electronic device can use the boundary matching algorithm (BMA) to erroneously hide the damaged macroblocks in the damaged video frame.

[0004] However, the above methods have two drawbacks. First, the BI method, used to incorrectly hide damaged macroblocks in damaged video frames, results in the loss of detail and makes the frame too blurry. Second, since the BMA algorithm hides damaged macroblocks based on the average motion vector of each macroblock in the damaged video frame, it fails to hide them when the difference between the motion vector of the damaged macroblock and the average motion vector is significant. Therefore, the related techniques for incorrectly hiding damaged macroblocks in damaged video frames are ineffective. Summary of the Invention

[0005] The purpose of this application is to provide a video frame error hiding method, apparatus, electronic device, and medium that can solve the problem of poor error hiding effect of damaged macroblocks in damaged video frames in related technologies.

[0006] In a first aspect, embodiments of this application provide a video frame error hiding method, the method comprising: determining a damaged video frame in a video encoded bitstream; if the damaged video frame is intra-coded, performing error hiding on the damaged macroblock according to the macroblock type of the damaged macroblock in the damaged video frame, the macroblock type including: flat block, edge block, and texture block; if the damaged video frame is inter-coded, performing error hiding on the damaged macroblock according to the macroblock state of a target macroblock; the target macroblock includes any one of the following: macroblocks adjacent to the damaged macroblock in the damaged video frame; macroblocks adjacent to the damaged macroblock in the damaged video frame; and macroblocks in a reference frame of the damaged video frame.

[0007] Secondly, embodiments of this application provide a video frame error hiding device, which includes a determining module and a processing module. The determining module is used to determine a damaged video frame in a video encoded bitstream. The processing module is used to, when the damaged video frame is intra-frame coded, to perform error hiding on the damaged macroblock according to the macroblock type of the damaged macroblock in the damaged video frame, wherein the macroblock type includes: flat block, edge block, and texture block; or, when the damaged video frame is inter-frame coded, to perform error hiding on the damaged macroblock according to the macroblock state of a target macroblock. The target macroblock includes any one of the following: macroblocks adjacent to the damaged macroblock in the damaged video frame; macroblocks adjacent to the damaged macroblock in the damaged video frame; and macroblocks in a reference frame of the damaged video frame.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In this embodiment, a damaged video frame in the video encoded bitstream can be determined; and if the damaged video frame is intra-coded, the damaged macroblock is error-hidden according to its macroblock type, including flat blocks, edge blocks, and texture blocks; if the damaged video frame is inter-coded, the damaged macroblock is error-hidden according to its macroblock state; the target macroblock includes any one of the following: macroblocks adjacent to the damaged macroblock in the damaged video frame; macroblocks adjacent to the damaged macroblock in the damaged video frame; and macroblocks in a reference frame of the damaged video frame. This scheme allows for error hiding of damaged macroblocks of different types within intra-frame coded video frames. For inter-frame coded damaged video frames, error hiding can be achieved using macroblocks adjacent to the damaged macroblocks, or macroblocks in a reference frame of the damaged video frame. Therefore, compared to related technologies that use a single error hiding algorithm for both intra-frame and inter-frame coded damaged video frames, the video frame error hiding method provided in this application can more accurately hide each damaged macroblock. This improves the effectiveness of error hiding of damaged macroblocks in damaged video frames. Attached Figure Description

[0013] Figure 1 This is a schematic diagram illustrating the use of the BI method in related technologies to erroneously hide damaged macroblocks;

[0014] Figure 2 This is a flowchart of the video frame error hiding method provided in the embodiments of this application;

[0015] Figure 3 This is a schematic diagram of the method for determining the adjustment amount of M pixels corresponding to a damaged pixel in the video frame error concealment method provided in the embodiments of this application;

[0016] Figure 4 This is a schematic diagram of at least one adjacent region of a damaged macroblock in the video frame error concealment method provided in this application embodiment;

[0017] Figure 5 This is a schematic diagram of the method for determining macroblocks adjacent to the boundary of a damaged macroblock in the video frame error concealment method provided in this application embodiment;

[0018] Figure 6 This is a schematic diagram of the video frame error concealment device provided in the embodiments of this application;

[0019] Figure 7 This is a schematic diagram of the electronic device provided in the embodiments of this application;

[0020] Figure 8This is a hardware schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0022] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] The following section will first explain some of the terms or terms used in the specification and claims of this application.

[0024] Internally encoded frames (i.e., I-frames): also known as internal frames, belong to intra-frame compression. The image of an I-frame is completely preserved, and only the data of the current frame is needed to decode an I-frame.

[0025] One-way predictive coded frame (P-frame): also known as difference frame or inter-frame compression, the P-frame represents the difference information between the current frame and the I-frame or the P-frame before the current frame after encoding; when decoding the P-frame, the image of the current frame needs to be superimposed with the encoded difference information defined in the current frame using the image of the P-frame or I-frame before the current frame.

[0026] Bidirectional predictive coded frames (i.e. B-frames): B-frames are bidirectional difference frames. The encoded B-frame records the difference information between the current frame and the frames before and after it. In other words, to decode a B-frame, it is necessary not only to obtain the previous buffered images, but also to decode the subsequent images and reconstruct the image of the current frame by using the encoded data of the previous and subsequent frames and the current frame.

[0027] The video frame error hiding method, apparatus, electronic device, and medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0028] The unreliability of communication channels can lead to damage or loss of video data during transmission, while the high compression rate of the video source file encoding makes the video decoder very sensitive to damaged or lost video data in the encoded video.

[0029] For example, since video (video file) is usually encoded using predictive coding and variable-length coding, if a macroblock in a certain video frame of the encoded video is damaged or lost, all video frames that require prediction of motion vectors based on the decoded video frame may not be decoded correctly. This causes the spread of bit errors in space and time, resulting in a serious deterioration in the playback quality of the decoded video, or even making it unplayable, thus distorting the decoded video source.

[0030] To reduce video source distortion caused by erroneous bitstreams (i.e., corrupted or lost video data), the correlation between video data in the spatial and temporal domains can be utilized to hide erroneous information in the video using the correct information that has already been decoded from the video.

[0031] Error concealment methods in related technologies include intra-frame error concealment and inter-frame error concealment.

[0032] 1. Intra-frame error concealment, also known as spatial error concealment, is a method for recovering data lost in the internal coded frames of a encoded video. It mainly utilizes the correlation between the damaged macroblock and the adjacent and correctly received macroblocks within the video frame containing the damaged macroblock, and uses the BI method to estimate the pixel value of the damaged pixels in the damaged macroblock, thereby concealing the error information in the video.

[0033] 2. Inter-frame error concealment (IoC) is a method for recovering lost data in P-frames / B-frames of encoded video. It primarily utilizes the temporal correlation between adjacent video frames to recover (or estimate) the motion vector (MV) of damaged macroblocks in the damaged video frame. The MV estimation of the damaged macroblock mainly uses the MVs of correctly received macroblocks surrounding the damaged macroblock as candidate MVs. Then, if the average motion amplitude of the damaged video frame is small, the MV of the damaged macroblock in the damaged video frame can be directly recovered based on the MV of a macroblock in the preceding video frame that has the same position as a damaged macroblock in the damaged video frame. If the average motion amplitude of the damaged video frame is large, the BMA or OBMA algorithm can be used to recover the MV of the damaged macroblock in the damaged video frame. In this way, error information in the video can be concealed.

[0034] For example, taking the error hiding of video using H.264 / Advanced Video Coding (AVC) reference software as an example, firstly, the electronic device can perform packet loss / error detection on the received video encoded bitstream. If packet loss / error occurs in an I-frame, the electronic device can perform error hiding on the damaged macroblocks in the corresponding video frame through intra-frame error hiding. The electronic device can take advantage of the stable spatial changes of video images and use the BI method to perform weighted interpolation on the pixel values ​​of damaged pixels in the damaged macroblock based on the boundary pixels of the damaged macroblock, thereby performing error hiding on the damaged macroblock. If packet loss / error occurs in a P-frame or B-frame, the electronic device can perform error hiding on the damaged macroblock through inter-frame error hiding. The electronic device can use the BMA algorithm to determine the corresponding candidate motion compensation macroblocks based on the smoothness of the reconstructed frame after error hiding and the MV of adjacent macroblocks, and restore the MV of the damaged macroblock based on the MV of the macroblock with the smallest boundary distortion, thereby minimizing the brightness change along the boundary of the damaged macroblock, thus performing error hiding on the damaged macroblock. Then, the electronic device can send the bitstream with the error hidden to the H.264 / AVC reference software for decoding, so that the video can be played normally.

[0035] The principles of error hiding in damaged macroblocks using the BI method and BMA algorithm are explained in detail below.

[0036] The BI method uses linear interpolation of the four pixels closest to the damaged pixel in the four correctly received neighboring macroblocks around the damaged macroblock to obtain the estimated pixel value of the damaged pixel. Figure 1 A schematic diagram illustrating the error hiding of damaged macroblocks using the BI method is shown, such as... Figure 1 As shown in the figure, the gray area indicates a damaged macroblock. For the damaged pixel 10 in the damaged macroblock, the electronic device can determine pixels 11 and 13 in the same column as the damaged pixel 10, and pixels 12 and 14 in the same row as the damaged pixel 10, from the boundary pixels of the adjacent macroblocks that have been correctly received or recovered. The estimated brightness value P of the damaged pixel 10 is then calculated using the following interpolation formula (1). i,j :

[0037]

[0038] Where P1 is the brightness value of pixel 11, P2 is the brightness value of pixel 12, P3 is the brightness value of pixel 13, P4 is the brightness value of pixel 14, d1 is the distance between pixel 11 and pixel 10, d2 is the distance between pixel 12 and pixel 10, d3 is the distance between pixel 13 and pixel 10, and d4 is the distance between pixel 14 and pixel 10.

[0039] It is understandable that after the electronic device restores each damaged pixel in the aforementioned damaged macroblock using the BI method, it can complete the error hiding of the damaged macroblock.

[0040] In the BMA algorithm, the electronic device can calculate the average motion vector V of all correctly received macroblocks in the damaged video frame based on the MV of all correctly received macroblocks using the following formula (2):

[0041]

[0042] Where N is the number of correctly received macroblocks in the damaged video frame, and V x and V y These are the horizontal and vertical components of MV, respectively.

[0043] If V is less than a preset threshold T, it indicates that the overall motion of the above-mentioned damaged video frame is relatively small. In this case, the electronic device can use the temporal error concealment mode without motion compensation to replace all the damaged macroblocks in the damaged video frame with macroblocks at the same position in the previous frame of the damaged video frame. If V is greater than or equal to the threshold T, the electronic device can use the temporal error concealment mode with motion compensation to determine a candidate MV set including the MV of macroblocks adjacent to the damaged macroblock and the zero motion vector ZMV, and select the cost function D as shown in (3) from the candidate MV set. BMA The MV with the smallest output value:

[0044]

[0045] Where, ω T ω B ω L ω R These represent whether the macroblocks adjacent to the damaged macroblock (top, bottom, left, right) are correctly received macroblocks; a value of 1 indicates a correctly received macroblock, and a value of 0 indicates a correctly received macroblock. N is the number of correctly received macroblocks in the damaged video frame; f(,,t) and f(,,t-1) represent the damaged video frame and the reference frame, respectively; f(x,y,,) represents the pixel value at the integer pixel position (x,y); mv x and MV y These are the horizontal and vertical components of a given value (MV), f(x+mv) x +i,y+mv y ,t-1) is the pixel at position (x,y) of the damaged video frame using (mv x ,mv y f(x,y-1,t) is the pixel value recovered from the reference frame; f(x,y-1,t) is the pixel value of the vertically adjacent pixel above the (x,y) position in the damaged video frame.

[0046] In determining the mission to D BMA After finding the MV with the smallest output value, since the MV can indicate a macroblock in the reference frame, the luminance value of each damaged pixel in the damaged macroblock can be updated to the luminance value of the pixel at the same position within the macroblock indicated by the MV. This completes the error hiding of the damaged macroblock.

[0047] However, when the texture of the damaged macroblock is rich, or when the damaged macroblock is located at the edge of the damaged video frame, the image reconstructed by the BI method will produce significant block artifacts and be blurry. While the BMA algorithm recovers video frames without object boundaries well, it estimates the motion degree of each damaged macroblock in the damaged video by using the average motion vector of the video frame. It is only accurate when the motion degree across the entire frame is relatively consistent. Furthermore, if the outer boundary pixels of the damaged macroblock are object boundaries, the BMA algorithm will not be able to select suitable macroblocks. Thus, related error concealment methods suffer from poor reconstruction results for edge parts and areas with rich texture details in video frames, mismatch errors at object boundaries, and poor recovery results at high bit error rates.

[0048] To address the aforementioned issues, the video frame error hiding method provided in this application embodiment can identify damaged video frames in the video encoded bitstream. If the damaged video frame is intra-coded, the damaged macroblock is error-hidden based on its macroblock type, including flat blocks, edge blocks, and texture blocks. If the damaged video frame is inter-coded, the damaged macroblock is error-hidden based on its macroblock state. The target macroblock includes any one of the following: macroblocks adjacent to the damaged macroblock in the damaged video frame; macroblocks adjacent to the damaged macroblock in the damaged video frame; and macroblocks in a reference frame of the damaged video frame. This scheme allows for error hiding of damaged macroblocks of different types within intra-frame coded video frames. For inter-frame coded damaged video frames, error hiding can be achieved using macroblocks adjacent to the damaged macroblocks, or macroblocks in a reference frame of the damaged video frame. Therefore, compared to related technologies that use a single error hiding algorithm for both intra-frame and inter-frame coded damaged video frames, the video frame error hiding method provided in this application can more accurately hide each damaged macroblock. This improves the effectiveness of error hiding of damaged macroblocks in damaged video frames.

[0049] This application provides a method for hiding video frame errors. Figure 2 A flowchart of a video frame error concealment method provided in an embodiment of this application is shown. Figure 2As shown, the video frame error hiding method provided in this application embodiment may include steps 201 to 203 as described below, or may include steps 201, 202 and 204 as described below. The following is an exemplary description using an electronic device performing the method as an example.

[0050] Step 201: The electronic device identifies the damaged video frames in the video encoded bitstream.

[0051] Optionally, in this embodiment of the application, the electronic device can perform packet loss detection or bit error detection on the received video encoded bitstream, and determine the video frames with packet loss or bit error as damaged video frames.

[0052] Optionally, in this embodiment of the application, the electronic device may perform packet loss detection or bit error detection during the process of receiving the video encoded bitstream; or, the electronic device may perform packet loss detection or bit error detection after receiving the video encoded bitstream.

[0053] Step 202: The electronic device determines the encoding type of the damaged video frame.

[0054] For methods on how electronic devices determine the encoding type of damaged video frames, please refer to the specific descriptions in relevant technologies. To avoid repetition, they will not be elaborated here.

[0055] In this embodiment, the electronic device can determine the encoding type of the damaged video frame. If the damaged video frame is intra-frame coded, the electronic device can continue to execute step 203 (one possible implementation); if the damaged video frame is inter-frame coded, the electronic device can continue to execute step 204 (another possible implementation).

[0056] One possible implementation

[0057] Step 203: When the damaged video frame is intra-coded, the electronic device performs error hiding of the damaged macroblock according to the macroblock type of the damaged macroblock in the damaged video frame.

[0058] Optionally, in embodiments of this application, the macroblock type may include: (a) flat block, (b) edge block, and (c) texture block.

[0059] Specifically, if the macroblock type of the damaged macroblock is a flat block, it means that the damaged macroblock image is relatively flat and has no complex texture characteristics; if the macroblock type of the damaged macroblock is an edge block, it means that the edge information of the damaged macroblock image is relatively simple; if the macroblock type of the damaged macroblock is a texture block, it means that the edge information of the damaged macroblock image is relatively complex.

[0060] The specific method for determining the macroblock type of a damaged macroblock in an electronic device will be described in detail in the following embodiments, and will not be repeated here to avoid repetition.

[0061] Optionally, in this embodiment of the application, the intra-frame encoded damaged video frame can be an I-frame.

[0062] Optionally, in the embodiments of this application, when the macroblock type is the flat block described above (a), the above step 203 can be implemented by the following step 203a; when the macroblock type is the edge block described above (b), the above step 203 can be implemented by the following step 203b.

[0063] Step 203a: When the damaged video frame is intra-coded and the macroblock type of the damaged macroblock in the damaged video frame is flat block, the electronic device uses the BI method to error-hide the damaged macroblock.

[0064] For a detailed description of the BI method, please refer to the relevant description of formula (1) in the above embodiments. To avoid repetition, it will not be repeated here.

[0065] In this embodiment of the application, since the computational complexity of the BI method is low, using the BI method to hide errors in flat blocks can result in a better recovery effect for damaged macroblocks.

[0066] Step 203b: When the damaged video frame is intra-coded and the macroblock type of the damaged macroblock in the damaged video frame is edge block, the electronic device uses one-way interpolation to erroneously hide the damaged macroblock.

[0067] Optionally, in this embodiment of the application, when the electronic device uses unidirectional interpolation to erroneously hide a damaged macroblock, the interpolation direction can be the prediction direction corresponding to the maximum cumulative gradient intensity among all prediction directions.

[0068] For a detailed description of the unidirectional interpolation method, prediction direction, and cumulative gradient strength, please refer to the relevant descriptions in the following embodiments. To avoid repetition, they will not be repeated here.

[0069] In this embodiment, the electronic device can use different error hiding methods (i.e., BI method or one-way interpolation method) to hide the errors of the damaged macroblock according to the different macroblock types, thus improving the flexibility of the electronic device in hiding the errors of the damaged macroblock.

[0070] Optionally, in the embodiments of this application, when the macroblock type is the texture block described above (c), step 203 can be implemented by steps 203c and 203d as described below.

[0071] It is understandable that the electronic device completes the error hiding of the damaged macroblock after performing the following steps on each damaged pixel in the damaged macroblock of the macroblock type of texture block.

[0072] Step 203c: When the damaged video frame is intra-coded and the macroblock type of the damaged macroblock in the damaged video frame is a texture block, the electronic device determines the M pixel adjustment amounts corresponding to the damaged pixel based on the M prediction directions and the position information of the damaged pixel in the damaged macroblock.

[0073] In this embodiment of the application, each prediction direction corresponds to a pixel adjustment amount, and M is an integer greater than 2.

[0074] Optionally, in the embodiments of this application, the above-mentioned M prediction directions can be evenly distributed, that is, the included angle between any two adjacent prediction directions is the same.

[0075] For example, the above M prediction directions can be 8 directions uniformly distributed in the range of 0-π, namely: 0, π / 8, π / 4, 3π / 8, π / 2, 5π / 8, 3π / 4, and 7π / 8.

[0076] It should be noted that the embodiments of this application do not limit the method of setting the above-mentioned M prediction directions. For example, the electronic device can set the M prediction directions to 6 directions that are evenly distributed in the range of 0-π.

[0077] The following description, in conjunction with the accompanying drawings, uses the example of an electronic device determining a pixel adjustment amount (hereinafter referred to as the target pixel adjustment amount) corresponding to a damaged pixel (hereinafter referred to as the target damaged pixel) in a damaged macroblock to illustrate the specific method by which an electronic device determines the M pixel adjustment amounts corresponding to the target damaged pixel.

[0078] For example, such as Figure 3 As shown, assuming the direction indicated by arrow a->b is one of the above M prediction directions, the electronic device can determine the intersection points X and Y of the extension line l of the target damaged pixel 31 in the prediction direction and the boundary of the damaged macroblock 30 based on the prediction direction and the position information of the target damaged pixel 31 in the damaged macroblock 30. It can also determine the pixel 32 closest to the intersection point X and the pixel 33 closest to the intersection point Y in the surrounding macroblocks of the damaged macroblock 30. If the pixel value of pixel 32 is P1, the pixel value of pixel 33 is P2, the distance between pixel 32 and pixel 31 is d1, and the distance between pixel 33 and pixel 31 is d2, the electronic device can use unidirectional interpolation to calculate the target pixel adjustment amount P corresponding to the target damaged pixel 31 using the following formula (4). i,j :

[0079]

[0080] It can be seen that the prediction direction indicated by arrow a->b corresponds to the target pixel adjustment amount.

[0081] It can be understood that the electronic device can determine a pixel adjustment amount for the target damaged pixel in each of the above M prediction directions except for the prediction direction shown by arrow a->b, thereby determining the M pixel adjustment amounts corresponding to the target damaged pixel.

[0082] Step 203d: The electronic device updates the pixel value of the damaged pixel based on the M pixel adjustment amount.

[0083] Optionally, in this embodiment of the application, the pixel value of the damaged pixel can be the brightness value of the damaged pixel.

[0084] Optionally, in this embodiment of the application, taking the above-mentioned damaged target pixel as an example: the updated pixel value of the damaged target pixel is: the initial pixel value of the damaged target pixel + the sum of the M pixel adjustment amounts corresponding to the damaged target pixel value.

[0085] In this embodiment, the initial pixel value of the damaged pixel is the remaining pixel value after the damaged pixel is damaged.

[0086] Specifically, the electronic device can update the pixel value of the target damaged pixel using an iterative method based on M pixel adjustment values. Alternatively, the electronic device can first sum the M pixel adjustment values, and then add the initial pixel value of the target damaged pixel to the sum of the M pixel adjustment values.

[0087] It is understandable that for each damaged macroblock, the electronic device can update the pixel values ​​of all damaged pixels in a damaged macroblock according to the above method, thereby completing the error hiding of the damaged macroblock.

[0088] In this embodiment, since the macroblock type of the damaged macroblock is a texture block type, the electronic device can update the pixel value of the damaged pixel in each prediction direction based on the pixel adjustment amount corresponding to each prediction direction, thereby improving the accuracy of the electronic device in updating the pixel value of the damaged pixel.

[0089] Optionally, in this embodiment of the application, before step 203d above, the video frame error hiding method provided in this embodiment of the application may further include step 203e below; then, step 203d above can be specifically implemented by step 203d1 below.

[0090] Step 203e: The electronic device determines the pixel adjustment weight corresponding to each prediction direction based on each prediction direction.

[0091] Optionally, in this embodiment, the electronic device may determine the pixel adjustment weight corresponding to each prediction direction based on the cumulative gradient intensity corresponding to each prediction direction (method one), or based on each prediction direction and the location information of the damaged pixel (method two), or may include both methods one and two at the same time.

[0092] The following sections will provide detailed explanations of Method 1 and Method 2.

[0093] Method 1

[0094] Optionally, in the embodiments of this application, the above step 203e can be specifically implemented by the following step 203e1.

[0095] Step 203e1: The electronic device determines the pixel adjustment weights for each prediction direction based on the cumulative gradient intensity corresponding to each prediction direction.

[0096] Optionally, in this embodiment, the electronic device can normalize the cumulative gradient intensity corresponding to each prediction direction to determine the pixel adjustment weight corresponding to each prediction direction. Specifically, assuming the prediction directions set by the electronic device are 8 directions uniformly distributed within the range of 0-π, the electronic device can normalize the cumulative gradient intensity corresponding to each prediction direction using the following formula (5):

[0097]

[0098] Where, φ k W is one of the eight prediction directions. k For the predicted direction φ k The corresponding pixel adjustment weight, Stren(φ) k ) represents the predicted direction φ k The corresponding cumulative gradient intensity, It is the sum of the cumulative gradient intensities corresponding to the 8 prediction directions.

[0099] It can be seen that, through the above formula (5), the electronic device can determine the pixel adjustment weight corresponding to each prediction direction, and the pixel adjustment weight of different damaged pixels in the damaged macroblock is the same in the same prediction direction.

[0100] In this embodiment of the application, W k It can be used to weight the interpolated pixels for each prediction direction, W k It represents the contribution of the predicted value in each prediction direction to the final interpolated pixel.

[0101] Optionally, in this embodiment of the application, before step 203e1 above, the video frame error hiding method provided in this embodiment of the application may further include the following steps A to C.

[0102] Step A: The electronic device detects the orientation angle and gradient intensity of pixels in at least one adjacent region of the damaged macroblock.

[0103] In this embodiment, the adjacent region of a damaged macroblock is the region adjacent to the boundary of the damaged macroblock (excluding corner adjacent regions).

[0104] In this embodiment of the application, the above-mentioned at least one adjacent region and the region where the damaged macroblock is located are different regions in the same damaged video frame.

[0105] Optionally, in the embodiments of this application, each of the above-mentioned at least one adjacent region can be any of the following: a region in the damaged video frame adjacent to the upper boundary of the damaged macroblock, a region in the damaged video frame adjacent to the lower boundary of the damaged macroblock, a region in the damaged video frame adjacent to the left boundary of the damaged macroblock, or a region in the damaged video frame adjacent to the right boundary of the damaged macroblock.

[0106] It is understood that the number of adjacent regions in at least one of the above adjacent regions can be 2, 3 or 4, which can be determined based on the position of the damaged macroblock in the damaged video frame.

[0107] Optionally, in this embodiment of the application, each of the above-mentioned at least one adjacent region can be: at least a portion of a macroblock adjacent to the boundary of the damaged macroblock, and the at least a portion of the region is adjacent to the damaged macroblock.

[0108] It should be noted that in the embodiments of this application, when there are multiple adjacent regions in at least one adjacent region, the size of each adjacent region is the same, that is, the number of pixels in each adjacent region is the same.

[0109] For example, such as Figure 4 As shown, regions 41, 42, 43, and 44 are adjacent regions (i.e., at least one adjacent region) of the damaged macroblock, and regions 41, 42, 43, and 44 are different regions within the same damaged video frame 40 as the region where the damaged macroblock is located; and the damaged macroblock includes 16×16 pixels, the pixels in region 41 are 8×16 pixels adjacent to the upper boundary of the damaged macroblock, the pixels in region 42 are 8×16 pixels adjacent to the lower boundary of the damaged macroblock, the pixels in region 43 are 16×8 pixels adjacent to the left boundary of the damaged macroblock, and the pixels in region 44 are 16×8 pixels adjacent to the right boundary of the damaged macroblock.

[0110] Optionally, in this embodiment of the application, the electronic device can detect the orientation angle and gradient intensity of pixels in at least one adjacent region of a damaged macroblock by calculating the edge vectors of pixels in at least one adjacent region.

[0111] The following example illustrates a specific method for an electronic device to detect the orientation angle of a pixel in at least one adjacent region of a damaged macroblock.

[0112] For example, assuming the first pixel is the pixel at position (i,j) in the damaged video frame, and the brightness value of the first pixel is Pi,j, then the electronic device can define the edge vector of the first pixel. for:

[0113]

[0114] in, for The component in the X-axis direction, for The component in the Y-axis direction, and the electronic device can be calculated using the Sobel operator or the Prewitt operator. and The formulas for the convolution kernels of the Sobel operator on the X and Y axes are as follows:

[0115]

[0116] The formulas for the convolution kernels of the Prewitt operator on the X and Y axes are as follows:

[0117]

[0118] Computation of electronic devices using the Sobel operator and For example, electronic devices can be calculated using the above formula (7). and as follows:

[0119]

[0120]

[0121] Thus, the electronic device calculates the edge vector of the first pixel. of and Then, it can be calculated using the following formula (11). Direction angle

[0122]

[0123] Such electronic devices can be based on The direction angle θ of the first pixel was detected. (i,j) for:

[0124]

[0125] For a description of how an electronic device detects the directional angle of other pixels in at least one adjacent region, please refer to the above description of the directional angle of the first pixel. To avoid repetition, it will not be repeated here.

[0126] The following example, using the first pixel in at least one adjacent region as an example, illustrates a specific method for an electronic device to detect the gradient intensity of pixels in at least one adjacent region of a damaged macroblock.

[0127] For example, if the electronic device calculates the edge vector of the first pixel using the method described above... of and Electronic devices can be based on and And calculate using the following formula (13) amplitude

[0128]

[0129] Thus, the electronic device can detect the gradient intensity G of the first pixel. (i,j) for:

[0130]

[0131] For a description of how an electronic device detects the gradient intensity of other pixels in at least one adjacent region, please refer to the above description of detecting the gradient intensity of the first pixel. To avoid repetition, it will not be repeated here.

[0132] Step B: The electronic device determines the prediction direction corresponding to the orientation angle of each pixel.

[0133] In this embodiment of the application, each predicted direction corresponds to a range of directional angles.

[0134] It is understandable that electronic devices can pre-set the correspondence between predicted directions and directional angle ranges. Thus, after detecting the directional angle of a pixel, the electronic device can determine the predicted direction corresponding to that pixel's directional angle based on the directional angle range within which it falls.

[0135] In this embodiment, after detecting the orientation angle of pixels within at least one adjacent region, the electronic device can determine the predicted direction corresponding to the orientation angle of each pixel based on the correspondence between the predicted direction and the orientation angle range. This allows for the classification of pixels within at least one adjacent region based on their orientation angles.

[0136] For example, the electronic device can pre-set the correspondence between the predicted direction and the range of direction angles, as shown in Table 1 below;

[0137] Table 1

[0138]

[0139] As shown in Table 1 above, the predicted direction is... Corresponding directional angle range [0, π / 8); predicted direction Corresponding directional angle range [π / 8, π / 4); predicted direction Corresponding directional angle range [π / 4, 3π / 8); predicted direction Corresponding directional angle range [3π / 8, π / 2); predicted direction Corresponding directional angle range [π / 2, 5π / 8); predicted direction Corresponding directional angle range [5π / 8, 3π / 4); predicted direction Corresponding directional angle range [3π / 8, 7π / 8); predicted direction The corresponding directional angle range is [7π / 8, π].

[0140] As can be seen from Table 1 above, there is a corresponding directional angle range for each predicted direction. Therefore, after detecting the directional angle of a pixel in at least one adjacent region, the electronic device can determine the predicted direction corresponding to the directional angle of each pixel based on the correspondence between the predicted direction and the directional angle range in Table 1, and the directional angle range to which the directional angle of each pixel belongs.

[0141] It is understandable that electronic devices can be configured to correspond to different ranges of predicted directions and directional angles based on actual usage requirements.

[0142] Step C: The electronic device adds up the gradient intensities of pixels corresponding to the same prediction direction to obtain the cumulative gradient intensity for each prediction direction.

[0143] Optionally, in this embodiment of the application, the electronic device can accumulate and sum the gradient intensities of each pixel corresponding to each prediction direction to obtain the cumulative gradient intensity corresponding to the prediction direction.

[0144] For example, taking the first pixel mentioned above as an example, for the predicted direction If the electronic device detects that the orientation angle of the first pixel corresponds to the predicted direction The electronic device can then predict the direction using the following formula (15). Corresponding cumulative gradient strength Update:

[0145]

[0146] Among them, G (i,j) The gradient intensity of the first pixel.

[0147] Thus, by following the above method, the electronic device can obtain the cumulative gradient intensity corresponding to each predicted direction.

[0148] Method 2

[0149] Optionally, in the embodiments of this application, the above step 203e can be specifically implemented by the following steps 203e2 and 203e3.

[0150] Step 203e2: The electronic device determines the first boundary pixel and the second boundary pixel corresponding to each prediction direction based on the location information of each prediction direction and the damaged macroblock.

[0151] For the method of determining the first boundary pixel and the second boundary pixel corresponding to each prediction direction for electronic devices, please refer to the above. Figure 3 The relevant descriptions in the method embodiments will not be repeated here to avoid repetition.

[0152] Step 203e3: The electronic device determines the pixel adjustment weight corresponding to each prediction direction based on the pixel value and position information of the first boundary pixel and the second boundary pixel corresponding to each prediction direction.

[0153] The following description, with reference to the accompanying drawings, uses the example of an electronic device determining the pixel adjustment weights corresponding to the target prediction direction to illustrate step 203e3.

[0154] For example, combined Figure 3 Assuming the target prediction direction is a diagonal (a->b) direction, and the electronic device determines the first boundary pixel as pixel 32 and the second boundary pixel as pixel 33 based on the target prediction direction and the position information of the damaged pixel 31, then the electronic device can set a pixel adjustment weight for the target prediction direction based on the pixel values ​​of pixel 32 and pixel values ​​of pixel 33 using the following formula (16). (i.e., the similarity factor of the edge pixels on both sides of the damaged macroblock):

[0155]

[0156] Where exp(.) represents an exponential function with the natural number e as the base, P1 is the pixel value of pixel 42, P2 is the pixel value of pixel 43, and γ is a constant.

[0157] Furthermore, through the above formula (16), the electronic device can determine the pixel adjustment weight corresponding to each prediction direction, and the pixel adjustment weights of different damaged pixels corresponding to the same prediction direction are different.

[0158] In the embodiments of this application, the following is adopted: Error hiding of damaged macroblocks can make the recovered damaged video frame image smoother.

[0159] It is understandable that if the electronic device determines the pixel adjustment weight corresponding to each prediction direction using methods one and two, then the pixel adjustment weight is equal to the pixel adjustment weight corresponding to each prediction direction.

[0160] In this embodiment, on the one hand, the electronic device can determine the pixel adjustment weight corresponding to each prediction direction through the above-described method one, so that the electronic device can determine the adjustment degree of each damaged pixel in the damaged macroblock according to the image edge information of the damaged macroblock in each prediction direction; on the other hand, the electronic device can determine the pixel adjustment weight corresponding to each prediction direction through the above-described method two, and a pixel adjustment weight can indicate the similarity of the two edge pixels of the damaged macroblock in the corresponding prediction direction, so that the electronic device can restore the damaged image more smoothly through the pixel adjustment weight. This can improve the flexibility of the electronic device in determining the pixel adjustment weight, thereby improving the effect of error hiding of the damaged macroblock. Step 203d1: The electronic device updates the pixel value of the damaged pixel based on M pixel adjustment amounts and M pixel adjustment weights.

[0161] In this embodiment of the application, the above-mentioned M pixel adjustment amounts correspond one-to-one with the above-mentioned M pixel adjustment weights.

[0162] The following section uses a damaged pixel as an example to explain in detail the specific method for updating the pixel value of a damaged pixel using an electronic device.

[0163] Optionally, in this embodiment of the application, the electronic device can, based on the aforementioned M pixel adjustment amounts and M pixel adjustment weights, first target the predicted direction. The pixel value P of the target damaged pixel is updated using the following formula (17):

[0164] P = P i,j +W×P′ i,j (17)

[0165] Among them, P i,j Let P′ be the initial pixel value of the target damaged pixel. i,jFor the above M pixel adjustment amounts, the target damaged pixel in the prediction direction The corresponding pixel adjustment amount, W, is the target damaged pixel in the predicted direction among the above M pixel adjustment weights. The corresponding pixel adjustment weight. It can be understood that W can be any of the following: W k ,

[0166] Then, the electronic device can target the next predicted direction. The pixel values ​​of the damaged target pixels are updated using the method described above. At this time, P in the above formula (17) is... i,j For electronic devices to predict direction The updated pixel value P′ for the damaged pixel of the target i,j For the above M pixel adjustment amounts, the target damaged pixel in the prediction direction The corresponding pixel adjustment amount, W, is the target damaged pixel in the predicted direction among the above M pixel adjustment weights. The corresponding pixel adjustment weights are then applied. This process continues until the electronic device has cumulatively updated the pixel values ​​of the damaged target pixels for each prediction direction, thus completing the update of the pixel values ​​of the damaged target pixels.

[0167] Optionally, in this embodiment of the application, after the electronic device has updated the pixel value of a damaged pixel in a damaged macroblock, it can update the pixel value of the next damaged pixel in the damaged macroblock that is closest to the damaged pixel and located to the right or below the damaged pixel, until all damaged pixels in the damaged macroblock have been updated, thus completing the hiding of the error information of the damaged macroblock.

[0168] Furthermore, the electronic device can use the above method to update all damaged pixels in all damaged macroblocks, thereby completing the error hiding of the entire video and improving the playback effect of the video on the electronic device.

[0169] In this embodiment, since the electronic device can update the pixel value of the damaged pixel based on M pixel adjustment amounts and M pixel adjustment weights determined according to each prediction direction and corresponding one-to-one with the M pixel adjustment amounts, the pixel value of the damaged pixel can be updated to different degrees in different prediction directions, thereby further improving the accuracy of the electronic device in updating the pixel value of the damaged pixel.

[0170] Another possible implementation

[0171] Step 204: When the damaged video frame is inter-frame coded, the electronic device performs error hiding of the damaged macroblock according to the macroblock status of the target macroblock.

[0172] Optionally, in the embodiments of this application, the target macroblock includes any one of the following: a macroblock adjacent to the damaged macroblock in a damaged video frame; a macroblock adjacent to the damaged macroblock in a damaged video frame; and a macroblock in a reference frame of the damaged video frame.

[0173] Optionally, in this embodiment of the application, the macroblock in the reference frame of the damaged video frame can be: a macroblock in the reference frame of the damaged video frame that is at the same position as the damaged macroblock in the damaged video frame.

[0174] Optionally, in this embodiment of the application, the macroblock status may include: correctly received or damaged.

[0175] Optionally, in this embodiment of the application, the damaged video frame coded by inter-frame coding can be a P-frame or a B-frame.

[0176] It is understandable that electronic devices can complete the error hiding of damaged video frames after they have incorrectly hidden all damaged macroblocks in the damaged video frames.

[0177] Furthermore, after the electronic device performs error hiding on each damaged video frame in the video encoding stream, it can complete the error hiding of the entire video encoding stream, thereby improving the video playback effect of the electronic device.

[0178] It should be noted that, for ease of description, the following embodiments do not limit the number of damaged video frames in the video encoded bitstream, nor the number of damaged macroblocks in each damaged video frame. That is, the number of damaged video frames in the video encoded bitstream can be one or more; correspondingly, the number of damaged macroblocks in each damaged video frame can be one or more; the specific number can be determined according to actual usage requirements.

[0179] Optionally, in the embodiments of this application, step 204 can be implemented by steps 204a and 204b as described below.

[0180] Step 204a: The electronic device determines the first candidate motion vector set of the damaged macroblock based on the first quantity or the second quantity.

[0181] In this embodiment of the application, the first quantity is the number of macroblocks in the damaged video frame that are adjacent to the damaged macroblock and are correctly received, and the second quantity is the number of macroblocks in the damaged video frame that are adjacent to the damaged macroblock and are damaged.

[0182] It can be understood that a macroblock that is adjacent to a damaged macroblock and is correctly received is a macroblock that is adjacent to the boundary of the damaged macroblock and has no missing or erroneous pixels.

[0183] Optionally, in this embodiment of the application, when the first quantity is greater than or equal to the first preset quantity, or the second quantity is less than or equal to the second preset quantity, the first candidate motion vector set includes the second candidate motion vector set.

[0184] Optionally, in this embodiment of the application, when the first quantity is less than the first preset quantity, or the second quantity is greater than the second preset quantity, the first candidate motion vector set includes the second candidate motion vector set and the third candidate motion vector set.

[0185] Optionally, in this embodiment of the application, the first preset quantity can be an integer greater than 0 and less than or equal to 4; for example, the first preset quantity is 2.

[0186] Optionally, in this embodiment of the application, the second preset quantity can be an integer greater than 0 and less than or equal to 4; the second preset quantity can be the same as or different from the first preset quantity.

[0187] In this embodiment of the application, the second candidate motion vector set includes: motion vectors of macroblocks adjacent to and correctly received in the damaged video frame, zero motion vectors, and median motion vectors of motion vectors of macroblocks adjacent to and correctly received in the damaged video frame; the third candidate motion vector set includes: motion vectors of macroblocks in the reference frame of the damaged video frame that are at the same position as the damaged macroblock.

[0188] Optionally, in this embodiment of the application, the median motion vector can be: a motion vector determined based on the median of all X-axis components and the median of all Y-axis components of the motion vector of the macroblock adjacent to the damaged macroblock in the damaged video frame and correctly received.

[0189] Optionally, in this embodiment of the application, the reference frame of the damaged video frame can be any video frame in the video encoding bitstream that is continuous with the damaged video frame and located before the damaged video frame, where L is a positive integer less than or equal to 16.

[0190] For example, the reference frame for a damaged video frame is the video frame preceding the damaged video frame; in other words, the reference frame is the last video frame received before the damaged video frame.

[0191] Optionally, in the embodiments of this application, a macroblock whose position in the reference frame is the same as the position of the damaged macroblock can be understood as: the position information of the macroblock in the reference frame is the same as the position information of the damaged macroblock in the damaged video frame.

[0192] It is understandable that the motion vectors of macroblocks that are adjacent to and correctly received in the damaged video frame have the highest correlation with the motion vectors of the damaged macroblocks. Furthermore, the number of selected motion vectors included in the first candidate motion vector set is moderate, which can reduce the computational complexity while ensuring the accuracy of incorrect hiding of damaged macroblocks.

[0193] The video frame error hiding method provided in the embodiments of this application will be described exemplarily below with reference to the accompanying drawings.

[0194] For example, such as Figure 5 As shown, each macroblock adjacent to the boundary of the damaged macroblock E in the damaged video can be divided into multiple sub-macroblocks. It can be seen that the electronic device identifies the topmost sub-macroblock A among the multiple sub-macroblocks adjacent to the left boundary of the damaged macroblock E as the macroblock adjacent to the left boundary of the damaged macroblock E; it identifies the leftmost sub-macroblock B among the multiple sub-macroblocks adjacent to the upper boundary of the damaged macroblock E as the macroblock adjacent to the upper boundary of the damaged macroblock E; and it identifies the leftmost sub-macroblock D among the multiple sub-macroblocks adjacent to the lower boundary of the damaged macroblock E as the macroblock adjacent to the lower boundary of the damaged macroblock E. It can be understood that if the macroblocks adjacent to the right boundary of the damaged macroblock E also include the divided sub-macroblocks, then the electronic device can identify the topmost sub-macroblock among them as the macroblock adjacent to the right boundary of the damaged macroblock E.

[0195] Furthermore, assuming the first preset number is 2, if the number of correctly received macroblocks in macroblock A, macroblock B, macroblock C, and macroblock D (i.e., the first number) is greater than or equal to 2, the electronic device can determine that the first candidate motion vector set is the second candidate motion vector set; if the number of correctly received macroblocks in macroblock A, macroblock B, macroblock C, and macroblock D is less than 2, the electronic device can determine that the first candidate motion vector set includes the second candidate motion vector set and the third candidate motion vector set.

[0196] In this embodiment, since the electronic device can determine different first candidate motion vector sets of the damaged macroblock based on the relationship between the first quantity or the second quantity and the corresponding preset quantity, it can ensure that there are enough candidate motion vectors when the bit error rate is large (i.e., the number of macroblocks adjacent to the damaged macroblock and correctly received is small), thereby improving the robustness of the electronic device in erroneously hiding the damaged macroblock.

[0197] Step 204b: The electronic device performs error hiding on the damaged macroblock based on the first candidate motion vector set.

[0198] In this embodiment of the application, each motion vector in the first candidate motion vector set indicates a macroblock in a reference video frame.

[0199] Optionally, in this embodiment of the application, the electronic device can determine a reference macroblock with the same position information in the reference frame based on the position information of the zero motion vector in the first candidate motion vector set, and can determine a macroblock in the reference frame corresponding to each of the other motion vectors in the first candidate motion vector set relative to the zero motion vector based on the offset of each of the other motion vectors in the first candidate motion vector set relative to the zero motion vector.

[0200] In this embodiment of the application, since the damaged video frame is inter-frame coded, the electronic device can perform error hiding of the damaged macroblock based on the first candidate motion vector set corresponding to the damaged macroblock determined by the first quantity or the second quantity, thus further improving the accuracy of the electronic device in performing error hiding of the damaged macroblock.

[0201] The following section details the specific method for electronic devices to error-hide damaged macroblocks based on the first candidate motion vector set.

[0202] Optionally, in the embodiments of this application, step 204b can be specifically implemented through steps 204b1 to 204b3 as described below.

[0203] Step 204b1: The electronic device determines the candidate motion vector that minimizes the loss cost function value from the first candidate motion vector set as the target motion vector.

[0204] In the embodiments of this application, the aforementioned loss cost function D tot Represented as:

[0205] D tot =α1·D BMA +α2·D OBMA (18)

[0206] Among them, D BMA The boundary matching cost function is given by formula (3) above, D OBMA The cost function is the overlapping boundary matching function, and α1 and α2 are preset weights.

[0207] In this embodiment of the application, D OBMA It can be represented as:

[0208]

[0209] For a detailed description of formula (19), please refer to the relevant description in formula (3) above. To avoid repetition, it will not be repeated here.

[0210] Optionally, in this embodiment of the application, after determining the first candidate motion vector set, the electronic device can substitute each candidate motion vector in the first candidate motion vector set into the above formula (18) and determine a value from it that allows D to be used. totThe smallest target motion vector.

[0211] Optionally, in this embodiment of the application, if α1 = 1 and α2 = 0, then D tot =D BMA That is, electronic devices can use D BMA The algorithm performs error hiding on damaged macroblocks; if α1 = 0 and α2 = 1, then D tot =D OBMA That is, electronic devices can use D OBMA The algorithm incorrectly hides the damaged macroblocks; if α1 and α2 are both positive real numbers greater than 0, the electronic device can pass through D. BMA With D OBMA By combining loss cost functions, errors in damaged macroblocks are incorrectly hidden. This improves the flexibility of electronic devices in incorrectly hiding damaged macroblocks.

[0212] Step 204b2: The electronic device determines the motion compensation block of the target motion vector in the reference frame of the damaged video frame.

[0213] Optionally, in this embodiment of the application, the electronic device can determine the motion compensation block of the target motion vector in the reference frame of the damaged video frame based on the offset of the target motion vector relative to the zero motion vector.

[0214] Step 204b3: The electronic device performs error hiding of the damaged macroblock based on the motion compensation block.

[0215] Optionally, in this embodiment, the electronic device can replace the pixel value of each damaged pixel in the damaged macroblock with the pixel value of a pixel in the motion compensation block that has the same positional information as each damaged pixel. This allows for the erroneous concealment of the damaged macroblock.

[0216] In this embodiment of the application, since the electronic device can be connected via D BMA With D OBMA By combining the loss cost function, a motion compensation block in the reference frame is determined, and the damaged macroblock can be incorrectly hidden based on the motion compensation block. Therefore, the mismatch error problem when the electronic device restores the object boundary can be solved, and the pixels on both sides of the restored damaged macroblock can be smoother, thereby further improving the effect of the electronic device in incorrectly hiding the damaged macroblock.

[0217] In the video frame error hiding method provided in this application embodiment, for intra-coded damaged video frames, different methods can be used to hide damaged macroblocks of different macroblock types within the damaged video frame; for inter-coded damaged video frames, the damaged macroblocks can be hidden based on macroblocks adjacent to the damaged macroblocks in the damaged video frame, or macroblocks in the reference frame of the damaged video frame and the adjacent macroblocks. Therefore, compared to related technologies that use a single error hiding algorithm to hide damaged macroblocks for both intra-coded and inter-coded damaged video frames, the video frame error hiding method provided in this application embodiment can more accurately hide each damaged macroblock. This improves the effectiveness of error hiding for damaged macroblocks in damaged video frames.

[0218] Optionally, in one possible implementation of this application embodiment, before step 203, the video frame error hiding method provided in this application embodiment may further include step 205.

[0219] Step 205: The electronic device performs edge detection on the damaged macroblock and determines the macroblock type of the damaged macroblock based on the edge information detected by the edge detection.

[0220] Optionally, in this embodiment, edge detection may include flatness detection; or, edge detection may include flatness detection and texture complexity detection.

[0221] It is understandable that the edge information of a macroblock can indicate the edge texture characteristics of the macroblock image, thereby enabling electronic devices to determine the macroblock type of a damaged macroblock based on this edge information.

[0222] It should be noted that in actual implementation, electronic devices can also use any other possible methods to determine the macroblock type of a damaged macroblock, which can be determined according to actual usage requirements.

[0223] In this embodiment of the application, since the electronic device can perform edge detection on the damaged macroblock, it can obtain the flatness and texture complexity of the damaged macroblock image. Thus, when the electronic device determines the macroblock type of the damaged macroblock based on the edge information of the edge detection, it can do so based on the flatness and texture complexity of the damaged macroblock image, thereby improving the accuracy of the electronic device in determining the macroblock type.

[0224] Optionally, in the embodiments of this application, step 205 can be implemented by steps 205a to 204c as described below, or by steps 205a, 205b, 205d and 205e as described below.

[0225] Step 205a: The electronic device detects the gradient intensity of pixels in at least one adjacent region of the damaged macroblock.

[0226] For a detailed description of step 205a, please refer to the relevant description in the above embodiments. To avoid repetition, it will not be repeated here.

[0227] Step 205b: The electronic device determines the flatness threshold of the damaged macroblock based on the gradient intensity.

[0228] Optionally, in this embodiment of the application, since the gradient intensity of a pixel can indicate the flatness of an image, the electronic device can determine the flatness threshold of a damaged macroblock based on a gradient intensity threshold (hereinafter referred to as the first gradient intensity threshold).

[0229] Step 205c: When the flatness of a damaged macroblock is less than the flatness threshold, the electronic device determines the macroblock type of the damaged macroblock as a flat block.

[0230] Optionally, in this embodiment of the application, the electronic device can determine the flatness of the damaged macroblock based on the maximum gradient intensity and / or the second largest gradient intensity among the gradient intensities of pixels in at least one adjacent region.

[0231] For example, taking the determination of the flatness of a damaged macroblock by an electronic device based on the aforementioned maximum gradient intensity as an example, if the maximum gradient intensity is less than a first gradient intensity threshold, the electronic device can determine that the flatness of the damaged macroblock is less than the flatness threshold, meaning the damaged macroblock image is relatively flat and lacks complex texture characteristics; if the maximum gradient intensity is greater than or equal to the first gradient intensity threshold, the electronic device can determine that the flatness of the damaged macroblock is greater than or equal to the flatness threshold, meaning the damaged macroblock image contains relatively rich texture regions. Therefore, the electronic device can determine the macroblock type of the damaged macroblock as a flat block when the flatness of the damaged macroblock is less than the flatness threshold.

[0232] In this embodiment of the application, since the gradient intensity of a pixel can reflect the edge features of an image, the electronic device can determine the macroblock type of the damaged macroblock as a flat block based on the gradient intensity of pixels in at least one adjacent region of the detected damaged macroblock, which can improve the accuracy of the electronic device in determining the macroblock type of the damaged macroblock.

[0233] Step 205d: When the flatness of the damaged macroblock is greater than or equal to the flatness threshold, the electronic device detects the texture complexity of the damaged macroblock.

[0234] Optionally, in the embodiments of this application, the above step 205d can be implemented by the following steps 205d1 and 205d2.

[0235] Step 205d1: When the flatness of the damaged macroblock is greater than or equal to the flatness threshold, the electronic device acquires the pixel gradient intensity of at least one pixel prediction direction.

[0236] In this embodiment of the application, each pixel prediction direction in the above-mentioned at least one pixel prediction direction corresponds to a pixel gradient intensity, and the pixel gradient intensity is the sum of the gradient intensities of pixels within the angle range corresponding to the pixel prediction direction.

[0237] For the other descriptions in step 205d1, please refer to the relevant descriptions of the prediction direction and gradient intensity in the above embodiments. To avoid repetition, they will not be repeated here.

[0238] Step 205d2: The electronic device determines the edge texture complexity of the damaged macroblock based on the cumulative gradient intensity and the maximum gradient intensity among at least one pixel gradient intensity.

[0239] In this embodiment of the application, the cumulative gradient intensity is the sum of the gradient intensities of at least one pixel.

[0240] Optionally, in this embodiment of the application, the electronic device can determine the texture complexity of the damaged macroblock based on the ratio of the maximum gradient intensity to the cumulative gradient intensity.

[0241] Optionally, in this embodiment of the application, if the above ratio is less than a gradient intensity threshold (hereinafter referred to as the second gradient intensity threshold), the electronic device can determine that the texture complexity of the damaged macroblock is large, that is, the edge information of the damaged macroblock image is more complex, and there is more edge information in other prediction directions besides the prediction direction corresponding to the maximum cumulative gradient intensity; if the above ratio is greater than or equal to the second gradient intensity threshold, the electronic device can determine that the texture complexity of the damaged macroblock is small, that is, the edge information of the damaged macroblock image is less.

[0242] It should be noted that in actual implementation, the electronic device can also determine the texture complexity of the damaged macroblock based on the ratio of the second largest gradient intensity to the cumulative gradient intensity among the above-mentioned at least one pixel gradient intensity. The specific value can be set according to actual usage requirements, and this application embodiment does not limit it.

[0243] In this embodiment, since image edges are one of the basic features of an image, they appear in the form of local feature discontinuities, such as abrupt changes in grayscale values, color, and texture, and are characterized by a large gradient intensity in a certain direction; while macroblocks with rich detail components have large gradient intensities in many or even all directions; therefore, the electronic device can determine the texture complexity of the damaged macroblock based on the pixel gradient intensity in the prediction direction of at least one pixel.

[0244] In this embodiment, since the electronic device can determine the texture complexity of the damaged macroblock based on the ratio of the maximum gradient intensity to the cumulative gradient intensity in at least one pixel gradient intensity, and this ratio can accurately reflect the edge information of the damaged macroblock image, the accuracy of the electronic device in determining the texture complexity of the damaged macroblock can be improved.

[0245] Step 205e: The electronic device determines the macroblock type of the damaged macroblock based on texture complexity.

[0246] Optionally, in this embodiment of the application, if the flatness of the damaged macroblock is less than the aforementioned flatness threshold and the texture complexity of the damaged macroblock is greater than or equal to the texture complexity threshold, the electronic device can determine that the macroblock type of the damaged macroblock is an edge block; if the flatness of the damaged macroblock is less than the aforementioned flatness threshold and the texture complexity of the damaged macroblock is less than the texture complexity threshold, the electronic device can determine that the macroblock type of the damaged macroblock is a texture block.

[0247] Optionally, in this embodiment of the application, the texture complexity threshold can be determined by the electronic device based on the second gradient intensity threshold.

[0248] It is understandable that electronic devices can determine the macroblock type of a damaged macroblock as a flat block through the above flatness detection, and can determine the macroblock type of a damaged macroblock as an edge block or a texture block through the above texture complexity detection.

[0249] In this embodiment of the application, since the electronic device can determine whether the macroblock type of the damaged macroblock is an edge block or a texture block based on the texture complexity of the detected damaged macroblock, the accuracy of the electronic device in determining the macroblock type of the damaged macroblock can be further improved.

[0250] For example, suppose an electronic device encodes a Quarter Common Intermediate Format (QCIF) (176×144) video sequence, where each frame contains 99 macroblocks of size 16×16. During video encoding, the electronic device disables B-frame prediction and only uses I-frames and P-frames for encoding, with I-frames inserted periodically, one I-frame inserted every second. The encoder in the electronic device is set to frame encoding mode (note: if field encoding mode is used, the relevant configuration parameters need to be multiplied by 2), the frame rate is 24 frames / second, the motion estimation search range is W=16, the number of reference video frames is set to 1, rate-distortion optimization encoding is enabled, and the quantization parameter QP=28. The encoded bitstream is transmitted through a channel with a packet loss rate of 5%. After receiving the bitstream, the electronic device can enable error detection. If no error is detected, normal decoding proceeds. If an error is detected, the error hiding module is invoked to recover the erroneous bitstream. Electronic devices can use the error hiding module to hide damaged macroblocks using the video frame error hiding method provided in the embodiments of this application.

[0251] Scenario 1: Error concealment of damaged macroblocks in I-frames

[0252] Step 1: The electronic device detects the flatness of the damaged macroblock. If the estimated image of the damaged macroblock is relatively flat and lacks complex texture characteristics, the electronic device determines that the damaged macroblock is a flat block and uses the BI method to hide the error. Otherwise, the electronic device continues with texture complexity detection.

[0253] Step 2: The electronic device detects the texture complexity of the damaged macroblock; if the damaged macroblock is an edge block, the electronic device uses unidirectional interpolation to incorrectly hide the damaged macroblock; if the damaged macroblock is a texture block, the electronic device uses an improved multidirectional interpolation method (i.e., a method that updates the pixel value of the damaged pixel based on M pixel adjustment amounts and M pixel adjustment weights) to incorrectly hide the damaged macroblock.

[0254] Step 3: Repeat steps 1 to 2. The electronic device can hide the damaged macroblocks in the order of first the damaged macroblocks on both sides and then the damaged macroblocks in the middle, until all the damaged macroblocks in the current I-frame have been processed.

[0255] Scenario 2: Error concealment of damaged macroblocks in P-frames or B-frames

[0256] Step 1: The electronic device determines the status of the four macroblocks adjacent to the current damaged macroblock above, below, left, and right. If at least two of these macroblocks are correctly received macroblocks, the first candidate motion vector set is determined to be the second candidate motion vector set. Otherwise, the first candidate motion vector set is determined to include the second candidate motion vector set and the third candidate motion vector set (i.e., the extended candidate motion vector set).

[0257] Step 2: The electronic device selects the loss cost function D from the first candidate motion vector set. tot The smallest motion vector is taken as the optimal motion vector for the damaged macroblock.

[0258] Step 3: Based on the optimal motion vector determined in Step 2, determine the corresponding motion compensation macroblock in the reference frame of the damaged video frame, and insert it into the position of the current damaged macroblock (that is, replace the pixel value of the damaged pixel with the pixel value of the pixel with the same position information in the motion compensation macroblock).

[0259] Step 4: Repeat steps 1 to 3. The electronic device can hide the damaged macroblocks in the order of first the damaged macroblocks on both sides and then the damaged macroblocks in the middle, until all damaged macroblocks in the current P-frame have been processed.

[0260] As can be seen, the video frame error hiding method provided in this application fully utilizes the correlation of video signals in the spatial and temporal domains, effectively improving the performance of error hiding algorithms in traditional technologies, thereby enabling electronic devices to still decode and recover clear and smooth video images from erroneous bitstreams.

[0261] The video frame error hiding method provided in this application can be executed by a video frame error hiding device. This application uses the execution of the video frame error hiding method by a video frame error hiding device as an example to illustrate the video frame error hiding device provided in this application.

[0262] Combination Figure 6 This application provides a video frame error hiding device 60, which may include a determining module 61 and a processing module 62. The determining module 61 is used to determine damaged video frames in the video encoded bitstream. The processing module 62 is used to, when the damaged video frame is intra-frame coded, to perform error hiding on the damaged macroblocks according to their macroblock types, including flat blocks, edge blocks, and texture blocks; or, when the damaged video frame is inter-frame coded, to perform error hiding on the damaged macroblocks according to the macroblock state of a target macroblock; the target macroblock includes any one of the following: macroblocks adjacent to the damaged macroblocks in the damaged video frame; macroblocks adjacent to the damaged macroblocks in the damaged video frame; and macroblocks in a reference frame of the damaged video frame.

[0263] In one possible implementation, the processing module 62 can also be used to perform edge detection on the damaged macroblock before incorrectly hiding the damaged macroblock according to the macroblock type of the damaged macroblock in the damaged video frame, and determine the macroblock type of the damaged macroblock based on the edge information of the edge detection.

[0264] In one possible implementation, the processing module 62 may include a detection submodule and a determination submodule. The detection submodule may be used to detect the gradient intensity of pixels in at least one adjacent region of the damaged macroblock. The determination submodule may be used to determine a flatness threshold for the damaged macroblock based on the gradient intensity detected by the detection submodule, and if the flatness of the damaged macroblock is less than the flatness threshold, determine the macroblock type of the damaged macroblock as a flat block.

[0265] In one possible implementation, the detection submodule described above can also be used to detect the texture complexity of the damaged macroblock if the flatness of the damaged macroblock is greater than or equal to a flatness threshold. The determination submodule described above can also be used to determine the macroblock type of the damaged macroblock based on the texture complexity detected by the detection submodule.

[0266] In one possible implementation, the detection submodule may include a first acquisition submodule and a first determination submodule. The first acquisition submodule is used to acquire pixel gradient intensities along at least one pixel prediction direction, where each pixel prediction direction corresponds to one pixel gradient intensity, and the pixel gradient intensity is the sum of the gradient intensities of pixels within the angle range corresponding to the pixel prediction direction. The first determination submodule is used to determine the texture complexity of the damaged macroblock based on the accumulated gradient intensities and the maximum gradient intensity among the at least one pixel gradient intensities acquired by the first acquisition submodule; wherein the accumulated gradient intensities are the sum of the at least one pixel gradient intensities.

[0267] In one possible implementation, the processing module 62 may further include an update submodule. The aforementioned determining submodule may also be used to determine M pixel adjustment amounts corresponding to the damaged pixel based on the M prediction directions and the position information of the damaged pixel in the damaged macroblock, where each prediction direction corresponds to one pixel adjustment amount, and M is an integer greater than 2. The update submodule may be used to update the pixel value of the damaged pixel based on the M pixel adjustment amounts determined by the determining submodule.

[0268] In one possible implementation, the determining submodule can also be used to determine the pixel adjustment weight corresponding to each prediction direction based on each prediction direction before the updating submodule updates the pixel value of the damaged pixel based on the M pixel adjustment amounts. Specifically, the updating submodule can be used to update the pixel value of the damaged pixel based on the M pixel adjustment amounts and the M pixel adjustment weights determined by the determining submodule, wherein the M pixel adjustment amounts and the M pixel adjustment weights correspond one-to-one.

[0269] In one possible implementation, the aforementioned determining submodule can be specifically used to determine the pixel adjustment weight corresponding to each prediction direction based on the cumulative gradient intensity corresponding to each prediction direction; the aforementioned determining submodule can also be used to determine the first boundary pixel and the second boundary pixel corresponding to each prediction direction based on each prediction direction and the location information of the damaged macroblock, and to determine the pixel adjustment weight corresponding to each prediction direction based on the pixel value and location information of the determined first boundary pixel and the second boundary pixel corresponding to each prediction direction.

[0270] In one possible implementation, the processing module 62 can be used to perform error hiding of the damaged macroblock by using bilinear interpolation when the macroblock type of the damaged macroblock is a flat block; or, specifically, it can be used to perform error hiding of the damaged macroblock by using unidirectional interpolation when the macroblock type of the damaged macroblock is an edge block.

[0271] In one possible implementation, processing module 62 may further include a processing submodule. The aforementioned determining submodule may also be used to determine a first candidate motion vector set for the damaged macroblock based on a first quantity or a second quantity, where the first quantity is the number of macroblocks in the damaged video frame that are adjacent to and correctly received and the second quantity is the number of damaged macroblocks in the damaged video frame that are adjacent to and damaged. The processing submodule may be used to perform error hiding of the damaged macroblock based on the first candidate motion vector set determined by the determining submodule; wherein each motion vector in the first candidate motion vector set indicates a macroblock in a reference video frame.

[0272] In one possible implementation, if the first quantity is greater than or equal to a first preset quantity, or the second quantity is less than or equal to a second preset quantity, the first candidate motion vector set includes the second candidate motion vector set; or, if the first quantity is less than the first preset quantity, or the second quantity is greater than the second preset quantity, the first candidate motion vector set includes the second candidate motion vector set and the third candidate motion vector set; wherein the second candidate motion vector set includes: motion vectors of macroblocks adjacent to and correctly received macroblocks in the damaged video frame, zero motion vectors, and median motion vectors of motion vectors of macroblocks adjacent to and correctly received macroblocks in the damaged video frame; the third candidate motion vector set includes: motion vectors of macroblocks in the reference frame of the damaged video frame that are at the same position as the damaged macroblock.

[0273] In one possible implementation, the aforementioned processing submodule may include a second determining submodule and a first processing submodule. The second determining submodule may be used to determine the candidate motion vector that minimizes the loss cost function value from the first candidate motion vector set as the target motion vector, and to determine the motion compensation block of the target motion vector in the reference frame of the damaged video frame. The first processing submodule may be used to perform error hiding on the damaged macroblock based on the motion compensation block determined by the second determining submodule; wherein, the loss cost function D... tot Represented as: D tot =α1·D BMA +α2·D OBMA ; where D BMA D is the boundary matching cost function. OBMA The cost function is the overlapping boundary matching function, and α1 and α2 are preset weights.

[0274] In the video frame error hiding device provided in this application embodiment, for intra-frame coded damaged video frames, different methods can be used to hide damaged macroblocks of different macroblock types within the damaged video frame; for inter-frame coded damaged video frames, the damaged macroblocks can be hidden based on macroblocks adjacent to the damaged macroblocks in the damaged video frame, or macroblocks in the reference frame of the damaged video frame and the adjacent macroblocks. Therefore, compared to related technologies that use a single error hiding algorithm to hide damaged macroblocks for both intra-frame and inter-frame coded damaged video frames, the video frame error hiding device provided in this application embodiment can more accurately hide each damaged macroblock. This improves the effectiveness of error hiding of damaged macroblocks in damaged video frames.

[0275] The video frame error hiding device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0276] The video frame error hiding device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0277] The video frame error concealment device provided in this application embodiment can achieve... Figures 1 to 5 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0278] Optionally, such as Figure 7 As shown, this application embodiment also provides an electronic device 700, including a processor 701 and a memory 702. The memory 702 stores a program or instructions that can run on the processor 701. When the program or instructions are executed by the processor 701, they implement the various steps of the above-described video error hiding method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0279] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0280] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0281] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.

[0282] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0283] The processor 1010 can be used to determine the damaged video frame in the video encoded bitstream, and when the damaged video frame is intra-coded, to perform error hiding on the damaged macroblock according to the macroblock type of the damaged macroblock in the damaged video frame. The macroblock type includes: flat block, edge block, and texture block; or, when the damaged video frame is inter-coded, to perform error hiding on the damaged macroblock according to the macroblock state of the target macroblock. The target macroblock includes any one of the following: macroblocks adjacent to the damaged macroblock in the damaged video frame; macroblocks adjacent to the damaged macroblock in the damaged video frame; and macroblocks in the reference frame of the damaged video frame.

[0284] In one possible implementation, the processor 1010 can also be used to perform edge detection on the damaged macroblock before performing error hiding on the damaged macroblock according to the macroblock type of the damaged macroblock in the damaged video frame, and determine the macroblock type of the damaged macroblock based on the edge information of the edge detection.

[0285] In one possible implementation, the processor 1010 can specifically be used to detect the gradient intensity of pixels in at least one adjacent region of the damaged macroblock, and determine the flatness threshold of the damaged macroblock based on the detected gradient intensity, and determine the macroblock type of the damaged macroblock as a flat block if the flatness of the damaged macroblock is less than the flatness threshold.

[0286] In one possible implementation, the processor 1010 can also be used to detect the texture complexity of a damaged macroblock when the flatness of the damaged macroblock is greater than or equal to a flatness threshold, and determine the macroblock type of the damaged macroblock based on the detected texture complexity.

[0287] In one possible implementation, the processor 1010 can be specifically used to obtain the pixel gradient intensity of at least one pixel prediction direction, each pixel prediction direction corresponds to a pixel gradient intensity, and the pixel gradient intensity is the sum of the gradient intensities of pixels within the angle range corresponding to the pixel prediction direction; and determine the texture complexity of the damaged macroblock based on the cumulative gradient intensity and the maximum gradient intensity among the at least one pixel gradient intensities obtained; wherein the cumulative gradient intensity is the sum of the at least one pixel gradient intensities.

[0288] In one possible implementation, the processor 1010 can also be used to determine M pixel adjustment amounts corresponding to the damaged pixel based on M prediction directions and the position information of the damaged pixel in the damaged macroblock, where each prediction direction corresponds to one pixel adjustment amount, M is an integer greater than 2, and update the pixel value of the damaged pixel based on the determined M pixel adjustment amounts.

[0289] In one possible implementation, the processor 1010 can also be used to determine the pixel adjustment weight corresponding to each prediction direction based on each prediction direction before updating the pixel value of the damaged pixel based on the above M pixel adjustment amounts, and update the pixel value of the damaged pixel based on the M pixel adjustment amounts and the determined M pixel adjustment weights, wherein the M pixel adjustment amounts and the M pixel adjustment weights correspond one-to-one.

[0290] In one possible implementation, the processor 1010 can be specifically used to determine the pixel adjustment weight corresponding to each prediction direction based on the cumulative gradient intensity corresponding to each prediction direction; the processor 1010 can also be used to determine the first boundary pixel and the second boundary pixel corresponding to each prediction direction according to each prediction direction and the location information of the damaged macroblock, and determine the pixel adjustment weight corresponding to each prediction direction based on the pixel value and location information of the determined first boundary pixel and the second boundary pixel corresponding to each prediction direction.

[0291] In one possible implementation, the processor 1010 can be used to perform error hiding of the damaged macroblock by using bilinear interpolation when the macroblock type of the damaged macroblock is a flat block; or, specifically, it can be used to perform error hiding of the damaged macroblock by using unidirectional interpolation when the macroblock type of the damaged macroblock is an edge block.

[0292] In one possible implementation, the processor 1010 can also be used to determine a first candidate motion vector set for the damaged macroblock based on a first quantity or a second quantity, wherein the first quantity is the number of macroblocks in the damaged video frame that are adjacent to the damaged macroblock and are correctly received, and the second quantity is the number of damaged macroblocks in the damaged video frame that are adjacent to the damaged macroblock; and to perform error hiding on the damaged macroblock based on the determined first candidate motion vector set; wherein each motion vector in the first candidate motion vector set indicates a macroblock in a reference video frame.

[0293] In one possible implementation, if the first quantity is greater than or equal to a first preset quantity, or the second quantity is less than or equal to a second preset quantity, the first candidate motion vector set includes the second candidate motion vector set; or, if the first quantity is less than the first preset quantity, or the second quantity is greater than the second preset quantity, the first candidate motion vector set includes the second candidate motion vector set and the third candidate motion vector set; wherein the second candidate motion vector set includes: motion vectors of macroblocks adjacent to and correctly received macroblocks in the damaged video frame, zero motion vectors, and median motion vectors of motion vectors of macroblocks adjacent to and correctly received macroblocks in the damaged video frame; the third candidate motion vector set includes: motion vectors of macroblocks in the reference frame of the damaged video frame that are at the same position as the damaged macroblock.

[0294] In one possible implementation, the processor 1010 can further be used to determine the candidate motion vector that minimizes the loss cost function value from the first candidate motion vector set as the target motion vector, and to determine the motion compensation block of the target motion vector in the reference frame of the damaged video frame, and to perform error hiding on the damaged macroblock based on the determined motion compensation block; wherein, the loss cost function D tot Represented as: D tot =α1·D BMA +α2·D OBMA ; where D BMA D is the boundary matching cost function. OBMA The cost function is the overlapping boundary matching function, and α1 and α2 are preset weights.

[0295] In the electronic device provided in this application embodiment, for intra-frame coded damaged video frames, different methods can be used to error-hide damaged macroblocks of different macroblock types within the damaged video frame; for inter-frame coded damaged video frames, the damaged macroblocks can be error-hide based on adjacent macroblocks in the damaged video frame, or the adjacent macroblocks and macroblocks in the reference frame of the damaged video frame. Therefore, compared to related technologies that use a single error-hide algorithm to error-hide damaged macroblocks for both intra-frame and inter-frame coded damaged video frames, the electronic device provided in this application embodiment can more accurately error-hide each damaged macroblock. This improves the effectiveness of error-hiders for damaged macroblocks in damaged video frames.

[0296] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0297] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0298] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0299] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described video frame error hiding method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0300] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0301] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described video frame error hiding method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0302] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0303] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the video frame error hiding method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0304] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0305] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0306] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for concealing video frame errors, characterized in that, The method includes: Identify damaged video frames in the video encoded bitstream; When the damaged video frame is intra-coded, the damaged macroblocks are incorrectly hidden according to their macroblock types, including flat blocks, edge blocks, and texture blocks. When the damaged video frame is inter-frame coded, the damaged macroblock is error-hidden according to the macroblock state of the target macroblock, where the macroblock state includes: correctly received or damaged. The target macroblock includes any of the following: The macroblocks adjacent to the damaged macroblock in the damaged video frame; The macroblocks adjacent to the damaged macroblock in the damaged video frame, and the macroblocks in the reference frame of the damaged video frame; Before incorrectly hiding the damaged macroblocks based on their macroblock type in the damaged video frame, the method further includes: Edge detection is performed on the damaged macroblock, and the macroblock type of the damaged macroblock is determined based on the edge information detected by the edge detection.

2. The method according to claim 1, characterized in that, The step of performing edge detection on the damaged macroblock and determining the macroblock type of the damaged macroblock based on the edge information detected includes: Detect the gradient intensity of pixels in at least one adjacent region of the damaged macroblock; Based on the gradient intensity, determine the flatness threshold of the damaged macroblock; If the flatness of the damaged macroblock is less than the flatness threshold, the macroblock type of the damaged macroblock is determined to be a flat block.

3. The method according to claim 2, characterized in that, If the flatness of the damaged macroblock is greater than or equal to the flatness threshold, the texture complexity of the damaged macroblock is detected. The macroblock type of the damaged macroblock is determined based on the texture complexity.

4. The method according to claim 3, characterized in that, The detection of the texture complexity of the damaged macroblock includes: Obtain the pixel gradient intensity of at least one pixel prediction direction, each pixel prediction direction corresponds to a pixel gradient intensity, and the pixel gradient intensity is the sum of the gradient intensities of pixels within the angle range corresponding to the pixel prediction direction; The texture complexity of the damaged macroblock is determined based on the cumulative gradient intensity and the maximum gradient intensity among at least one of the pixel gradient intensities. The cumulative gradient intensity is the sum of the gradient intensities of at least one pixel.

5. The method according to claim 1, characterized in that, When the macroblock type of the damaged macroblock is the texture block, the step of error-hiding the damaged macroblock according to the macroblock type in the damaged video frame includes: Based on the M prediction directions and the location information of the damaged pixels in the damaged macroblock, the M pixel adjustment amounts corresponding to the damaged pixels are determined, where each prediction direction corresponds to one pixel adjustment amount, and M is an integer greater than 2; Based on the M pixel adjustment amounts, update the pixel value of the damaged pixel.

6. The method according to claim 5, characterized in that, Before updating the pixel value of the damaged pixel based on the M pixel adjustment amounts, the method further includes: Based on each prediction direction, determine the pixel adjustment weight corresponding to each prediction direction; The step of updating the pixel value of the damaged pixel based on the M pixel adjustment amounts includes: Based on the M pixel adjustment amounts and the M pixel adjustment weights, the pixel value of the damaged pixel is updated, and the M pixel adjustment amounts and the M pixel adjustment weights correspond one-to-one.

7. The method according to claim 6, characterized in that, The step of determining the pixel adjustment weight corresponding to each prediction direction based on each prediction direction includes at least one of the following: Based on the cumulative gradient intensity corresponding to each prediction direction, the pixel adjustment weight corresponding to each prediction direction is determined respectively; Based on each prediction direction and the location information of the damaged macroblock, determine the first boundary pixel and the second boundary pixel corresponding to each prediction direction; Based on the pixel values ​​and position information of the first boundary pixel and the second boundary pixel corresponding to each prediction direction, the pixel adjustment weight corresponding to each prediction direction is determined.

8. The method according to claim 1, characterized in that, The step of error-hiding the damaged macroblocks based on their macroblock type in the damaged video frame includes: When the macroblock type of the damaged macroblock is the flat block, bilinear interpolation is used to hide the errors in the damaged macroblock. When the macroblock type of the damaged macroblock is the edge block, a one-way interpolation method is used to erroneously hide the damaged macroblock.

9. The method according to claim 1, characterized in that, The step of incorrectly hiding the damaged macroblock based on the macroblock state of the target macroblock includes: Based on a first quantity or a second quantity, a first candidate motion vector set for the damaged macroblock is determined, wherein the first quantity is the number of macroblocks in the damaged video frame that are adjacent to the damaged macroblock and are correctly received, and the second quantity is the number of damaged macroblocks in the damaged video frame that are adjacent to the damaged macroblock; Based on the first candidate motion vector set, the damaged macroblock is incorrectly hidden; Each motion vector in the first candidate motion vector set indicates a macroblock in a reference video frame.

10. The method according to claim 9, characterized in that, When the first quantity is greater than or equal to the first preset quantity, or the second quantity is less than or equal to the second preset quantity, the first candidate motion vector set includes the second candidate motion vector set; or, If the first quantity is less than the first preset quantity, or the second quantity is greater than the second preset quantity, the first candidate motion vector set includes the second candidate motion vector set and the third candidate motion vector set. The second candidate motion vector set includes: motion vectors of macroblocks that are adjacent to and correctly received from the damaged macroblock in the damaged video frame, zero motion vectors, and median motion vectors of motion vectors of macroblocks that are adjacent to and correctly received from the damaged macroblock in the damaged video frame. The third candidate motion vector set includes the motion vectors of macroblocks in the reference frame of the damaged video frame that are at the same position as the damaged macroblock.

11. The method according to claim 9 or 10, characterized in that, The step of error-hiding the damaged macroblock based on the first candidate motion vector set includes: The candidate motion vector that minimizes the loss cost function value in the first set of candidate motion vectors is determined as the target motion vector. Determine the motion compensation block of the target motion vector in the reference frame of the damaged video frame; The damaged macroblock is incorrectly hidden according to the motion compensation block; Wherein, the loss cost function D tot Represented as: D tot =α1·D BMA +α2·D OBMA ; Among them, D BMA D is the boundary matching cost function. OBMA The cost function is the overlapping boundary matching function, and α1 and α2 are preset weights.

12. A video frame error concealment device, characterized in that, The device includes a determining module and a processing module; The determining module is used to determine damaged video frames in the video encoded bitstream; The processing module is configured to, when the damaged video frame is intra-coded, perform error hiding on the damaged macroblocks according to the macroblock type of the damaged macroblocks in the damaged video frame, wherein the macroblock type includes: flat blocks, edge blocks, and texture blocks; or, Used to perform error hiding of the damaged macroblock based on the macroblock state of the target macroblock when the damaged video frame is inter-frame coded, wherein the macroblock state includes: correctly received or damaged; The target macroblock includes any of the following: The macroblocks adjacent to the damaged macroblock in the damaged video frame; The macroblocks adjacent to the damaged macroblock in the damaged video frame, and the macroblocks in the reference frame of the damaged video frame; The processing module is further configured to perform edge detection on the damaged macroblock before incorrectly hiding the damaged macroblock according to the macroblock type of the damaged macroblock in the damaged video frame, and determine the macroblock type of the damaged macroblock according to the edge information of the edge detection.

13. The apparatus according to claim 12, characterized in that, The processing module includes a detection submodule and a determination submodule; The detection submodule is used to detect the gradient intensity of pixels in at least one adjacent region of the damaged macroblock; The determining submodule is used to determine the flatness threshold of the damaged macroblock based on the gradient intensity detected by the detection submodule, and to determine the macroblock type of the damaged macroblock as a flat block if the flatness of the damaged macroblock is less than the flatness threshold.

14. The apparatus according to claim 12, characterized in that, The processing module includes a determination submodule and a processing submodule; The determining submodule is used to determine a first candidate motion vector set of the damaged macroblock based on a first quantity or a second quantity, wherein the first quantity is the number of macroblocks in the damaged video frame that are adjacent to the damaged macroblock and are correctly received, and the second quantity is the number of damaged macroblocks in the damaged video frame that are adjacent to the damaged macroblock; The processing submodule is used to perform error hiding on the damaged macroblock based on the first candidate motion vector set determined by the determining submodule; Each motion vector in the first candidate motion vector set indicates a macroblock in a reference video frame.

15. The apparatus according to claim 14, characterized in that, When the first quantity is greater than or equal to the first preset quantity, or the second quantity is less than or equal to the second preset quantity, the first candidate motion vector set includes the second candidate motion vector set; or, If the first quantity is less than the first preset quantity, or the second quantity is greater than the second preset quantity, the first candidate motion vector set includes the second candidate motion vector set and the third candidate motion vector set. The second candidate motion vector set includes: motion vectors of macroblocks that are adjacent to and correctly received from the damaged macroblock in the damaged video frame, zero motion vectors, and median motion vectors of motion vectors of macroblocks that are adjacent to and correctly received from the damaged macroblock in the damaged video frame. The third candidate motion vector set includes the motion vectors of macroblocks in the reference frame of the damaged video frame that are at the same position as the damaged macroblock.

16. The apparatus according to claim 14 or 15, characterized in that, The processing submodule includes a second determining submodule and a first processing submodule; The second determining submodule is used to determine the candidate motion vector that minimizes the loss cost function value in the first candidate motion vector set as the target motion vector, and to determine the motion compensation block of the target motion vector in the reference frame of the damaged video frame; The first processing submodule is used to perform error hiding of the damaged macroblock according to the motion compensation block determined by the second determining submodule; Wherein, the loss cost function D tot Represented as: D tot =α1·D BMA +α2·D OBMA ; Among them, D BMA D is the boundary matching cost function. OBMA The cost function is the overlapping boundary matching function, and α1 and α2 are preset weights.

17. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the video frame error hiding method as described in any one of claims 1-11.

18. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the video frame error hiding method as described in any one of claims 1-11.

Images