A method for repairing image quality degradation caused by packet loss in real-time video communication

Abstract

The present invention discloses a method for repairing image quality degradation caused by packet loss in real-time video communication, comprising: step S1, decoding a successfully received data packet, recording the decoded image as the current frame, and marking the macroblock area of ​​the current frame where data is lost as the target area; step S2, marking the neighboring areas of the target area based on the target area; step S3, performing intra-frame prediction on the current frame to obtain intra-frame predicted pixel values ​​and intra-frame confidence of the target area; step S4, performing inter-frame prediction on the current frame to obtain inter-frame predicted pixel values ​​and inter-frame confidence of the target area; step S5, performing fusion repair, calculating the repaired pixels and repair confidence of the target area, and filling each repaired pixel into the target area to obtain a repaired image. The present invention can effectively improve the effects of intra-frame prediction and inter-frame prediction by using known macroblock information for comprehensive prediction to repair packet loss errors.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a method for repairing image quality degradation caused by packet loss in real-time video communication. Background Art

[0002] As we all know, current communication technology is mainly based on data transmission. The speed and quality of data transmission have a direct impact on the actual communication effect between user terminals. For example, in common real-time video communication, packet loss may occur during the data transmission due to various factors, resulting in a decrease in the video quality of the user terminal, resulting in a poor user experience for the real-time video communication effect on the user terminal. To address this problem, existing conventional approaches and their corresponding drawbacks include: Solution 1: Filling the missing macroblock area with the mean or median of the surrounding pixels within the frame. This approach directly uses the mean or median to fill pixels, resulting in graininess in colorful areas and a significant degradation in image quality. Solution 2: Filling the missing macroblock area with the pixel values ​​of the same position in adjacent frames. This approach works well for static or slowly changing areas in the video, but results in significant distortion for fast-moving content, significantly degrading image quality. Solution 3: Filling the missing area with pixels by weighting the intra-frame and inter-frame prediction results. This approach, by weighting the intra-frame and inter-frame prediction results, can address their respective shortcomings to a certain extent, but is constrained by the intra-frame and inter-frame prediction results. Furthermore, there is no universally applicable method for selecting the weighting coefficients. Therefore, it is necessary to research and design new systems and methods to improve the image quality degradation caused by packet loss in real-time video communication. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method for repairing image quality degradation caused by packet loss in real-time video communication, comprising the following steps:

[0004] Step S1: Decode the successfully received data packet, record the decoded image as the current frame, and mark the macroblock area of ​​the current frame where data is lost as the target area;

[0005] Step S2, marking neighboring areas of the target area based on the target area;

[0006] Step S3, performing intra-frame prediction on the current frame, predicting the pixel value and pixel confidence in each target area using the pixel values ​​and pixel confidences of the neighboring areas, and calculating the intra-frame predicted pixel value and intra-frame confidence of the target area;

[0007] Step S4, performing inter-frame prediction on the current frame, using the previous frame image of the current frame as a reference frame, recording the region in the reference frame at the same position as the target region in the current frame as a reference target region, performing motion prediction on the reference target region to obtain a motion vector of the reference target region, predicting pixel values ​​and pixel confidence levels in each target region based on information of the reference frame and information of the motion vector of the reference target region, and calculating inter-frame predicted pixel values ​​and inter-frame confidence levels of the target region;

[0008] Step S5, fusion restoration, performs fusion calculation based on the intra-frame predicted pixel value, intra-frame confidence, inter-frame predicted pixel value and inter-frame confidence of the target area to obtain the restored pixels and restoration confidence of the target area, and fills each restored pixel into the target area to obtain the restored image, and uses the restored image as the final output of the restoration algorithm.

[0009] As a further explanation of the present invention, the neighboring area is a number of peripheral macroblocks adjacent to the target area.

[0010] Furthermore, the pixel confidence is the probability that the pixel value is a true value, and the pixel confidence of the pixels in the non-lost macroblock area is 1.

[0011] Furthermore, in step S3, intra-frame prediction is performed on the target region according to the average confidence values ​​of neighboring regions from high to low.

[0012] Furthermore, if a neighboring region is a non-lost macroblock, the neighboring region confidence of the region is 1;

[0013] If a neighboring region is a lost macroblock, ie, a target region, the neighboring region confidence of the region is the average intra-frame confidence of all pixels in the region.

[0014] Furthermore, during the intra-frame prediction of the region, the prediction calculation method of any point T in the target region is:

[0015] Intra-predicted pixel value at point T = (L2*Nb1*C1+L1*Nb2*C2) / ((Nb1+Nb2)(L1+L2));

[0016] Intra-frame confidence at point T = (L2*Nb1+L1*Nb2) / (L1+L2);

[0017] Among them, the areas of the four upper, lower, left and right neighboring areas of the target area are recorded as Q1, Q2, Q3, and Q4, and their neighbor confidences are recorded as N1, N2, N3, and N4 respectively; compare the sizes of N1+N3 and N2+N4, and the neighbor areas corresponding to the larger ones are recorded as Qb1 and Qb2, and the corresponding neighbor confidences are Nb1 and Nb2; the pixels on the edges where Qb1 and Qb2 intersect with the target area are recorded as E1 and E2; the pixel values ​​of the perpendicular line from T to E1 and E2 and the intersection of E1 and E2 are recorded as C1 and C2; the distance from T to C1 is recorded as L1, and the distance from T to C2 is recorded as L2.

[0018] Furthermore, the step of obtaining the motion vector by motion prediction in step S4 includes:

[0019] 1) The area to be motion predicted is denoted as T, the area is a square, and its side length is denoted as L;

[0020] 2) The predicted motion vector is denoted as V, where V is a two-dimensional vector with the horizontal dimension denoted as Vx and the vertical dimension denoted as Vy;

[0021] 3) Any point P in the region T has its coordinates denoted as (Px, Py) and its pixel value as Sp;

[0022] 4) The motion prediction corresponding point of P is Pv in the reference frame, its coordinates are marked as (Px-Vx, Py-Vy), and its pixel value in the reference frame is recorded as Spv;

[0023] 5) The regional prediction error of T is denoted as E. For any point P in T, E = Σ(Sp–Spv)^2;

[0024] 6) For different V values, i.e. different Vx and Vy, different E can be calculated;

[0025] 7) Traverse all Vx and Vy between -L and L and calculate the minimum E value, which is recorded as Emin;

[0026] 8) Vx and Vy when E is equal to Emin are denoted as Vbx and Vby, and the corresponding motion vector is denoted as Vb. Then Vb is the motion vector output by the motion prediction of region T.

[0027] Furthermore, the step of performing the inter-frame prediction in step S4 includes:

[0028] 1) The target region is denoted as T, and the motion vector obtained by motion prediction of the reference target region corresponding to the target region in the reference frame is denoted as Vb, and the coordinates are denoted as (Vbx, Vby);

[0029] 2) Any point P in the target area T, its coordinates are marked as (Px, Py);

[0030] 3) In the reference frame, the point corresponding to the motion of point P is denoted as Pv. The coordinates of Pv are (Px-Vbx, Py-Vby), the pixel value of Pv is denoted as Svf, and the confidence of Pv is denoted as Nvf.

[0031] If Pv is a point in the lost macroblock, the pixel value of Pv uses its restored pixel value, and the confidence uses its restored confidence

[0032] If Pv is a point in a non-lost macroblock, the pixel value of Pv uses its original pixel value and the confidence level uses 1.

[0033] 4) Use the pixel value and confidence of point Pv as the inter-frame prediction result of point P, that is, the inter-frame prediction result of point P is:

[0034] Point P inter-frame prediction pixel value = Svf

[0035] Inter-frame confidence at point P = Nvf

[0036] 5) Traverse all pixels in the target area and calculate their inter-frame predicted pixel values ​​and inter-frame confidence.

[0037] Furthermore, the step of performing fusion repair in step S5 includes:

[0038] 1) The target area is denoted as T, with any point P in it;

[0039] 2) The intra-frame prediction pixel value of point P is recorded as Si, the intra-frame confidence of point P is recorded as Ni, the inter-frame prediction pixel value of point P is recorded as So, and the inter-frame confidence of point P is recorded as No;

[0040] 3) The repair result of P is:

[0041] The restored pixel value of point P = (Si*Ni+So*No) / (Ni+No);

[0042] Repair confidence of point P = (Ni+No–Ni*No))*(Ni+No-abs(Ni-No)) / (Ni+No);

[0043] 4) Traversing all pixels in the target area and calculating their restored pixel values ​​and restoration confidence;

[0044] 5) Traverse all target areas to obtain the repaired image.

[0045] Beneficial effects of the present invention:

[0046] The present invention can effectively improve the effects of intra-frame prediction and inter-frame prediction by using known macroblock information for comprehensive prediction to repair packet loss errors, and provides a weighting coefficient calculation method that dynamically decides based on the packet loss situation to obtain better weighting effects.

[0047] 1. More effective use of intra-frame information to provide better intra-frame prediction results;

[0048] 2. Inter-frame prediction recursively uses repaired pixel information to further improve information utilization efficiency;

[0049] 3. Each calculation process retains confidence information, so that the weight of information is more reasonably distributed in the prediction;

[0050] 4. The final restoration fuses the intra-frame results and inter-frame results according to the confidence level, making the predicted information utilization higher. The confidence level of the fused restoration results is evaluated to ensure that more accurate information can be obtained in subsequent inter-frame predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of a method for repairing image quality degradation caused by packet loss in real-time video communication according to the present invention;

[0052] Figure 2 Schematic diagram of the principle of intra-frame prediction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following is a detailed description of the embodiments of the present invention with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0054] In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", "first", "second", etc., indicating directions or positions or sequential relationships, are based on the directions or positions or sequential relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0055] A method for repairing image quality degradation caused by packet loss in real-time video communication is invented, which specifically includes the following steps:

[0056] Step S1: decode the successfully received data packet, record the decoded image as the current frame, and mark the macroblock area of ​​the current frame where data is lost as the target area.

[0057] Step S2, based on the target area, mark the neighboring area of ​​the target area; the neighboring area here refers to the macroblock area adjacent to the target area. As shown in the accompanying drawings, in this embodiment, the four neighboring areas of the upper, lower, left and right of the target area are defined as Q1, Q2, Q3, and Q4 respectively.

[0058] Step S3, intra-frame prediction is performed on the current frame, and the pixel values ​​and pixel confidences in each target area are predicted by the pixel values ​​and pixel confidences of the neighboring areas, and the intra-frame predicted pixel values ​​and intra-frame confidences of the target area are calculated. The pixel confidence here is used to represent the probability that the pixel value is a true value. For pixels in the non-lost macroblock area, the pixel confidence is 1. In the above-mentioned intra-frame prediction process, this embodiment uses the concept of neighbor area confidence. Specifically, if a neighbor area is a non-lost macroblock, the neighbor area confidence of the area is 1; if a neighbor area is a lost macroblock, that is, the target area, the neighbor area confidence of the area is the average intra-frame confidence of all pixels in the area.

[0059] Referring to the accompanying drawings, the specific steps of the method for performing the above intra-frame prediction in this embodiment are as follows:

[0060] 1) The areas of the four neighboring areas above, below, left and right of the target area are recorded as Q1, Q2, Q3, and Q4 respectively, and their neighbor confidence levels are recorded as N1, N2, N3, and N4 respectively;

[0061] 2) Performing intra-frame prediction on the target region according to the average confidence values ​​of the neighboring regions from high to low. In this embodiment, N1+N3 and N2+N4 are specifically compared, and the neighboring regions corresponding to the larger ones are recorded as Qb1 and Qb2, and the corresponding neighboring confidence values ​​are Nb1 and Nb2;

[0062] 3) The pixels on the edges where Qb1 and Qb2 intersect the target area are denoted as E1 and E2;

[0063] 4) The pixel values ​​of the intersection of the perpendicular line from any point T to E1 and E2 and E1 and E2 are recorded as C1 and C2, the distance from point T to C1 is recorded as L1, and the distance from point T to C2 is recorded as L2;

[0064] 5) The prediction result of point T is:

[0065] Intra-frame prediction pixel value at point T = (L2*Nb1*C1+L1*Nb2*C2) / ((Nb1+Nb2)(L1+L2))

[0066] Confidence within the frame at point T = (L2*Nb1+L1*Nb2) / (L1+L2);

[0067] 6) Traverse all pixels in the target area and calculate their intra-frame predicted pixel values ​​and intra-frame confidence.

[0068] Step S4, perform inter-frame prediction on the current frame, use the previous frame image of the current frame as a reference frame (there is no reference frame in the first frame), and record the area in the reference frame that has the same position as the target area in the current frame as the reference target area. Perform motion prediction on the reference target area to obtain the motion vector of the reference target area, predict the pixel value and pixel confidence in each target area based on the information of the reference frame and the motion vector information of the reference target area, and calculate the inter-frame predicted pixel value and inter-frame confidence of the target area.

[0069] In this embodiment, the steps of obtaining the motion vector by motion prediction are specifically as follows:

[0070] 1) The area to be motion predicted is denoted as T, the area is a square, and its side length is denoted as L;

[0071] 2) The predicted motion vector is denoted as V, where V is a two-dimensional vector with the horizontal dimension denoted as Vx and the vertical dimension denoted as Vy;

[0072] 3) Any point P in the region T has its coordinates denoted as (Px, Py) and its pixel value as Sp;

[0073] 4) The motion prediction corresponding point of P is Pv in the reference frame, its coordinates are marked as (Px-Vx, Py-Vy), and its pixel value in the reference frame is recorded as Spv;

[0074] 5) The regional prediction error of T is denoted as E. For any point P in T, E = Σ(Sp–Spv)^2;

[0075] 6) For different V values, i.e. different Vx and Vy, different E can be calculated;

[0076] 7) Traverse all Vx and Vy between -L and L and calculate the minimum E value, which is recorded as Emin;

[0077] 8) Vx and Vy when E is equal to Emin are denoted as Vbx and Vby, and the corresponding motion vector is denoted as Vb. Then Vb is the motion vector output by the motion prediction of region T.

[0078] After the motion vector Vb of the reference target region is predicted, the next step is to perform inter-region prediction, which includes the following steps:

[0079] 1) The target region is denoted as T, and the motion vector obtained by motion prediction of the reference target region corresponding to the target region in the reference frame is denoted as Vb, and the coordinates are denoted as (Vbx, Vby);

[0080] 2) Any point P in the target area T, its coordinates are marked as (Px, Py);

[0081] 3) In the reference frame, the point corresponding to the motion of point P is denoted as Pv. The coordinates of Pv are (Px-Vbx, Py-Vby), the pixel value of Pv is denoted as Svf, and the confidence of Pv is denoted as Nvf.

[0082] If Pv is a point in the lost macroblock, the pixel value of Pv uses its restored pixel value, and the confidence uses its restored confidence

[0083] If Pv is a point in a non-lost macroblock, the pixel value of Pv uses its original pixel value and the confidence level uses 1.

[0084] 4) Use the pixel value and confidence of point Pv as the inter-frame prediction result of point P, that is, the inter-frame prediction result of point P is:

[0085] Point P inter-frame prediction pixel value = Svf

[0086] Inter-frame confidence at point P = Nvf

[0087] 5) Traverse all pixels in the target area and calculate their inter-frame predicted pixel values ​​and inter-frame confidence.

[0088] Step S5, fusion restoration, performs fusion calculation based on the intra-frame predicted pixel values, intra-frame confidence, inter-frame predicted pixel values ​​and inter-frame confidence of the target area to obtain the restored pixels and restoration confidence of the target area. The specific method steps of fusion restoration in this embodiment include:

[0089] 1) The target area is denoted as T, with any point P in it;

[0090] 2) The intra-frame prediction pixel value of point P is recorded as Si, the intra-frame confidence of point P is recorded as Ni, the inter-frame prediction pixel value of point P is recorded as So, and the inter-frame confidence of point P is recorded as No;

[0091] 3) The repair result of P is:

[0092] The restored pixel value of point P = (Si*Ni+So*No) / (Ni+No);

[0093] Repair confidence of point P = (Ni+No–Ni*No))*(Ni+No-abs(Ni-No)) / (Ni+No);

[0094] 4) Traversing all pixels in the target area and calculating their restored pixel values ​​and restoration confidence;

[0095] 5) Traverse all target areas and fill each restored pixel into the target area to obtain a restored image, which is used as the final output of the restoration algorithm.

[0096] The present invention can effectively improve the effects of intra-frame prediction and inter-frame prediction by using known macroblock information for comprehensive prediction to repair packet loss errors, and provides a weighting coefficient calculation method that dynamically decides based on the packet loss situation to obtain better weighting effects.

[0097] The above description is merely an explanation of the preferred embodiments of the present invention and should not be construed as limiting the claims. The present invention is not limited to the above embodiments, and variations in the specific structure are permitted. In short, all variations made within the scope of the independent claims of the present invention are also within the scope of protection of the present invention.

Claims

1. A method for repairing image quality degradation caused by packet loss in real-time video communication, characterized in that: The following steps are involved: Step S1: Decode the successfully received data packet, record the decoded image as the current frame, and mark the macroblock area of ​​the current frame where data is lost as the target area; Step S2, marking neighboring areas of the target area based on the target area; Step S3, performing intra-frame prediction on the current frame, predicting the pixel value and pixel confidence in each target area using the pixel values ​​and pixel confidences of the neighboring areas, and calculating the intra-frame predicted pixel value and intra-frame confidence of the target area; Step S4, performing inter-frame prediction on the current frame, using the previous frame image of the current frame as a reference frame, recording the region in the reference frame at the same position as the target region in the current frame as a reference target region, performing motion prediction on the reference target region to obtain a motion vector of the reference target region, predicting pixel values ​​and pixel confidence levels in each target region based on information of the reference frame and information of the motion vector of the reference target region, and calculating inter-frame predicted pixel values ​​and inter-frame confidence levels of the target region; Step S5, fusion restoration, performs fusion calculation based on the intra-frame predicted pixel value, intra-frame confidence, inter-frame predicted pixel value and inter-frame confidence of the target area to obtain the restored pixels and restoration confidence of the target area, and fills each restored pixel into the target area to obtain the restored image, and uses the restored image as the final output of the restoration algorithm.

2. The method for repairing image quality degradation caused by packet loss in real-time video communication according to claim 1, characterized in that: The neighboring area is a number of peripheral macroblocks adjacent to the target area.

3. The method for repairing image quality degradation caused by packet loss in real-time video communication according to claim 1, characterized in that: The pixel confidence is the probability that the pixel value is a true value. The pixel confidence of the pixels in the non-lost macroblock area is 1.

4. The method for repairing image quality degradation caused by packet loss in real-time video communication according to claim 1, characterized in that: In step S3, intra-frame prediction is performed on the target region according to the average confidence values ​​of neighboring regions from high to low.

5. The method for repairing image quality degradation caused by packet loss in real-time video communication according to claim 4, characterized in that: If a neighboring region is a non-lost macroblock, the neighboring region confidence of the region is 1; If a neighboring region is a lost macroblock, ie, a target region, the neighboring region confidence of the region is the average intra-frame confidence of all pixels in the region.

6. The method for repairing image quality degradation caused by packet loss in real-time video communication according to claim 4, characterized in that: During the intra-frame prediction of the region, the prediction calculation method of any point T in the target region is: Intra-frame predicted pixel value at point T = (L2*Nb1*C1+L1*Nb2*C2) / ((Nb 1+Nb2)(L1+L2)); Intra-frame confidence at point T = (L2*Nb1+L1*Nb2) / (L1+L2); Among them, the areas of the four upper, lower, left and right neighboring areas of the target area are recorded as Q1, Q2, Q3, and Q4, respectively, and their neighbor confidences are recorded as N1, N2, N3, and N4, respectively; compare the sizes of N1+N3 and N2+N4, and the neighbor areas corresponding to the larger ones are recorded as Qb1 and Qb2, and the corresponding neighbor confidences are Nb1 and Nb2; the pixels on the edges where Qb1 and Qb2 intersect with the target area are recorded as E1 and E2; the pixel values ​​of the perpendicular line from T to E1 and E2 and the intersection of E1 and E2 are recorded as C1 and C2; the distance from T to C1 is recorded as L1, and the distance from T to C2 is recorded as L2.

7. The method for repairing image quality degradation caused by packet loss in real-time video communication according to claim 1, characterized in that: The step S4 of obtaining the motion vector by motion prediction includes: 1) The area to be motion predicted is denoted as T, the area is a square, and its side length is denoted as L; 2) The predicted motion vector is denoted as V, where V is a two-dimensional vector with the horizontal dimension denoted as Vx and the vertical dimension denoted as Vy; 3) Any point P in the region T has its coordinates denoted as (Px, Py) and its pixel value as Sp; 4) The motion prediction corresponding point of P is Pv in the reference frame, its coordinates are marked as (Px-Vx, Py-Vy), and its pixel value in the reference frame is recorded as Spv; 5) The regional prediction error of T is denoted as E. For any point P in T, E = Σ(Sp–Spv)^2; 6) For different V values, i.e. different Vx and Vy, different E can be calculated; 7) Traverse all Vx and Vy between -L and L and calculate the minimum E value, which is recorded as Emin; 8) Vx and Vy when E is equal to Emin are denoted as Vbx and Vby, and the corresponding motion vector is denoted as Vb. Then Vb is the motion vector output by the motion prediction of region T.

8. The method for repairing image quality degradation caused by packet loss in real-time video communication according to claim 1, characterized in that: The step S4 of performing the inter-frame prediction includes: 1) The target region is denoted as T, and the motion vector obtained by motion prediction of the reference target region corresponding to the target region in the reference frame is denoted as Vb, and the coordinates are denoted as (Vbx, Vby); 2) Any point P in the target area T, its coordinates are marked as (Px, Py); 3) In the reference frame, the point corresponding to the motion of point P is denoted as Pv. The coordinates of Pv are (Px-Vbx, Py-Vby), the pixel value of Pv is denoted as Svf, and the confidence of Pv is denoted as Nvf. If Pv is a point in the lost macroblock, the pixel value of Pv uses its restored pixel value, and the confidence uses its restored confidence If Pv is a point in a non-lost macroblock, the pixel value of Pv uses its original pixel value and the confidence level uses 1. 4) Use the pixel value and confidence of point Pv as the inter-frame prediction result of point P, that is, the inter-frame prediction result of point P is: Point P inter-frame prediction pixel value = Svf Inter-frame confidence at point P = Nvf 5) Traverse all pixels in the target area and calculate their inter-frame predicted pixel values ​​and inter-frame confidence.

9. The method for repairing image quality degradation caused by packet loss in real-time video communication according to claim 1, characterized in that: The step S5 of performing fusion repair comprises: 1) The target area is denoted as T, with any point P in it; 2) The intra-frame prediction pixel value of point P is recorded as Si, the intra-frame confidence of point P is recorded as Ni, the inter-frame prediction pixel value of point P is recorded as So, and the inter-frame confidence of point P is recorded as No; 3) The repair result of P is: The restored pixel value of point P = (Si*Ni+So*No) / (Ni+No); Repair confidence of point P = (Ni+No–Ni*No)*(Ni+No-abs(Ni-No)) / (Ni+No); 4) Traversing all pixels in the target area and calculating their restored pixel values ​​and restoration confidence; 5) Traverse all target areas to obtain the repaired image.

Images