Image inpainting method and apparatus
By determining the pixel loss direction of the missing block and using the grayscale values of surrounding normal pixels to repair the image, the problem of image distortion caused by bit errors in video conferencing was solved, improving image quality and recovery effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-25
- Publication Date
- 2026-03-03
AI Technical Summary
Image distortion and bit errors caused by network quality issues during video conferencing are common, especially in high-definition video conferencing where image quality and network smoothness are required to be high. However, bit errors often occur during network transmission, leading to problems such as image blurring and pixelation.
The direction of pixel loss is determined based on the gray values of pixels at the edge of the lost block. The gray values of surrounding normal pixels are used to repair the area to be repaired in the lost block. The gray values of reference pixels are used to calculate the gray values of the area to be repaired, thereby achieving image restoration.
It effectively corrects image distortion caused by bit errors, improves the visual effect of image restoration, and reduces the computational load of image detection and restoration.
Smart Images
Figure CN115937060B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image restoration method and apparatus. Background Technology
[0002] Supported by high-speed internet and 4G / 5G networks, video conferencing has become a key area of development in communication and network technologies, experiencing explosive growth. Because video conferencing eliminates constraints such as distance, location, and cost, and allows for more efficient and convenient multi-point real-time interaction of audio and video, more and more companies and users are adopting video conferencing as their daily meeting method.
[0003] However, video conferencing, especially high-definition video conferencing, has high requirements for image quality and network smoothness. Due to the large number of network nodes involved, the compressed video image often suffers from bit errors due to network quality issues after transmission and interaction, resulting in blurry video and pixelated images. Summary of the Invention
[0004] This application provides an image restoration method and apparatus to solve the problem of image distortion caused by bit errors.
[0005] In a first aspect, this application provides an image restoration method, comprising:
[0006] The direction of pixel loss in the image within the lost block is determined based on the grayscale values of the pixels at the edge of the lost block.
[0007] The region to be repaired is determined based on the direction of pixel loss, and reference pixels for repairing the region to be repaired are also determined; wherein, the reference pixels are normal pixels surrounding the lost block.
[0008] The grayscale value of the area to be repaired is obtained using the grayscale value of the reference pixel.
[0009] The grayscale value of the area to be repaired is used to repair the area. By using the grayscale value of the reference pixel to obtain the grayscale value of the area to be repaired in the lost block, and using the obtained grayscale value to repair the area to be repaired, the image is repaired, and the image distortion problem caused by bit errors is solved.
[0010] In one possible implementation, determining the pixel loss direction of the image in the lost block based on the grayscale values of the edge pixels of the lost block includes:
[0011] When the absolute value of the grayscale difference between the top and bottom opposite pixels in the edge pixels of the lost block is greater than the grayscale value of the lost block, and the absolute value of the grayscale difference between the left and right opposite pixels in the edge pixels of the lost block is less than one-quarter of the grayscale value of the lost block, the pixel loss direction of the lost block is determined to be horizontal.
[0012] When the absolute value of the grayscale difference between the left and right opposite pixels in the edge pixels of the lost block is greater than the grayscale value of the lost block, and the absolute value of the grayscale difference between the top and bottom opposite pixels in the edge pixels of the lost block is less than one-quarter of the grayscale value of the lost block, the pixel loss direction of the lost block is determined to be the vertical direction.
[0013] When the difference between the first and second differences in the edge pixels of the lost block is less than one-sixth of the lost block, the pixel loss direction of the lost block is determined to be the diagonal direction.
[0014] When the second difference minus the first difference in the edge pixels of the lost block is less than one-sixth of the lost block, the pixel loss direction of the lost block is determined to be the second diagonal direction; wherein, the first difference is the absolute value of the difference between the sum of the two adjacent pixels at the upper right corner and the sum of the two adjacent pixels at the lower left corner in the edge pixels of the lost block; the second difference is the absolute value of the difference between the sum of the two adjacent pixels at the upper left corner and the sum of the two adjacent pixels at the lower right corner in the edge pixels of the lost block.
[0015] In one possible implementation, among the edge pixels of the lost block, the first difference minus the second difference is less than one-sixth of the lost block, including:
[0016] │(sum2+sum3)-(sum6+sum7)│-│(sum1+sum8)-(sum4+sum5)│ <T / 6;
[0017] Wherein, sum1 is the sum of the grayscale values of the pixels in the upper left half of the edge of the lost block; sum2 is the sum of the grayscale values of the pixels in the upper right half of the edge of the lost block; sum3 is the sum of the grayscale values of the pixels in the upper right half of the edge of the lost block; sum4 is the sum of the grayscale values of the pixels in the lower right half of the edge of the lost block; sum5 is the sum of the grayscale values of the pixels in the lower right half of the edge of the lost block; sum6 is the sum of the grayscale values of the pixels in the lower left half of the edge of the lost block; sum7 is the sum of the grayscale values of the pixels in the lower left half of the edge of the lost block; and sum8 is the sum of the grayscale values of the pixels in the upper left half of the edge of the lost block.
[0018] In one possible implementation, determining the region to be repaired based on the pixel loss direction, and determining reference pixels for repairing the region to be repaired, includes:
[0019] Within the lost block, a straight line is drawn along the direction of pixel loss as the bottom line of the area to be repaired;
[0020] The area on one side of the baseline that is surrounded by the edge of the missing block is the area to be repaired;
[0021] Using a point on the baseline as the intersection point, and assuming N straight lines passing through the intersection point, the area to be repaired is divided into N+1 partitions, where the included angles of the N+1 partitions at the intersection point are equal.
[0022] On the other side of the baseline, two supplementary straight lines are provided that pass through the intersection point. The angles between the supplementary straight lines and the baseline are equal to the angles between the partition and the intersection point.
[0023] The pixels adjacent to the edge of the lost block on the baseline, N straight lines, and two supplementary straight lines are the reference pixels for the area to be repaired.
[0024] In one possible implementation, obtaining the grayscale value of the region to be repaired using the grayscale value of the reference pixel includes:
[0025] Using formula
[0026]
[0027] Obtain the grayscale values of each partition of the area to be repaired;
[0028] Wherein, M is the Mth partition in the region to be repaired, M = 1, 2, ..., N+1; V M The grayscale value of the Mth partition;
[0029] in, Let k1 be the grayscale value of the pixel point k1 that is adjacent to the edge of the lost block on one side of the intersection point on the k-th straight line. Let k2 be the gray value of the pixel point k2 that is adjacent to the edge of the lost block on the other side of the intersection point on the k-th straight line.
[0030]
[0031] Where T1 is the grayscale value of the lost block;
[0032] Wherein, P is the intersection point. Let k be the distance between point P and point k1. Let k be the distance between point P and point k2.
[0033] Secondly, this application provides an image restoration apparatus, comprising:
[0034] The first determining module is used to determine the pixel loss direction of the image in the lost block based on the gray value of the edge pixels of the lost block;
[0035] The second determining module is used to determine the area to be repaired based on the pixel loss direction, and to determine reference pixels for repairing the area to be repaired; wherein, the reference pixels are normal pixels around the lost block;
[0036] A grayscale calculation module is used to obtain the grayscale value of the area to be repaired using the grayscale value of the reference pixel.
[0037] The repair module is used to repair the area to be repaired using the grayscale values of the area to be repaired.
[0038] In one possible implementation, the first determining module is used to:
[0039] When the absolute value of the grayscale difference between the top and bottom opposite pixels in the edge pixels of the lost block is greater than the grayscale value of the lost block, and the absolute value of the grayscale difference between the left and right opposite pixels in the edge pixels of the lost block is less than one-quarter of the grayscale value of the lost block, the pixel loss direction of the lost block is determined to be horizontal.
[0040] When the absolute value of the grayscale difference between the left and right opposite pixels in the edge pixels of the lost block is greater than the grayscale value of the lost block, and the absolute value of the grayscale difference between the top and bottom opposite pixels in the edge pixels of the lost block is less than one-quarter of the grayscale value of the lost block, the pixel loss direction of the lost block is determined to be the vertical direction.
[0041] When the difference between the first and second differences in the edge pixels of the lost block is less than one-sixth of the lost block, the pixel loss direction of the lost block is determined to be the diagonal direction.
[0042] When the second difference minus the first difference in the edge pixels of the lost block is less than one-sixth of the lost block, the pixel loss direction of the lost block is determined to be the second diagonal direction; wherein, the first difference is the absolute value of the difference between the sum of the two adjacent pixels at the upper right corner and the sum of the two adjacent pixels at the lower left corner in the edge pixels of the lost block; the second difference is the absolute value of the difference between the sum of the two adjacent pixels at the upper left corner and the sum of the two adjacent pixels at the lower right corner in the edge pixels of the lost block.
[0043] In one possible implementation, the first determining module is configured to perform the following operation when, among the edge pixels of the lost block, the result of a first difference minus a second difference is less than one-sixth of the lost block:
[0044] │(sum2+sum3)-(sum6+sum7)│-│(sum1+sum8)-(sum4+sum5)│ <T / 6;
[0045] Wherein, sum1 is the sum of the grayscale values of the pixels in the upper left half of the edge of the lost block; sum2 is the sum of the grayscale values of the pixels in the upper right half of the edge of the lost block; sum3 is the sum of the grayscale values of the pixels in the upper right half of the edge of the lost block; sum4 is the sum of the grayscale values of the pixels in the lower right half of the edge of the lost block; sum5 is the sum of the grayscale values of the pixels in the lower right half of the edge of the lost block; sum6 is the sum of the grayscale values of the pixels in the lower left half of the edge of the lost block; sum7 is the sum of the grayscale values of the pixels in the lower left half of the edge of the lost block; and sum8 is the sum of the grayscale values of the pixels in the upper left half of the edge of the lost block.
[0046] In one possible implementation, the second determining module is used to:
[0047] Within the lost block, a straight line is drawn along the direction of pixel loss as the bottom line of the area to be repaired;
[0048] The area on one side of the baseline that is surrounded by the edge of the missing block is the area to be repaired;
[0049] Using a point on the baseline as the intersection point, and assuming N straight lines passing through the intersection point, the area to be repaired is divided into N+1 partitions, where the included angles of the N+1 partitions at the intersection point are equal.
[0050] On the other side of the baseline, two supplementary straight lines are provided that pass through the intersection point. The angles between the supplementary straight lines and the baseline are equal to the angles between the partition and the intersection point.
[0051] The pixels adjacent to the edge of the lost block on the baseline, N straight lines, and two supplementary straight lines are the reference pixels for the area to be repaired.
[0052] In one possible implementation, the grayscale calculation module is used for:
[0053] Using formula
[0054]
[0055] Obtain the grayscale values of each partition of the area to be repaired;
[0056] Wherein, M is the Mth partition in the region to be repaired, M = 1, 2, ..., N+1; V M The grayscale value of the Mth partition;
[0057] in, Let k1 be the grayscale value of the pixel point k1 that is adjacent to the edge of the lost block on one side of the intersection point on the k-th straight line. Let k2 be the gray value of the pixel point k2 that is adjacent to the edge of the lost block on the other side of the intersection point on the k-th straight line.
[0058]
[0059] Where T1 is the grayscale value of the lost block;
[0060] Wherein, P is the intersection point. Let k be the distance between point P and point k1. Let k be the distance between point P and point k2.
[0061] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method of any one of the first aspects.
[0062] Fourthly, this application provides a chip system that performs the method of any one of the first aspects.
[0063] Fifthly, this application provides a computer program that, when executed by a computer, performs the method of any one of the first aspects.
[0064] In one possible design, the program in the fifth aspect can be stored wholly or partially on a storage medium packaged with the processor, or it can be stored wholly or partially on a memory not packaged with the processor. Attached Figure Description
[0065] Figure 1 This is a flowchart of an image restoration method according to an embodiment of this application;
[0066] Figure 2 This is a schematic diagram of the missing block and its edge pixels in an embodiment of this application;
[0067] Figure 3 This is a schematic diagram illustrating the determination of the area to be repaired and reference pixels in an embodiment of this application;
[0068] Figure 4 This is a schematic diagram illustrating the application of the image restoration method in the embodiments of this application;
[0069] Figure 5 This is a schematic diagram of an image restoration device according to an embodiment of this application. Detailed Implementation
[0070] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.
[0071] One embodiment of this application is, for example... Figure 1 As shown, the image restoration method includes steps 11-14.
[0072] Step 11. Determine the pixel loss direction of the image in the lost block based on the grayscale value of the edge pixels of the lost block.
[0073] For example, when the absolute value of the grayscale difference between the top and bottom opposite pixels in the edge pixels of the lost block is greater than the grayscale value of the lost block, and the absolute value of the grayscale difference between the left and right opposite pixels in the edge pixels of the lost block is less than one-quarter of the grayscale value of the lost block, the pixel loss direction of the lost block is determined to be horizontal.
[0074] When the absolute value of the grayscale difference between the left and right opposite pixels in the edge pixels of the lost block is greater than the grayscale value of the lost block, and the absolute value of the grayscale difference between the top and bottom opposite pixels in the edge pixels of the lost block is less than one-quarter of the grayscale value of the lost block, the pixel loss direction of the lost block is determined to be the vertical direction.
[0075] When the difference between the first and second differences in the edge pixels of the lost block is less than one-sixth of the lost block, the pixel loss direction of the lost block is determined to be the diagonal direction.
[0076] When the second difference minus the first difference in the edge pixels of the lost block is less than one-sixth of the lost block, the pixel loss direction of the lost block is determined to be the second diagonal direction; wherein, the first difference is the absolute value of the difference between the sum of the two adjacent pixels at the upper right corner and the sum of the two adjacent pixels at the lower left corner in the edge pixels of the lost block; the second difference is the absolute value of the difference between the sum of the two adjacent pixels at the upper left corner and the sum of the two adjacent pixels at the lower right corner in the edge pixels of the lost block.
[0077] like Figure 2 As shown, assuming the size of the lost block is 16×16 pixels, the edge pixels of the lost block are divided into 8 parts: s1, s2, s3, s4, s5, s6, s7, and s8. The grayscale sum of the pixels in these 8 parts is sum... k (k = 1, ..., 8), i.e., sum k =sum(s) k) The grayscale sum of each partial pixel is the grayscale sum of 7 pixel points in each part. The grayscale value of each pixel point is obtained according to the bimodal distribution of the regional domain and the grayscale domain or the pixel waveform trend.
[0078] If |(sum5 + sum6) - (sum1 + sum2)| > T && |(sum7 + sum8) - (sum3 + sum4)| < T / 4, then it is determined that the pixel loss direction of the lost block is the horizontal direction; where T is the grayscale value of the lost block.
[0079] Otherwise, if |(sum7 + sum8) - (sum3 + sum4)| > T && |(sum1 + sum2) - (sum5 + sum6)| < T / 4, then it is determined that the pixel loss direction of the lost block is the vertical direction;
[0080] Otherwise, if |(sum2 + sum3) - (sum6 + sum7)| - |(sum1 + sum8) - (sum4 + sum5)| < T / 6, then it is determined that the pixel loss direction of the lost block is the diagonal direction;
[0081] Otherwise, if |(sum1 + sum8) - (sum4 + sum5)| - |(sum2 + sum3) - (sum6 + sum7)| < T / 6, then it is determined that the pixel loss direction of the lost block is the sub - diagonal direction.
[0082] Step 12. Determine the area to be repaired according to the pixel loss direction, and determine the reference pixel points for repairing the area to be repaired; where the reference pixel points are the normal pixel points around the lost block.
[0083] For example, within the lost block, set a straight line in the pixel loss direction as the bottom line of the area to be repaired;
[0084] The area surrounded by the lost block edge on one side of the bottom line is the area to be repaired;
[0085] Take a point on the bottom line as the intersection point, set N straight lines passing through the intersection point, divide the area to be repaired into N + 1 partitions, and the angles of the N + 1 partitions at the intersection point are equal;
[0086] On the other side of the bottom line, set two supplementary straight lines passing through the intersection point, and the angles between the supplementary straight lines and the bottom line are equal to the angles of the partitions at the intersection point;
[0087] The pixel points adjacent to the lost block edge on the bottom line, N straight lines and two supplementary straight lines are the reference pixel points of the area to be repaired.
[0088] like Figure 3 As shown, the determined pixel loss direction is horizontal, indicating that image pixels within the dashed line area are missing. Let the bottom line be the (N+1)th straight line, and let N other straight lines numbered 1, 2, ..., N-1, N. Divide the lost pixel area above the (N+1)th straight line into N+1 partitions numbered 1, 2, ..., N-1, N, N+1. Furthermore, the angles between the partitions at their intersection point P are θ1 = θ2 = ... = θ N =θ N+1 The supplementary lines are numbered -1 and N+2, which coincide with the lines numbered N and 1 respectively. θ -1 =θ N θ N+2 =θ1. Each of these straight lines has two correct pixels adjacent to the edge of the missing block as reference pixels. For example, the two reference pixels on line -1 are -11 and -12, on line 1 are 11 and 12, on line 2 are 21 and 22, ... on line N-1 are N-11 and N-12, on line N are N1 and N2, on line N+1 are N+11 and N+12, and on line N+2 are N+21 and N+22. Reference pixels 11 and N+22 coincide, reference pixels 12 and N+21 coincide, reference pixels N1 and -12 coincide, and reference pixels -11 and N2 coincide.
[0089] Step 13. Obtain the gray value of the area to be repaired using the gray value of the reference pixel.
[0090] For example, using formulas
[0091]
[0092] Obtain the grayscale values of each partition of the area to be repaired;
[0093] Wherein, M is the Mth partition in the region to be repaired, M = 1, 2, ..., N+1; V M The grayscale value of the Mth partition;
[0094]
[0095] in, Let k1 be the grayscale value of the pixel point k1 that is adjacent to the edge of the lost block on one side of the intersection point on the k-th straight line. Let k be the gray value of pixel k2 on the other side of the intersection point on the k-th straight line, which is adjacent to the edge of the lost block; k = M-1, M, M+1.
[0096] For example
[0097] The absolute value of the difference between the gray values of reference pixel M1 and reference pixel M2 on the Mth straight line;
[0098] The absolute value of the difference between the gray values of reference pixel M-11 and reference pixel M-12 on the (M-1)th straight line;
[0099] The absolute value of the difference between the gray values of reference pixel M+11 and reference pixel M+12 on the (M+1)th straight line.
[0100]
[0101] Where T1 is the grayscale value of the lost block;
[0102] Wherein, P is the intersection point. Let k be the distance between point P and point k1. Let k be the distance between point P and point k2. k = M-1, M, M+1.
[0103] For example
[0104]
[0105] The sum of the gray value of reference pixel M1 on the Mth line multiplied by the distance between point P and reference pixel M2, and the gray value of reference pixel M2 on the Mth line multiplied by the distance between point P and reference pixel M1, is divided by the distance between reference pixels M1 and M2.
[0106]
[0107] The sum of the gray value of reference pixel M-11 on the (M-1)th line multiplied by the distance between point P and reference pixel M-12, and the gray value of reference pixel M-12 on the (M-1)th line multiplied by the distance between point P and reference pixel M-11, is divided by the distance between reference pixel M-11 and M-12.
[0108]
[0109] The sum of the gray value of reference pixel M+11 on the (M+1)th line multiplied by the distance between point P and reference pixel M+12, and the gray value of reference pixel M+12 on the (M+1)th line multiplied by the distance between point P and reference pixel M+11, is divided by the distance between reference pixel M+11 and M+12.
[0110] by Figure 3 For example, the grayscale value of the first partition is calculated by substituting the grayscale values of the reference pixels on the -1, N+1 and 1 lines into the above formula; the grayscale value of the second partition is calculated by substituting the grayscale values of the reference pixels on the 1, 2 and 3 lines into the above formula; ..., the grayscale value of the Nth partition is calculated by substituting the grayscale values of the reference pixels on the N-1, N and N+1 lines into the above formula; and the grayscale value of the N+1th partition is calculated by substituting the grayscale values of the reference pixels on the N, N+1 and N+2 lines into the above formula.
[0111] Step 14. Use the grayscale value of the area to be repaired obtained in Step 13 above to repair the area to be repaired.
[0112] For example, by replacing the original gray values of each partition in the lost block with the gray values of each partition obtained in step 13 above, the image can be repaired, thus solving the image distortion problem caused by bit errors.
[0113] like Figure 4 As shown, image repair involves the video decoder decoding the received image data and monitoring the decoded data to scan and analyze the status codes of all received images. When the macroblock status code is OK, it indicates that the decoding was correct, meaning the received macroblock is a normal block; when the status code is Concealed, it indicates that some pixels in the macroblock are missing, making it a lost macroblock; when the status code is Corrupted, it indicates that the macroblock has been masked for errors; and when the status code is Empty, it indicates that the macroblock is empty.
[0114] For macroblocks with a status code of Concealed obtained from scanning analysis, grayscale values for repair can be obtained using the methods described in the above embodiments. The obtained grayscale values are then used to repair the missing blocks, resulting in a normally viewable image.
[0115] Another embodiment of this application is, for example Figure 5 As shown, an image restoration device includes: a first determining module 51, a second determining module 52, a grayscale calculation module 53, and a restoration module 54.
[0116] The first determining module 51 is used to determine the pixel loss direction of the image in the lost block based on the gray value of the edge pixels of the lost block; the second determining module 52 is used to determine the region to be repaired based on the pixel loss direction, and to determine the reference pixel for repairing the region to be repaired; wherein, the reference pixel is the normal pixel around the lost block; the gray value calculation module 53 is used to obtain the gray value of the region to be repaired using the gray value of the reference pixel; the repair module 54 is used to repair the region to be repaired using the gray value of the region to be repaired.
[0117] In one possible implementation, the first determining module 51 is used to:
[0118] When the absolute value of the grayscale difference between the top and bottom opposite pixels in the edge pixels of the lost block is greater than the grayscale value of the lost block, and the absolute value of the grayscale difference between the left and right opposite pixels in the edge pixels of the lost block is less than one-quarter of the grayscale value of the lost block, the pixel loss direction of the lost block is determined to be horizontal.
[0119] When the absolute value of the grayscale difference between the left and right opposite pixels in the edge pixels of the lost block is greater than the grayscale value of the lost block, and the absolute value of the grayscale difference between the top and bottom opposite pixels in the edge pixels of the lost block is less than one-quarter of the grayscale value of the lost block, the pixel loss direction of the lost block is determined to be the vertical direction.
[0120] When the difference between the first and second differences in the edge pixels of the lost block is less than one-sixth of the lost block, the pixel loss direction of the lost block is determined to be the diagonal direction.
[0121] When the second difference minus the first difference in the edge pixels of the lost block is less than one-sixth of the lost block, the pixel loss direction of the lost block is determined to be the second diagonal direction; wherein, the first difference is the absolute value of the difference between the sum of the two adjacent pixels at the upper right corner and the sum of the two adjacent pixels at the lower left corner in the edge pixels of the lost block; the second difference is the absolute value of the difference between the sum of the two adjacent pixels at the upper left corner and the sum of the two adjacent pixels at the lower right corner in the edge pixels of the lost block.
[0122] In one possible implementation, the first determining module 51 is configured to, when the first difference minus the second difference in the edge pixels of the lost block is less than one-sixth of the lost block, perform the following:
[0123] │(sum2+sum3)-(sum6+sum7)│-│(sum1+sum8)-(sum4+sum5)│ <T / 6;
[0124] Wherein, sum1 is the sum of the grayscale values of the pixels in the upper left half of the edge of the lost block; sum2 is the sum of the grayscale values of the pixels in the upper right half of the edge of the lost block; sum3 is the sum of the grayscale values of the pixels in the upper right half of the edge of the lost block; sum4 is the sum of the grayscale values of the pixels in the lower right half of the edge of the lost block; sum5 is the sum of the grayscale values of the pixels in the lower right half of the edge of the lost block; sum6 is the sum of the grayscale values of the pixels in the lower left half of the edge of the lost block; sum7 is the sum of the grayscale values of the pixels in the lower left half of the edge of the lost block; and sum8 is the sum of the grayscale values of the pixels in the upper left half of the edge of the lost block.
[0125] In one possible implementation, the second determining module 52 is used to:
[0126] Within the lost block, a straight line is drawn along the direction of pixel loss as the bottom line of the area to be repaired;
[0127] The area on one side of the baseline that is surrounded by the edge of the missing block is the area to be repaired;
[0128] Using a point on the baseline as the intersection point, and assuming N straight lines passing through the intersection point, the area to be repaired is divided into N+1 partitions, where the included angles of the N+1 partitions at the intersection point are equal.
[0129] On the other side of the baseline, two supplementary straight lines are provided that pass through the intersection point. The angles between the supplementary straight lines and the baseline are equal to the angles between the partition and the intersection point.
[0130] The pixels adjacent to the edge of the lost block on the baseline, N straight lines, and two supplementary straight lines are the reference pixels for the area to be repaired.
[0131] In one possible implementation, the grayscale calculation module 53 is used for:
[0132] Using formula
[0133]
[0134] Obtain the grayscale values of each partition of the area to be repaired;
[0135] Wherein, M is the Mth partition in the region to be repaired, M = 1, 2, ..., N+1; V M The grayscale value of the Mth partition;
[0136] in, Let k1 be the grayscale value of the pixel point k1 that is adjacent to the edge of the lost block on one side of the intersection point on the k-th straight line. Let k2 be the gray value of the pixel point k2 that is adjacent to the edge of the lost block on the other side of the intersection point on the k-th straight line.
[0137]
[0138] Where T1 is the grayscale value of the lost block;
[0139] Wherein, P is the intersection point. Let k be the distance between point P and point k1. Let k be the distance between point P and point k2.
[0140] The above embodiments effectively reduce the computational load during image detection and restoration by adaptively repairing lost pixel blocks, thereby improving the visual effect of image restoration.
[0141] It is understood that some or all of the steps or operations in the above embodiments are merely examples, and other operations or variations thereof can be performed in the embodiments of this application. Furthermore, the steps may be performed in different orders as presented in the above embodiments, and it is not necessary to perform all the operations in the above embodiments.
[0142] This application also provides an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform... Figures 1-4 The method shown.
[0143] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to execute this application. Figures 1-4 The method provided in the illustrated embodiment.
[0144] This application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to execute this application. Figures 1-4 The method provided in the illustrated embodiment.
[0145] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0146] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0147] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0148] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.
Claims
1. An image inpainting method characterized by, The method comprises the following steps: determining the pixel loss direction of the image in the lost block according to the gray value of the edge pixel of the lost block; determining the region to be repaired according to the pixel loss direction and determining the reference pixel point for repairing the region to be repaired; wherein the reference pixel point is a normal pixel point around the lost block; obtaining the gray value of the region to be repaired by using the gray value of the reference pixel point; repairing the region to be repaired by using the gray value of the region to be repaired; wherein the step of determining the pixel loss direction of the image in the lost block according to the gray value of the edge pixel of the lost block comprises: when the absolute value of the gray difference between the upper and lower opposite edge pixels in the edge pixel of the lost block is greater than the gray value of the lost block, and the absolute value of the gray difference between the left and right opposite edge pixels in the edge pixel of the lost block is less than one fourth of the gray value of the lost block, determining that the pixel loss direction of the lost block is horizontal direction; when the absolute value of the gray difference between the left and right opposite edge pixels in the edge pixel of the lost block is greater than the gray value of the lost block, and the absolute value of the gray difference between the upper and lower opposite edge pixels in the edge pixel of the lost block is less than one fourth of the gray value of the lost block, determining that the pixel loss direction of the lost block is vertical direction; when the first difference minus the second difference in the edge pixel of the lost block is less than one sixth of the lost block, determining that the pixel loss direction of the lost block is diagonal direction; wherein the first difference is the absolute value of the difference between the sum of the two adjacent edge pixels at the upper right corner and the sum of the two adjacent edge pixels at the lower left corner in the edge pixel of the lost block; the second difference is the absolute value of the difference between the sum of the two adjacent edge pixels at the upper left corner and the sum of the two adjacent edge pixels at the lower right corner in the edge pixel of the lost block; when the second difference minus the first difference in the edge pixel of the lost block is less than one sixth of the lost block, determining that the pixel loss direction of the lost block is sub-diagonal direction; wherein the first difference minus the second difference in the edge pixel of the lost block is less than one sixth of the lost block, comprising: │(sum2+sum3)-(sum6+sum7)│-│(sum1+sum8)-(sum4+sum5)│<T / 6; wherein sum1 is the sum of the pixel gray values of the left half of the upper edge in the edge pixel of the lost block; sum2 is the sum of the pixel gray values of the right half of the upper edge in the edge pixel of the lost block; sum3 is the sum of the pixel gray values of the upper half of the right edge in the edge pixel of the lost block; sum4 is the sum of the pixel gray values of the lower half of the right edge in the edge pixel of the lost block; sum5 is the sum of the pixel gray values of the right half of the lower edge in the edge pixel of the lost block; sum6 is the sum of the pixel gray values of the left half of the lower edge in the edge pixel of the lost block; sum7 is the sum of the pixel gray values of the lower half of the left edge in the edge pixel of the lost block; sum8 is the sum of the pixel gray values of the upper half of the left edge in the edge pixel of the lost block; T is the gray value of the lost block. The method comprises the following steps: A line is set in the pixel loss direction in the loss block as a bottom line of the region to be repaired; The region surrounded by the loss block edge on one side of the bottom line is the region to be repaired; A point on the bottom line is set as a cross point, N lines passing through the cross point are set, and the region to be repaired is divided into N+1 sub-regions, the angles of the N+1 sub-regions at the cross point are equal; On the other side of the bottom line, two supplementary lines passing through the cross point are set, and the angles of the supplementary lines and the bottom line are equal to the angles of the sub-regions at the cross point; The pixel points adjacent to the loss block edge on the bottom line, the N lines and the two supplementary lines are the reference pixel points of the region to be repaired.
2. The method of claim 1, wherein, The gray value of the region to be repaired is obtained by using the gray values of the reference pixel points, which comprises the following steps: The gray values of the sub-regions of the region to be repaired are obtained by using the formula ； Wherein, Wherein, the M is the Mth partition in the region to be repaired, M = 1, 2, …, N + 1; V M is the gray value of the Mth partition. dp k =│ │, wherein, is the gray value of the pixel point k1 on the kth straight line, which is adjacent to the edge of the missing block on one side of the intersection point, is the gray value of the pixel point k2 on the kth straight line, which is adjacent to the edge of the missing block on the other side of the intersection point. ； The method comprises the following steps: RV k = f(P, k1, k2) = ( + ) / ( + ); wherein P is the intersection point, is the distance between P and k1, is the distance between P and k2.
3. An image inpainting apparatus characterized by comprising: The first determining module is configured to determine the pixel loss direction of the image in the loss block according to the gray values of the loss block edge pixels; The second determining module is configured to determine the region to be repaired according to the pixel loss direction, and determine the reference pixel points for repairing the region to be repaired; wherein the reference pixel points are normal pixel points around the loss block; The gray calculating module is configured to obtain the gray value of the region to be repaired by using the gray values of the reference pixel points; The repairing module is configured to repair the region to be repaired by using the gray value of the region to be repaired; When the absolute value of the gray difference between the upper and lower pixels of the loss block edge pixels is greater than the gray value of the loss block, and the absolute value of the gray difference between the left and right pixels of the loss block edge pixels is less than one fourth of the gray value of the loss block, the first determining module is configured to determine that the pixel loss direction of the loss block is the horizontal direction; When the absolute value of the gray difference between the left and right pixels of the loss block edge pixels is greater than the gray value of the loss block, and the absolute value of the gray difference between the upper and lower pixels of the loss block edge pixels is less than one fourth of the gray value of the loss block, the first determining module is configured to determine that the pixel loss direction of the loss block is the vertical direction; When the first difference minus the second difference in the loss block edge pixels is less than one sixth of the gray value of the loss block, the first determining module is configured to determine that the pixel loss direction of the loss block is the diagonal direction; wherein the first difference is the absolute value of the difference between the sum of the two adjacent pixels on the upper right corner and the sum of the two adjacent pixels on the lower left corner of the loss block edge pixels, and the second difference is the absolute value of the difference between the sum of the two adjacent pixels on the upper left corner and the sum of the two adjacent pixels on the lower right corner of the loss block edge pixels; When the second difference minus the first difference in the loss block edge pixels is less than one sixth of the gray value of the loss block, the first determining module is configured to determine that the pixel loss direction of the loss block is the sub-diagonal direction. The first determining module is configured to, when the first difference minus the second difference is less than one sixth of the missing block in the missing block edge pixels, perform: │(sum2+sum3)-(sum6+sum7)│-│(sum1+sum8)-(sum4+sum5)│<T / 6; wherein, sum1 is the sum of the gray scale of the pixels in the left half of the upper edge in the missing block edge pixels; sum2 is the sum of the gray scale of the pixels in the right half of the upper edge in the missing block edge pixels; sum3 is the sum of the gray scale of the pixels in the upper half of the right edge in the missing block edge pixels; sum4 is the sum of the gray scale of the pixels in the lower half of the right edge in the missing block edge pixels; sum5 is the sum of the gray scale of the pixels in the right half of the lower edge in the missing block edge pixels; sum6 is the sum of the gray scale of the pixels in the left half of the lower edge in the missing block edge pixels; sum7 is the sum of the gray scale of the pixels in the lower half of the left edge in the missing block edge pixels; sum8 is the sum of the gray scale of the pixels in the upper half of the left edge in the missing block edge pixels; and T is the gray scale value of the missing block. The second determining module is configured to: in the missing block, a straight line is set in the pixel loss direction as the bottom line of the to-be-repaired region; the region surrounded by the missing block edge on one side of the bottom line is the to-be-repaired region; a point on the bottom line is taken as a cross point, N straight lines passing through the cross point are set, the to-be-repaired region is divided into N+1 sub-regions, and the included angles of the N+1 sub-regions at the cross point are equal; on the other side of the bottom line, two supplementary straight lines passing through the cross point are set, and the included angles between the supplementary straight lines and the bottom line are equal to the included angles of the sub-regions at the cross point; the pixel points adjacent to the missing block edge on the bottom line, the N straight lines and the two supplementary straight lines are reference pixel points of the to-be-repaired region.
4. The apparatus of claim 3, wherein, The gray scale calculating module is configured to: obtain the gray scale values of the sub-regions of the to-be-repaired region by using the formula ； wherein, Wherein, the M is the Mth partition in the region to be repaired, M = 1, 2, …, N + 1; V M is the gray value of the Mth partition. dp k =│ │, wherein, is the gray value of the pixel point k1 on the kth straight line, which is adjacent to the edge of the missing block on one side of the intersection point, is the gray value of the pixel point k2 on the kth straight line, which is adjacent to the edge of the missing block on the other side of the intersection point; ； RV k = f(P, k1, k2) = ( + ) / ( + ); wherein P is the intersection point, is the distance between P and k1, is the distance between P and k2.
Citation Information
Patent Citations
Space adaptive H.264 video I frame error concealment method
CN103051903A
A three-dimensional video error concealment method based on an HEVC standard
CN109819230A