Video frame error repair method, apparatus, equipment, and medium based on spatial domain filtering
By employing spatial filtering technology and a three-stage cascaded post-processing filtering workflow, the problems of blurring and texture loss caused by packet loss in video frames under AVS3 ultra-low latency encoding mode were solved, achieving high-quality restoration of video frames.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MALANSHAN AUDIO & VIDEO LABORATORY
- Filing Date
- 2026-04-22
- Publication Date
- 2026-07-17
AI Technical Summary
In AVS3 ultra-low latency coding mode, the error repair methods for video frames lost in unreliable network environments suffer from blurring and lack of natural texture, and existing spatial interpolation techniques are not effective in repairing these issues.
The spatial filtering technique is used to obtain neighborhood texture information, adjust the boundary pixel values and filter them, and then superimpose them onto the boundary pixels to restore the video frame. Combined with a three-level cascaded post-processing filtering process, hard seams at the boundary are eliminated while retaining edge details, and natural textures are introduced.
It effectively eliminates hard seams at the boundaries in video frame error repair, preserves edge details, restores natural textures, and improves video quality.
Smart Images
Figure CN122093581B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video encoding and decoding technology, and in particular to a method, apparatus, device, and medium for video frame error repair based on spatial domain filtering. Background Technology
[0002] In AVS3's ultra-low latency coding mode, video frames are divided into multiple independently coded slices, each of which can be independently encapsulated into one or more network data packets, achieving subframe-level transmission and decoding. In unreliable network transmission environments, some data packets may fail to arrive at the decoding end on time due to packet loss, latency jitter, or other reasons.
[0003] Currently, most video frame error repair methods use spatial interpolation technology to repair the missing parts of the video frame. This method has the disadvantage that the pixels inside the repaired area lose natural texture details, presenting a blurry "watercolor painting" effect, and lacks the high-frequency texture information that a natural image should have. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a video frame error repair method, apparatus, device, and medium based on spatial domain filtering, which can solve the problem of lack of natural texture by obtaining texture information from the neighborhood and superimposing it onto the boundary pixels. The specific solution is as follows: Firstly, this application provides a video frame error repair method based on spatial domain filtering, applied at the decoding end, including: Acquire a target video frame and determine the reconstructed image block in the target video frame; wherein the reconstructed image block is an image block obtained by reconstructing the missing image block in the target video frame using spatial interpolation technology; Several target boundary pixels are determined based on the distance between each pixel in the reconstructed image block and the boundary of the reconstructed image block, and the pixel values of the target boundary pixels are adjusted to obtain an image block with adjusted boundary; the target boundary pixels are pixels in the boundary region where the distance between them and the boundary of the reconstructed image block is less than a preset distance threshold. Spatial filtering is performed on the boundary-adjusted image block to obtain the corresponding filtered image block; the filtering intensity of the boundary region in the boundary-adjusted image block is lower than the filtering intensity of the non-boundary region. The target texture information is obtained from the neighborhood of the filtered image block, and the target texture information is superimposed on the corresponding target boundary pixels in the filtered image block to complete the error repair of the target video frame.
[0005] Optionally, after acquiring the target video frame, the process further includes: Determine whether the target video frame is a keyframe. If the target video frame is a keyframe, proceed to the step of determining the reconstructed image block in the target video frame. If the target video frame is not a key frame, then the reference frame corresponding to the target video frame is obtained, and the target video frame is repaired using the corresponding reference frame.
[0006] Optionally, adjusting the pixel values of the target boundary pixels to obtain the boundary-adjusted image patch includes: The target fusion coefficient corresponding to each target boundary pixel is determined based on the distance between each target boundary pixel and the boundary of the reconstructed image block, and the target neighborhood radius corresponding to each target boundary pixel is determined based on the distance between each target boundary pixel and the boundary of the reconstructed image block. The local mean value corresponding to each of the target boundary pixels is calculated based on the radius of each target neighborhood, and the local mean value and the corresponding original target pixel value are fused according to the target fusion coefficient to obtain the image block after boundary adjustment; the original target pixel value is the original pixel value corresponding to the target boundary pixel.
[0007] Optionally, the step of performing spatial filtering on the boundary-adjusted image patch to obtain a corresponding filtered image patch includes: Spatial filtering is performed on the boundary-adjusted image block based on the local variance of the pixels in the boundary-adjusted image block to obtain the corresponding filtered image block.
[0008] Optionally, the step of performing spatial filtering on the boundary-adjusted image patch based on the local variance of the pixels in the boundary-adjusted image patch includes: The first weight corresponding to each pixel is determined based on the spatial distance between each pixel and its corresponding neighboring pixels. Calculate the local variance of each pixel in the boundary-adjusted image block within the neighborhood of the target size, and determine the second weight of each pixel based on the local variance of each pixel. Calculate the target weight corresponding to each pixel based on the first weight and the second weight, and perform spatial filtering on the boundary-adjusted image block based on the target weight.
[0009] Optionally, obtaining target texture information from the neighborhood of the filtered image patch includes: The target texture source corresponding to each target boundary pixel is determined based on the distance between each target boundary pixel and the boundary of the filtered image block, and the target texture information corresponding to each target boundary pixel is obtained from each target texture source.
[0010] Optionally, superimposing the target texture information onto the corresponding target boundary pixels in the filtered image patch includes: The target superposition intensity corresponding to each target texture information is determined based on the distance between each target boundary pixel and the boundary of the filtered image block, and the target texture information is superimposed onto the corresponding target boundary pixel in the filtered image block according to the target superposition intensity; wherein, the magnitude of the target superposition intensity is negatively correlated with the distance between the target and the target; the target distance is the distance between the target edge pixel and the boundary of the filtered image block.
[0011] Secondly, this application provides a video frame error repair device based on spatial domain filtering, applied at the decoding end, comprising: A video frame acquisition module is used to acquire a target video frame and determine the reconstructed image block in the target video frame; wherein, the reconstructed image block is an image block obtained by reconstructing the missing image block in the target video frame using spatial interpolation technology; A boundary pixel determination module is used to determine a number of target boundary pixels based on the distance between each pixel in the reconstructed image block and the boundary of the reconstructed image block, and to adjust the pixel values of the target boundary pixels to obtain an image block after boundary adjustment; the target boundary pixels are pixels in the boundary region where the distance between them and the boundary of the reconstructed image block is less than a preset distance threshold. The image filtering module is used to perform spatial filtering on the boundary-adjusted image block to obtain a corresponding filtered image block; the filtering intensity of the boundary region in the boundary-adjusted image block is lower than the filtering intensity of the non-boundary region. The texture information overlay module is used to obtain target texture information from the neighborhood of the filtered image block and overlay the target texture information onto the corresponding target boundary pixels in the filtered image block to complete the error repair of the target video frame.
[0012] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned video frame error repair method based on spatial domain filtering.
[0013] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned video frame error repair method based on spatial domain filtering.
[0014] This application first acquires a target video frame and identifies reconstructed image blocks within the target video frame. The reconstructed image blocks are obtained by reconstructing missing image blocks in the target video frame using spatial interpolation. Then, based on the distance between each pixel in the reconstructed image block and its boundary, several target boundary pixels are determined, and their pixel values are adjusted to obtain boundary-adjusted image blocks. The target boundary pixels are pixels in boundary regions where the distance to the boundary of the reconstructed image block is less than a preset distance threshold. Next, spatial filtering is performed on the boundary-adjusted image blocks to obtain corresponding filtered image blocks. The filtering intensity corresponding to the boundary regions in the boundary-adjusted image blocks is lower than the filtering intensity corresponding to the non-boundary regions. Finally, target texture information is obtained from the neighborhood of the filtered image blocks and superimposed onto the corresponding target boundary pixels in the filtered image blocks to complete the error repair of the target video frame. Therefore, this application, by adjusting the boundary pixel values according to the distance between the pixel and the boundary, to a certain extent reduces the hard seams between the reconstructed image block and the correctly decoded area; by performing spatial filtering on the image block after boundary adjustment, and making the filtering intensity of the boundary area lower than that of the non-boundary area, it eliminates interpolation noise while preserving edge details, thus solving the blurring problem caused by excessive smoothing; by obtaining texture information from the neighborhood and superimposing it on the boundary pixels, it transfers real high-frequency details to the reconstructed area, thus solving the problem of lacking natural texture. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0016] Figure 1 This is a flowchart of a video frame error repair method based on spatial domain filtering disclosed in this application; Figure 2 This is a schematic diagram of a video frame error repair device based on spatial domain filtering disclosed in this application; Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] See Figure 1 As shown, this embodiment of the invention discloses a video frame error repair method based on spatial domain filtering, applied at the decoding end, including: Step S11: Obtain the target video frame and determine the reconstructed image block in the target video frame; wherein, the reconstructed image block is an image block obtained by reconstructing the missing image block in the target video frame using spatial interpolation technology.
[0019] This embodiment proposes an error recovery filtering process based on a three-stage cascaded post-processing filter. The decoder performs a three-stage cascaded post-processing filter enhancement on the reconstructed region of the lost patch, effectively eliminating boundary seam artifacts, preserving edges and denoising, and migrating natural texture details from the already decoded neighborhood, significantly improving the visual quality of the error-hidden region.
[0020] The filtering enhancement is deployed after all loop filters (Deblock, SAO (Sample Adaptive Offset), ESAO (Enhanced Sample Adaptive Offset), CCSAO (Cross-Component Sample Adaptive Offset), and ALF (Adaptive Loop Filter)) at the decoding end and before image output. Only the patch regions marked as missing in the I-frame are subjected to the three-stage cascaded processing. The overall flow in this embodiment is shown in Table 1: Table 1. Solution Process Phase Table
[0021] In addition, the key parameters used in the three stages of this embodiment are shown in Table 2: Table 2 Key Parameters
[0022] The overall process in this embodiment is as follows: The post-processing filter input function ec_post_filter_lost_patches() first performs the following condition checks: Frame type check: Post-processing is only performed on I-frames (slice_type == SLICE_I). P / B frame loss patches are recovered using temporal copying, eliminating over-smoothing artifacts caused by spatial interpolation and requiring no post-processing.
[0023] Missing Patch Check: Traverse all Patch indices and use the array to determine which Patches were missing in the current frame and have undergone error hiding.
[0024] Patch information acquisition: Calculate the pixel-level boundary coordinates (left_pel, up_pel, right_pel, down_pel) of the patch based on its row and column position in the grid.
[0025] For each missing patch, process the three color components in sequence: Luminance component (Y): Perform three-level processing with the original parameters (i.e., the three-level processing flow in Table 1 above). Chromaticity components (Cb, Cr): Consider 4:2:0 chromaticity subsampling, coordinates and size divided by 2, blending band width and texture migration depth halved (minimum value of 2), texture migration intensity reduced to 50% of luminance.
[0026] It should be noted that in the AVS3 video codec standard, the ultra-low latency coding mode is designed for real-time video communication scenarios, requiring extremely low end-to-end latency. In this mode, video frames are divided into multiple independently coded patches, each of which can be independently encapsulated into one or more network data packets, achieving subframe-level transmission and decoding. In unreliable network transmission environments, some data packets may fail to arrive at the decoding end on time due to packet loss, latency jitter, etc. In this case, the corresponding patch data is lost, and the decoder needs to recover the pixel content of the lost area using error concealment techniques. Data loss can be divided into keyframe (I-frame) data loss and non-keyframe data loss, specifically: For missing I-frame patches: Since there are no reference frames available for I-frames, a spatial weighted interpolation error hiding method is adopted. The boundary pixels of the neighboring patches that have been correctly decoded in 8 directions (up, down, left, right, upper left, upper right, lower left, lower right) are used to fill the missing area through distance weighted interpolation.
[0027] For missing P / B frame patches: directly copy pixels from the corresponding position in the previous frame (temporal copy). Since the similarity between frames is usually high, the visual effect is relatively acceptable.
[0028] Accordingly, in this embodiment, after obtaining the target video frame, the method further includes: determining whether the target video frame is a key frame; if the target video frame is a key frame, then proceeding to the step of determining the reconstructed image block in the target video frame; if the target video frame is not a key frame, then obtaining the reference frame corresponding to the target video frame, and using the corresponding reference frame to perform video frame error repair on the target video frame.
[0029] That is, if a key frame loses data, the video frame error repair process in this embodiment is activated; if a non-key frame loses data, the corresponding reference frame is directly obtained, and data is obtained from the reference frame for error repair.
[0030] Step S12: Determine several target boundary pixels based on the distance between each pixel in the reconstructed image block and the boundary of the reconstructed image block, and adjust the pixel values of the target boundary pixels to obtain an image block after boundary adjustment; the target boundary pixels are pixels in the boundary region where the distance between them and the boundary of the reconstructed image block is less than a preset distance threshold.
[0031] In this embodiment, adjusting the pixel values of target boundary pixels to obtain a boundary-adjusted image patch includes: determining the target fusion coefficient corresponding to each target boundary pixel based on the distance between each target boundary pixel and the boundary of the reconstructed image patch; determining the target neighborhood radius corresponding to each target boundary pixel based on the distance between each target boundary pixel and the boundary of the reconstructed image patch; calculating the local mean corresponding to each target boundary pixel based on each target neighborhood radius; and fusing each local mean with the corresponding original target pixel value based on the target fusion coefficient to obtain a boundary-adjusted image patch; the original target pixel value is the original pixel value corresponding to the target boundary pixel.
[0032] Specifically, Phase 1, boundary seam fusion: When patch information is lost and incorrectly hidden, there is a significant discontinuity ("hard seam") between the pixel values at the hidden patch boundary and the correctly decoded neighboring pixels, resulting in noticeable blockiness. Before filtering, the original pixel values of the entire patch area are copied to a temporary buffer tmp to prevent data races caused by reading and writing to the same buffer during pixel-by-pixel modification.
[0033] For each pixel within the patch (i.e., each pixel in the reconstructed image patch) (gx, gy), calculate its minimum distance dist to the four patch boundaries (i.e., the boundaries of the reconstructed image patch). When dist < blend_w (i.e., the pixel is within the blending band), calculate the linear blending coefficients. (i.e., target fusion coefficient): = dist / blend_w; = 0 (at the boundary): Use the local mean avg entirely; → 1 (Inner edge of the fusion band): Close to using the original value orig; dist ≥ blend_w: The original value is completely preserved without any processing; The radius r of the calculated local mean (i.e., the target neighborhood radius) is dynamically adjusted based on the distance from the pixel to the boundary. r = clamp((blend_w - dist + 1) / 2, 1, 4); The closer to the boundary, the larger the neighborhood radius (maximum 4), and the smoother the mean; the farther away, the smaller the neighborhood (minimum 1), preserving more original details. The arithmetic mean avg (i.e., the local mean) is calculated within a (2r+1) × (2r+1) window.
[0034] Finally, pixel blending is performed: blended = × orig + (1 - ) × avg; output = CLIP3(0, max_val, blended); Where blended represents the merged pixels, and output represents the final merged pixels.
[0035] Step S13: Perform spatial filtering on the boundary-adjusted image block to obtain the corresponding filtered image block; the filtering intensity corresponding to the boundary region in the boundary-adjusted image block is lower than the filtering intensity corresponding to the non-boundary region.
[0036] In this embodiment, the step of performing spatial filtering on the boundary-adjusted image block to obtain the corresponding filtered image block includes: performing spatial filtering on the boundary-adjusted image block based on the local variance of the pixels in the boundary-adjusted image block to obtain the corresponding filtered image block.
[0037] The above-mentioned spatial filtering of the boundary-adjusted image block based on the local variance of pixels in the boundary-adjusted image block includes: determining a first weight corresponding to each pixel based on the spatial distance between each pixel and its corresponding neighboring pixels; calculating the local variance corresponding to each pixel in the boundary-adjusted image block within the neighborhood of the target size, and determining a second weight corresponding to each pixel based on the local variance corresponding to each pixel; calculating the target weight corresponding to each pixel based on the first weight and the second weight, and performing spatial filtering on the boundary-adjusted image block based on the target weight.
[0038] Specifically, Stage 2, Adaptive bilateral filtering: After spatial interpolation and stage 1 fusion, unnatural noise and smoothing inconsistencies still exist within the patch due to interpolation, requiring further global smoothing. Traditional Gaussian filtering blurs edges, while the computational complexity of standard bilateral filtering's exp() function is high, making it unsuitable for ULL scenarios.
[0039] We use rational functions to approximate the exponential operation of the traditional Gaussian function and pre-compute the spatial domain weight table: = radius (standard deviation of the spatial domain); ; For each offset (dx, dy): ; in, Here, is the spatial domain standard deviation (or spatial scale parameter), and radius is the filter radius (or spatial radius). To prevent the denominator from being zero, is the spatial normalization coefficient, sd is the squared spatial distance (or the squared neighborhood offset distance), and dx and dy are the horizontal and vertical offsets, respectively.
[0040] spatial_w[idx] (i.e., the first weight) = 1 / (1 + sd × inv_2ss); For each pixel, estimate the local variance within a 3×3 neighborhood: ; ; Where mean is the pixel average, and pixel is the pixel value. The sum of pixels is calculated, and count is the number of pixels. Let Variance be the variance.
[0041] `count` represents the total number of pixels in a 3x3 neighborhood, and the range parameter is dynamically adjusted based on the local variance. : = 3200 (10-bit) or 200 (8-bit); var_threshold = 800 (10-bit) or 50 (8-bit); adapt_factor = 1 + 2 / (1 + variance / var_threshold); ; in, The standard deviation of the range is the baseline value, var_threshold is the variance threshold, and adapt_factor is the adaptive factor. This is the adaptive range standard deviation (or final range variance).
[0042] Adaptive behavior: Low variance (smooth region): adapt_factor → 3, Increased size results in stronger filtering and elimination of interpolation noise; High variance (marginal regions): adapt_factor → 1, Reduce size to preserve edge details; Within a (2R+1) × (2R+1) window (R=2, i.e., a 5×5 window), calculate the combined weight w for each neighboring pixel: ; ; ; The above That is, the reciprocal of the second weight, w is the target weight, center_pixel is the center pixel, neighbor_pixel is the neighboring pixels, and result is the filtering result.
[0043] After the filtering results are stored in a temporary buffer, they are copied back to the original image buffer in batches.
[0044] Step S14: Obtain target texture information from the neighborhood of the filtered image block, and superimpose the target texture information onto the corresponding target boundary pixels in the filtered image block to complete the error repair of the target video frame.
[0045] In this embodiment, obtaining target texture information from the neighborhood of the filtered image block includes: determining the target texture source corresponding to each target boundary pixel based on the distance between each target boundary pixel and the boundary of the filtered image block, and obtaining the target texture information corresponding to each target boundary pixel from each target texture source.
[0046] In addition, superimposing the target texture information onto the corresponding target boundary pixels in the filtered image block includes: determining the target superposition intensity corresponding to each target texture information based on the distance between each target boundary pixel and the boundary of the filtered image block, and superimposing the target texture information onto the corresponding target boundary pixels in the filtered image block based on the target superposition intensity; wherein, the magnitude of the target superposition intensity is negatively correlated with the distance between the target and the target; the target distance is the distance between the target edge pixel and the boundary of the filtered image block.
[0047] Specifically, stage 3, texture detail transfer: After processing in stages 1 and 2, the patch boundary seams have been eliminated and the internal noise has been smoothed. However, the hidden areas still appear "overly smooth," lacking the high-frequency texture information (such as texture details of grass, walls, and fabrics) that a natural image should have. Texture information needs to be introduced from the correctly decoded neighboring areas to restore a natural visual appearance.
[0048] Only pixels within a depth range near the patch boundary are processed (default depth=12). Pixels outside this range are not processed because the texture relevance of the migration decreases as the distance from the neighborhood increases.
[0049] For the current pixel (gx, gy), calculate its distance to the four patch boundaries, determine the direction of the nearest boundary, and calculate the coordinates of the source pixel (i.e., the target texture source) at the symmetrical (mirrorized) position outside the patch. The core idea of mirroring is that the texture source corresponding to a point within the patch at a distance of dist from the boundary is the decoded pixel outside the patch at the same distance of dist. This is based on the assumption that "the textures of the neighborhood region and the hidden region are spatially continuous".
[0050] At the source pixel location (src_x, src_y), extract the high-frequency texture components.
[0051] src_val = buf[src_y][src_x]; avg = mean(3×3 neighborhood around (src_x, src_y)); HF = src_val – avg; Where src_val is the source pixel value, avg is the average value of neighboring pixels, and HF is the high-frequency texture component.
[0052] The high-frequency component is defined as the pixel value minus its 3×3 local mean. This high-frequency component represents the texture detail signal at the source location: positive values indicate local bright spots, and negative values indicate local dark spots. Texture migration intensity. The target overlay intensity (i.e., the intensity of the overlay) decreases as the distance from the pixel to the boundary increases, using a quadratic attenuation function: ; Where dist represents the distance from the boundary (i.e., the distance between each target boundary pixel and the boundary of the filtered image patch), the weighted high-frequency components are superimposed on the current pixel: enhanced = cur_val + strength × decay × HF; output = CLIP3(0, max_val, enhanced); Where strength is the global migration strength coefficient (default 0.25), which controls the overall amplitude of texture overlay, enhanced is the enhanced pixel value, and output is the output pixel value.
[0053] Therefore, this application, by adjusting the boundary pixel values according to the distance between the pixel and the boundary, to a certain extent reduces the hard seams between the reconstructed image block and the correctly decoded area; by performing spatial filtering on the image block after boundary adjustment, and making the filtering intensity of the boundary area lower than that of the non-boundary area, it eliminates interpolation noise while preserving edge details, thus solving the blurring problem caused by excessive smoothing; by obtaining texture information from the neighborhood and superimposing it on the boundary pixels, it transfers real high-frequency details to the reconstructed area, thus solving the problem of lacking natural texture.
[0054] See Figure 2 As shown, this embodiment of the invention discloses a video frame error repair device based on spatial domain filtering, applied at the decoding end, comprising: The video frame acquisition module 11 is used to acquire a target video frame and determine the reconstructed image block in the target video frame; wherein the reconstructed image block is an image block obtained by reconstructing the missing image block in the target video frame using spatial interpolation technology; The boundary pixel determination module 12 is used to determine a number of target boundary pixels based on the distance between each pixel in the reconstructed image block and the boundary of the reconstructed image block, and to adjust the pixel values of the target boundary pixels to obtain an image block after boundary adjustment; the target boundary pixels are pixels in the boundary region where the distance between them and the boundary of the reconstructed image block is less than a preset distance threshold. Image filtering module 13 is used to perform spatial filtering on the boundary-adjusted image block to obtain a corresponding filtered image block; the filtering intensity corresponding to the boundary region in the boundary-adjusted image block is lower than the filtering intensity corresponding to the non-boundary region. The texture information overlay module 14 is used to obtain target texture information from the neighborhood of the filtered image block and overlay the target texture information onto the corresponding target boundary pixels in the filtered image block to complete the error repair of the target video frame.
[0055] In some specific embodiments, the video frame acquisition module 11 further includes: The step jump unit is used to determine whether the target video frame is a key frame. If the target video frame is a key frame, the jump unit will proceed to the step of determining the reconstructed image block in the target video frame. The error repair unit is used to obtain a reference frame corresponding to the target video frame if the target video frame is a non-critical frame, and to use the corresponding reference frame to perform video frame error repair on the target video frame.
[0056] In some specific embodiments, the boundary pixel determination module 12 may specifically include: The neighborhood radius determination unit is used to determine the target fusion coefficient corresponding to each target boundary pixel based on the distance between each target boundary pixel and the boundary of the reconstructed image block, and to determine the target neighborhood radius corresponding to each target boundary pixel based on the distance between each target boundary pixel and the boundary of the reconstructed image block; The data fusion unit is used to calculate the local mean value corresponding to each of the target boundary pixels according to the radius of each target neighborhood, and to fuse each of the local mean values and the corresponding original target pixel values according to the target fusion coefficient to obtain the image block after boundary adjustment; the original target pixel value is the original pixel value corresponding to the target boundary pixel.
[0057] In some specific embodiments, the image filtering module 13 may specifically include: The image filtering submodule is used to perform spatial filtering on the boundary-adjusted image block based on the local variance of the pixels in the boundary-adjusted image block to obtain the corresponding filtered image block.
[0058] In some specific embodiments, the image filtering submodule may specifically include: The first weight determination unit is used to determine the first weight corresponding to each pixel based on the spatial distance between each pixel and its corresponding neighboring pixels. The second weight determination unit is used to calculate the local variance of each pixel in the boundary-adjusted image block within the neighborhood of the target size, and determine the second weight of each pixel based on the local variance of each pixel. The image filtering unit is used to calculate the target weight corresponding to each pixel according to the first weight and the second weight, and to perform spatial filtering on the boundary-adjusted image block according to the target weight.
[0059] In some specific embodiments, the texture information overlay module 14 may specifically include: The texture information acquisition unit is used to determine the target texture source corresponding to each target boundary pixel based on the distance between each target boundary pixel and the boundary of the filtered image block, and to acquire the target texture information corresponding to each target boundary pixel from each target texture source.
[0060] In some specific embodiments, the texture information overlay module 14 may specifically include: The texture information overlay unit is used to determine the target overlay intensity corresponding to each of the target texture information based on the distance between each target boundary pixel and the boundary of the filtered image block, and to overlay the target texture information onto the corresponding target boundary pixel in the filtered image block according to the target overlay intensity; wherein, the magnitude of the target overlay intensity is negatively correlated with the distance between the target and the target; the target distance is the distance between the target edge pixel and the boundary of the filtered image block.
[0061] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0062] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the video frame error repair method based on spatial domain filtering disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.
[0063] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0064] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0065] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the video frame error repair method based on spatial filtering as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.
[0066] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned video frame error repair method based on spatial domain filtering. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0067] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0068] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0069] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0070] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0071] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A video frame error repair method based on spatial domain filtering, characterized in that, Applied to the decoding end, including: Acquire a target video frame and determine the reconstructed image block in the target video frame; wherein the reconstructed image block is an image block obtained by reconstructing the missing image block in the target video frame using spatial interpolation technology; Several target boundary pixels are determined based on the distance between each pixel in the reconstructed image block and the boundary of the reconstructed image block, and the pixel values of the target boundary pixels are adjusted to obtain an image block with adjusted boundary; the target boundary pixels are pixels in the boundary region where the distance between them and the boundary of the reconstructed image block is less than a preset distance threshold. Spatial filtering is performed on the boundary-adjusted image block to obtain the corresponding filtered image block; the filtering intensity of the boundary region in the boundary-adjusted image block is lower than the filtering intensity of the non-boundary region. The target texture information is obtained from the neighborhood of the filtered image block, and the target texture information is superimposed on the corresponding target boundary pixels in the filtered image block to complete the error repair of the target video frame.
2. The video frame error repair method based on spatial domain filtering according to claim 1, characterized in that, After acquiring the target video frame, the process further includes: Determine whether the target video frame is a keyframe. If the target video frame is a keyframe, proceed to the step of determining the reconstructed image block in the target video frame. If the target video frame is not a key frame, then the reference frame corresponding to the target video frame is obtained, and the target video frame is repaired using the corresponding reference frame.
3. The video frame error repair method based on spatial domain filtering according to claim 1, characterized in that, The step of adjusting the pixel values of the target boundary pixels to obtain the image patch after boundary adjustment includes: The target fusion coefficient corresponding to each target boundary pixel is determined based on the distance between each target boundary pixel and the boundary of the reconstructed image block, and the target neighborhood radius corresponding to each target boundary pixel is determined based on the distance between each target boundary pixel and the boundary of the reconstructed image block. The local mean value corresponding to each of the target boundary pixels is calculated based on the radius of each target neighborhood, and the local mean value and the corresponding original target pixel value are fused according to the target fusion coefficient to obtain the image block after boundary adjustment; the original target pixel value is the original pixel value corresponding to the target boundary pixel.
4. The video frame error repair method based on spatial domain filtering according to claim 1, characterized in that, The step of performing spatial filtering on the boundary-adjusted image patch to obtain the corresponding filtered image patch includes: Spatial filtering is performed on the boundary-adjusted image block based on the local variance of the pixels in the boundary-adjusted image block to obtain the corresponding filtered image block.
5. The video frame error repair method based on spatial domain filtering according to claim 4, characterized in that, The step of performing spatial filtering on the boundary-adjusted image patch based on the local variance of pixels in the boundary-adjusted image patch includes: The first weight corresponding to each pixel is determined based on the spatial distance between each pixel and its corresponding neighboring pixels. Calculate the local variance of each pixel in the boundary-adjusted image block within the neighborhood of the target size, and determine the second weight of each pixel based on the local variance of each pixel. Calculate the target weight corresponding to each pixel based on the first weight and the second weight, and perform spatial filtering on the boundary-adjusted image block based on the target weight.
6. The video frame error repair method based on spatial domain filtering according to claim 1, characterized in that, The step of obtaining target texture information from the neighborhood of the filtered image patch includes: The target texture source corresponding to each target boundary pixel is determined based on the distance between each target boundary pixel and the boundary of the filtered image block, and the target texture information corresponding to each target boundary pixel is obtained from each target texture source.
7. The video frame error repair method based on spatial domain filtering according to claim 1, characterized in that, The step of superimposing the target texture information onto the corresponding target boundary pixels in the filtered image block includes: The target superposition intensity corresponding to each target texture information is determined based on the distance between each target boundary pixel and the boundary of the filtered image block, and the target texture information is superimposed onto the corresponding target boundary pixel in the filtered image block according to the target superposition intensity; wherein, the magnitude of the target superposition intensity is negatively correlated with the distance between the target and the target; the target distance is the distance between the target edge pixel and the boundary of the filtered image block.
8. A video frame error repair device based on spatial domain filtering, characterized in that, Applied to the decoding end, including: A video frame acquisition module is used to acquire a target video frame and determine the reconstructed image block in the target video frame; wherein, the reconstructed image block is an image block obtained by reconstructing the missing image block in the target video frame using spatial interpolation technology; A boundary pixel determination module is used to determine a number of target boundary pixels based on the distance between each pixel in the reconstructed image block and the boundary of the reconstructed image block, and to adjust the pixel values of the target boundary pixels to obtain an image block after boundary adjustment; the target boundary pixels are pixels in the boundary region where the distance between them and the boundary of the reconstructed image block is less than a preset distance threshold. The image filtering module is used to perform spatial filtering on the boundary-adjusted image block to obtain a corresponding filtered image block; the filtering intensity of the boundary region in the boundary-adjusted image block is lower than the filtering intensity of the non-boundary region. The texture information overlay module is used to obtain target texture information from the neighborhood of the filtered image block and overlay the target texture information onto the corresponding target boundary pixels in the filtered image block to complete the error repair of the target video frame.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the video frame error repair method based on spatial filtering as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the video frame error repair method based on spatial domain filtering as described in any one of claims 1 to 7.