Video data error hiding method based on sparse coding and dictionary learning

Abstract

The invention discloses a video data error hiding method based on sparse coding and dictionary learning, and relates to the technical field of image processing, and the method comprises the steps: obtaining a damaged image block in to-be-processed video data; based on a preset sparse dictionary, performing sparse coding operation on the damaged image block, and determining an initial sparse coefficient; based on neighborhood information of the damaged image block, constructing an adaptive weighted constraint term; adding the adaptive weighted constraint term into an optimization model constructed based on the initial sparse coefficient, carrying out a joint optimization solution operation, and determining a target sparse coefficient; and performing image reconstruction operation based on the target sparse coefficient and a preset sparse dictionary, and determining a reconstructed image block.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for hiding video data errors based on sparse coding and dictionary learning. Background Technology

[0002] In digital video communication and storage systems, due to unstable network transmission environments or damaged storage media, video data is prone to packet loss or bit errors during transmission or retrieval, resulting in image block corruption or information loss in decoded video frames. These errors degrade the subjective and objective quality of the video, impacting user experience. Traditional error concealment techniques often employ temporal or spatial interpolation, extrapolation, or motion compensation methods, utilizing correctly decoded pixels surrounding the damaged area to estimate and recover lost information. However, these methods often struggle to maintain visual coherence when dealing with complex textures or large-area damage, easily introducing noticeable block artifacts or blurring. Therefore, a video data error concealment method based on sparse coding and dictionary learning is urgently needed to address the aforementioned problems. Summary of the Invention

[0003] The summary section introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. This summary section is not intended to limit the key and essential technical features of the claimed technical solutions, nor is it intended to determine the scope of protection of the claimed technical solutions.

[0004] Firstly, this application provides a video data error concealment method based on sparse coding and dictionary learning, including: Retrieve damaged image blocks from the video data to be processed; Based on a pre-defined sparse dictionary, sparse coding operations are performed on damaged image blocks to determine the initial sparse coefficients. An adaptive weighted constraint term is constructed based on the neighborhood information of the damaged image patch; The adaptive weighted constraint term is added to the optimization model constructed based on the initial sparse coefficients, and a joint optimization solution operation is performed to determine the target sparse coefficients. Based on the target sparse coefficients and the preset sparse dictionary, an image reconstruction operation is performed to determine the reconstructed image blocks.

[0005] In some implementations, acquiring damaged image blocks in the video data to be processed includes: Based on the decoding results of the video data to be processed, determine the location information of the damaged areas in the video frame; Based on the location information of the damaged area, a set of candidate image patches containing the damaged area is extracted from the video frame; The size of each candidate image block in the candidate image block set is normalized based on a preset size to determine the damaged image block.

[0006] In some implementations, the step of performing sparse coding operations on the damaged image blocks based on a preset sparse dictionary to determine initial sparse coefficients includes: Based on the damaged image patch, a matching preset sparse dictionary is selected from a preset dictionary set; Based on the preset sparse dictionary and the damaged image patch, the initial sparse coefficients are determined by performing a sparse decomposition operation using an orthogonal matching pursuit algorithm.

[0007] In some implementations, constructing an adaptive weighted constraint term based on the neighborhood information of the damaged image patch includes: Based on the spatial location of the damaged image block, multiple neighboring image blocks are extracted from the video data to be processed; Based on the damaged image block and each of the neighboring image blocks, calculate the similarity weight between each of the neighboring image blocks and the damaged image block; The adaptive weighted constraint term is constructed based on the similarity weights corresponding to each of the neighboring image blocks.

[0008] In some implementations, the step of adding the adaptive weighted constraint term to the optimization model constructed based on the initial sparse coefficients and performing joint optimization to determine the target sparse coefficients includes: Based on the initial sparse coefficients and the adaptive weighted constraint term, a joint objective function is constructed that includes a data fidelity term, a sparse regularization term, and a weighted constraint term; Based on the iterative reweighted least squares method, the joint objective function is optimized and solved to determine the sparse coefficients after iterative update. Based on the preset convergence condition, determine whether the sparse coefficients after the iterative update meet the convergence requirements; If the convergence requirement is met, the iteratively updated sparse coefficients that meet the convergence requirement will be determined as the target sparse coefficients.

[0009] In some implementations, the step of performing image reconstruction based on the target sparse coefficients and the preset sparse dictionary to determine the reconstructed image patches includes: Based on the target sparse coefficients and the preset sparse dictionary, a linear combination operation is performed to determine the initial reconstructed image blocks; Based on the neighborhood information of the damaged image block, post-processing operations are performed on the initial reconstructed image block to determine the optimized reconstructed image block; Based on the original frame size of the video data to be processed, the optimized reconstructed image block is subjected to a size restoration operation to determine the reconstructed image block.

[0010] In some implementations, it also includes: The reconstructed image blocks replace the damaged image blocks in the video data to be processed, generating video data with the error hidden.

[0011] Secondly, this application proposes a video data error concealment device based on sparse coding and dictionary learning, the device comprising: The damaged image acquisition unit is used to acquire damaged image blocks in the video data to be processed; The sparse coefficient determination unit is used to perform sparse coding operations on the damaged image block based on a preset sparse dictionary to determine the initial sparse coefficients. A weighted constraint construction unit is used to construct an adaptive weighted constraint term based on the neighborhood information of the damaged image patch; The target coefficient determination unit is used to add the adaptive weighted constraint term to the optimization model constructed based on the initial sparse coefficients, perform joint optimization solution operation, and determine the target sparse coefficients. The image reconstruction unit is used to perform image reconstruction operations based on the target sparse coefficients and the preset sparse dictionary, and to determine the reconstructed image blocks.

[0012] Thirdly, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program stored in the memory to implement the steps of the video data error concealment method based on sparse coding and dictionary learning described in any of the first aspects.

[0013] Fourthly, this application also proposes a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the video data error concealment method based on sparse coding and dictionary learning according to any one of the first aspects.

[0014] In summary, the video data error concealment method based on sparse coding and dictionary learning proposed in this application effectively utilizes the surrounding information of damaged image patches to guide the reconstruction process by introducing an adaptive weighted constraint term based on neighborhood information and combining it with the sparse coding process for joint optimization. This improves the accuracy and visual coherence of damaged region reconstruction. The method can reduce block artifacts and blurring while preserving image texture details. Attached Figure Description

[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a video data error concealment method based on sparse coding and dictionary learning is provided for embodiments of this application. Figure 2 A schematic diagram of a video data error concealment device based on sparse coding and dictionary learning is provided for an embodiment of this application; Figure 3 This is a schematic diagram of an electronic device structure for hiding video data errors based on sparse coding and dictionary learning, provided as an embodiment of this application. Detailed Implementation

[0016] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0017] Please see Figure 1 This is a flowchart illustrating a video data error concealment method based on sparse coding and dictionary learning, provided in an embodiment of this application. Specifically, it may include: S110. Obtain damaged image blocks from the video data to be processed; For example, the system locates and extracts specific regions requiring repair from video streams containing transmission or storage errors. This process begins by decoding the received video stream. During decoding, the system detects and records the location information of damaged regions, such as lost macroblocks or failed checksums, caused by packet loss or bit errors. Based on this location information, the system crops image blocks containing these damaged regions from the decoded video frames. To accommodate subsequent sparse coding and dictionary learning processes, these extracted image blocks are typically normalized to a preset standard size, forming a uniform set of damaged image blocks.

[0018] S120. Based on a preset sparse dictionary, perform sparse coding on the damaged image blocks to determine the initial sparse coefficients. For example, based on the texture characteristics of the current damaged image patch, the best-matching preset sparse dictionary is selected from a pre-generated and stored dictionary set. This dictionary is typically an overcomplete dictionary obtained through machine learning training on a large number of lossless video image patches, capable of effectively representing various image structural features. After selecting the dictionary, the orthogonal matching pursuit (ORP) greedy algorithm is used to sparsely decompose the damaged image patch. This algorithm iteratively selects the atoms in the dictionary most relevant to the current residual, gradually approximating the original signal, and finally obtains a set of coefficients that can approximate the damaged patch with a linear combination of the fewest atoms, i.e., the initial sparse coefficients. This process transforms the high-dimensional pixel information of the damaged patch into a low-dimensional sparse representation.

[0019] S130. Based on the neighborhood information of the damaged image patch, construct an adaptive weighted constraint term; For example, constraints guiding the reconstruction process are constructed by mining the spatial correlation between damaged image patches and their surrounding valid pixel regions. Specifically, firstly, based on the spatial coordinates of the damaged patch in the video frame, multiple complete, undamaged image patches are extracted from its spatiotemporal neighborhood as references. Subsequently, by calculating the similarity metric between the damaged patch and each neighboring reference patch at the feature level, an adaptive weight factor is assigned to each reference patch. This weight factor reflects the contribution of the reference patch to the reconstruction of the current damaged region. Finally, based on these weight factors, a mathematical constraint term is constructed for the subsequent joint optimization process. This constraint term ensures that the reconstruction result maintains statistical consistency with the valid information of the surrounding neighborhood within the sparse representation domain.

[0020] S140. Add the adaptive weighted constraint term to the optimization model constructed based on the initial sparse coefficients, perform joint optimization solution operation, and determine the target sparse coefficients. For example, based on the initial sparse coefficients obtained in the previous step, an adaptive weighted constraint term constructed based on neighborhood similarity is incorporated to form an objective function containing a data fidelity term, a sparsity constraint term, and a weighted prior term. The objective function is solved using an iterative reweighted least squares method. This algorithm iteratively updates the sparse coefficients and weights, gradually approaching the optimal solution. The iterative process continues until the changes in the sparse coefficients satisfy a preset convergence condition; the stable solution obtained at this point is the final target sparse coefficient.

[0021] S150. Based on the target sparse coefficients and the preset sparse dictionary, perform image reconstruction operation to determine the reconstructed image blocks.

[0022] For example, image reconstruction is performed using the linear synthesis properties of sparse representation. By using the optimized target sparse coefficients as weights and linearly combining them with atoms in a pre-defined sparse dictionary, the pixel value estimates of the damaged regions are calculated, forming an initial reconstructed image patch. This reconstruction process essentially maps the signal from the sparse domain back to the original pixel domain. An overcomplete dictionary can approximate various image structures with high precision through the linear combination of base atoms. This method obtains target sparse coefficients that better reflect the true structure of the image through joint optimization, thereby ensuring the visual coherence and structural integrity of the reconstructed image patch.

[0023] In summary, this application provides a video data error hiding method based on sparse coding and dictionary learning. This method acquires damaged image blocks from video data and uses a pre-set sparse dictionary to sparsely encode them to obtain initial sparse coefficients. Then, it combines the neighborhood information of the damaged blocks to construct an adaptive weighted constraint term. This constraint term is introduced into an optimization model for joint solution to obtain the target sparse coefficients. Finally, the recovered image blocks are obtained through image reconstruction. This method integrates prior neighborhood structure into the sparse reconstruction process in the form of adaptive weighted constraints. This allows the reconstruction process to not only rely on the prior knowledge of the dictionary but also fully utilize the effective information around the damaged area for guidance. This improves the consistency between the reconstructed image blocks and the original undamaged areas in terms of texture details and visual structure, effectively mitigating the block artifacts and blurring phenomena common in traditional methods, and obtaining higher-quality, more visually natural error hiding results.

[0024] In some instances, damaged image patches are extracted from the video data to be processed, including: Based on the decoding results of the video data to be processed, determine the location information of the damaged areas in the video frames; Based on the location information of the damaged area, a set of candidate image patches containing the damaged area is extracted from the video frame; Based on a preset size, the size of each candidate image block in the candidate image block set is normalized to determine the damaged image block.

[0025] For example, acquiring damaged image blocks from the video data to be processed is the initial step in the error concealment process, and its accuracy and standardization directly affect the effect of subsequent processing. This process involves decoding the video bitstream that contains transmission or storage errors. When processing the bitstream, the decoder detects damaged areas such as lost macroblocks or damaged slices due to packet loss, check failure, or bit errors through verification mechanisms (such as CRC check) or syntax analysis, and records the position coordinates of these damaged areas in the video frames. This position information is usually in the form of macroblock indexes, pixel coordinate ranges, or region masks.

[0026] Based on the location information of the damaged regions provided by decoding, the system extracts a set of candidate image patches containing these damaged regions from the decoded and reconstructed video frames. The extraction operation typically uses the center coordinates or top-left corner coordinates of the damaged region as a reference, and follows a certain strategy (such as expanding outwards from the damaged pixel) to crop an image patch larger than or equal to the damaged region, ensuring complete coverage of all pixels to be repaired. For damaged regions located at frame edges, specific boundary processing strategies are employed, such as mirror padding, constant value padding, or cropping only the valid region, to ensure that all candidate image patches have the same dimensional structure, facilitating subsequent batch processing.

[0027] To meet the requirement of consistent input data size for subsequent sparse coding and dictionary learning algorithms, the extracted candidate image patch set undergoes size normalization. Based on a preset standard size (e.g., 8×8, 16×16 pixels), image scaling, cropping, or padding operations are used to uniformly transform candidate image patches of potentially different sizes to this preset size. After this step, a batch of damaged image patches with uniform size specifications is finally obtained, which can be used for subsequent sparse representation.

[0028] In some instances, based on a pre-defined sparse dictionary, sparse coding operations are performed on damaged image patches to determine initial sparse coefficients, including: Based on the damaged image patch, a matching preset sparse dictionary is selected from the preset dictionary set; Based on a pre-defined sparse dictionary and damaged image patches, sparse decomposition is performed using an orthogonal matching pursuit algorithm to determine the initial sparse coefficients.

[0029] For example, based on the texture characteristics, structural orientation, or statistical features exhibited by the damaged image patch to be repaired, a pre-set sparse dictionary that best matches the damaged image patch is selected from a pre-trained and stored set of sub-dictionaries. This pre-set dictionary set typically contains multiple overcomplete dictionaries specifically trained for different image texture types (such as smooth regions, edges, and texture-rich regions). Each dictionary is trained using dictionary learning algorithms such as K-SVD on a large number of lossless video image patches, effectively representing a specific type of image structure prior. The selection process is achieved by calculating the overall representation error or similarity measure between the damaged image patch and each dictionary in the dictionary set, and finally selecting the dictionary with the smallest representation error as the pre-set sparse dictionary used for the current sparse coding operation.

[0030] After selecting a preset sparse dictionary, an orthogonal matching pursuit algorithm is used to perform sparse decomposition on the damaged image patch to determine the initial sparse coefficients. This algorithm is an iterative greedy selection process. The initial residual is the damaged image patch itself. In each iteration, the algorithm calculates the inner product of the current residual and all atoms in the preset sparse dictionary, and selects the atom with the largest absolute value of the inner product, i.e., the atom most relevant to the current residual, and adds it to the support atom set. Subsequently, based on the currently selected atom set (support atom set), the algorithm solves a system of linear equations using the least squares method to calculate a new set of coefficients. This new set of coefficients ensures that the linear combination of these selected atoms best approximates the original damaged image patch, and updates the residual to the difference between the original signal and the current linear combination. This iterative process continues until the norm of the residual is below a preset threshold or the preset sparsity (i.e., the number of selected atoms) is reached. When the iteration terminates, the resulting coefficient vector is the initial sparse coefficient. This coefficient vector has values ​​at the selected atom positions and is zero at other positions, thus achieving a sparse representation of the damaged image patch.

[0031] In summary, this embodiment of the application selects the best-matching preset sparse dictionary for different damaged image patches and uses the orthogonal matching pursuit algorithm to efficiently calculate their sparse representations, ensuring that the initial sparse coefficients can accurately capture the main structural information of the image patches. This process transforms the damaged patches from a high-dimensional pixel space to a low-dimensional sparse space, providing a good starting point for subsequent optimization and laying the foundation for fully utilizing prior structural knowledge of the image for high-quality reconstruction, effectively improving the robustness and accuracy of the entire error concealment process.

[0032] In some instances, adaptive weighted constraints are constructed based on neighborhood information of damaged image patches, including: Based on the spatial location of the damaged image patch, multiple neighboring image patches are extracted from the video data to be processed; Based on the damaged image patch and each neighboring image patch, calculate the similarity weight between each neighboring image patch and the damaged image patch; An adaptive weighted constraint term is constructed based on the similarity weights corresponding to each neighboring image patch.

[0033] For example, based on the coordinate information of the damaged image patch in the current video frame, its spatial neighborhood is determined. This neighborhood is typically defined by a preset neighborhood window size, such as a rectangular area extending outwards by several pixels from the damaged patch. From this neighborhood, the system selects intact image patches that are not marked as damaged as candidate neighborhood image patches. To ensure that the extracted neighborhood image patches effectively represent the surrounding environment of the damaged area, the extraction process must avoid frame boundaries and span adjacent frames to introduce temporal information, thereby obtaining a set of neighborhood image patches containing multiple valid reference patches.

[0034] Based on the damaged image patch and its neighboring image patches, a similarity weight is calculated for each neighboring image patch. This calculation is achieved by comparing the distance or similarity between the damaged image patch and each candidate neighboring image patch in the feature space. Typically, grayscale statistical features, gradient features, or higher-level texture features of the image patches are extracted, and a similarity metric between the two image patches is calculated based on these features, such as calculating the reciprocal of their Euclidean distance, cosine similarity, or structural similarity index (SSIM). Subsequently, the calculated similarity metric is converted into a normalized weight value using a predefined mapping function (such as an exponential decay function or a softmax function). This weight value quantitatively characterizes the degree of visual content correlation between the neighboring image patch and the damaged image patch; a higher weight indicates a higher reference value for the reconstruction of the damaged patch.

[0035] An adaptive weighted constraint term is constructed based on the similarity weights corresponding to each neighboring image patch. This construction process aims to transform the weight information, calculated in the previous step and reflecting the importance of each neighboring patch, into a mathematical constraint term that can be used in the subsequent sparse coefficient optimization model. This constraint term is typically constructed as a regularization term in the form of a weighted norm, ensuring that the final target sparse coefficients not only sparsely represent the damaged block, but also that the representation result (i.e., the reconstructed image patch) is as similar as possible to those high-weight neighboring patches in the feature space. Specifically, this constraint term uses the difference between the sparse representation (or some transformation thereof) of each neighboring patch and the sparse representation of the current damaged block, with their corresponding similarity weights as weighting coefficients, and incorporates the weighted sum into the overall objective function. In this way, the reconstruction process is guided to preferentially approach those surrounding regions most similar to the damaged block, thus effectively integrating the prior knowledge of spatial neighborhood continuity into the optimization framework of sparse reconstruction in an adaptive weighted form.

[0036] In summary, this application's embodiments introduce an adaptive, data-driven spatial prior constraint into the sparse reconstruction process by quantifying and utilizing the similarity relationship between damaged blocks and their surrounding valid information. This constraint can dynamically adjust the optimization direction, ensuring that the reconstruction results not only meet the requirements of sparsity and data fidelity but also maintain maximum coherence and consistency with the visual content of the surrounding undamaged areas. This improves the visual quality performance of the reconstructed image blocks and effectively suppresses the block artifacts and texture discontinuities commonly found in traditional methods.

[0037] In some instances, adaptive weighted constraints are incorporated into the optimization model built upon the initial sparse coefficients, and joint optimization is performed to determine the target sparse coefficients, including: Based on the initial sparse coefficients and adaptive weighted constraints, a joint objective function is constructed that includes a data fidelity term, a sparse regularization term, and a weighted constraint term. Based on the iterative reweighted least squares method, the joint objective function is optimized and the sparse coefficients after iterative update are determined. Based on the preset convergence conditions, determine whether the sparse coefficients after iterative update meet the convergence requirements; If the convergence requirement is met, the iteratively updated sparse coefficients that meet the convergence requirement will be determined as the target sparse coefficients.

[0038] For example, constructing a joint objective function is the foundation of the joint optimization solution operation. This process uses the initial sparse coefficients obtained in the previous step as the starting point for iteration, incorporating adaptive weighted constraint terms into the optimization framework. Specifically, the joint objective function consists of three parts: a data fidelity term constrains the fit between the reconstructed image signal and the original observed data (i.e., the available pixel information of the damaged image patch), typically the residual norm between the observed data and the linear combination of dictionary atoms and sparse coefficients; a sparsity regularization term ensures that the solved coefficient vector satisfies the sparsity prior, usually using the L1 norm to constrain the number of non-zero elements in the coefficient vector; and a weighted constraint term, an adaptive weighted constraint, is a weighted L2 norm constraint based on neighborhood similarity weights, used to ensure that the reconstruction result corresponding to the target sparse coefficients remains close to the sparse representation of the high-weighted neighborhood blocks in the feature space. These three terms are weighted and balanced through preset regularization parameters to form the optimization objective.

[0039] Based on the constructed joint objective function, an iterative reweighted least squares method is used for optimization. This algorithm iteratively updates the sparse coefficients and weight matrix alternately to handle non-smooth terms in the objective function. In each iteration, the algorithm first fixes the weight matrix, transforming the problem into a weighted least squares problem, and obtains the estimated sparse coefficients for the current iteration using numerical methods such as matrix inversion or the conjugate gradient method. Subsequently, the algorithm updates the weight values ​​in the weight matrix based on the obtained sparse coefficient estimates. The update rule for the weight values ​​is a function in the form of the reciprocal, exponential, or power function of the current coefficient estimate. Through such functions, the system can calculate larger weight values ​​for coefficients with smaller estimated values ​​in the current iteration. These larger weight values ​​will be applied to the weighted least squares solution in the next iteration, thus applying stronger penalty pressure to the positions with smaller coefficient values, causing these coefficients to further decay and tend towards zero in subsequent iterations, ultimately enhancing the overall sparsity of the solution vector. For newly introduced weighted constraint terms, their weight matrices are pre-determined by the neighborhood similarity weights calculated in the previous steps and remain fixed during iteration. This iterative process is executed repeatedly, gradually updating the estimated values ​​of the sparse coefficients.

[0040] After obtaining the updated sparse coefficients in each iteration, a convergence condition is used to determine whether the current solution meets the convergence requirement. The preset convergence condition is typically set as follows: the difference norm between the sparse coefficients obtained in two adjacent iterations is less than a very small positive threshold; the decrease in the objective function value is less than another preset threshold; or the number of iterations has reached a preset maximum number of iterations. Once any convergence condition is met, the updated sparse coefficients are considered stable, and the optimization loop exits.

[0041] If the iterative process meets the preset convergence requirements, the sparse coefficients obtained after the final iterative update that meet the convergence requirements are determined as the target sparse coefficients. These target sparse coefficients are the optimal sparse representation after a joint optimization process that balances multiple requirements such as data fidelity, inherent sparsity, and neighborhood similarity. Compared to the initial sparse coefficients, the target sparse coefficients not only more accurately characterize the essential structure of the damaged image patch, but also ensure that the corresponding reconstruction results maintain better visual coherence with the surrounding undamaged areas.

[0042] In summary, this embodiment transforms prior knowledge of the neighborhood into mathematical constraints using an adaptive weighting method and solves them using an iterative algorithm. This allows the final target sparse coefficients to fully utilize both the global representation capabilities of the dictionary and the local guidance of neighborhood information. This approach improves the accuracy of reconstructing damaged regions, especially in cases of complex textures or large damaged areas. It effectively preserves the detailed information of the reconstructed region and reduces visual artifacts, resulting in a higher-quality error-hiding effect that blends more naturally with the surrounding environment.

[0043] In some instances, image reconstruction is performed based on the target sparse coefficients and a pre-defined sparse dictionary to determine the reconstructed image patches, including: Based on the target sparse coefficients and the preset sparse dictionary, a linear combination operation is performed to determine the initial reconstructed image patch; Based on the neighborhood information of the damaged image patch, post-processing operations are performed on the initial reconstructed image patch to determine the optimized reconstructed image patch; Based on the original frame size of the video data to be processed, the size restoration operation is performed on the optimized reconstructed image blocks to determine the reconstructed image blocks.

[0044] For example, a linear combination operation is performed based on the target sparse coefficients and a preset sparse dictionary to determine the initial reconstructed image patch. This linear combination operation uses the optimized target sparse coefficients as a set of weights and performs a weighted summation with the corresponding atoms in the preset sparse dictionary. The preset sparse dictionary is an overcomplete set of atoms, each of which is essentially a vector representing a specific basic image structure (such as edges or texture primitives). Through this linear combination, the signal is mapped from the low-dimensional sparse representation domain back to the high-dimensional pixel domain, thereby calculating the estimated value of each pixel in the damaged region and generating the initial reconstructed image patch. This initial reconstructed image patch can already reflect the main structural information of the damaged region.

[0045] Based on neighborhood information of the damaged image patch, post-processing is performed on the initial reconstructed image patch to determine the optimized reconstructed image patch. Although the initial reconstructed image patch is structurally relatively complete, it may still have some inconsistencies with the surrounding areas in certain details, such as slight differences in brightness or color. Therefore, post-processing is performed to further improve its visual consistency. This post-processing operation uses the illumination and color statistical characteristics provided by the previously extracted undamaged neighboring image patches as a reference to filter or correct the initial reconstructed image patch. Specifically, by calculating the differences in statistical features between the initial reconstructed image patch and each neighboring image patch, and performing a weighted average based on the previously calculated similarity weights, the pixel values ​​of the initial reconstructed image patch are fine-tuned to make it more consistent with the most important neighboring patches in terms of overall brightness and color distribution, ultimately resulting in a visually more integrated and natural optimized reconstructed image patch.

[0046] Based on the original frame size of the video data to be processed, the optimized reconstructed image patch undergoes a size restoration operation to determine the reconstructed image patch. Since the damaged image patch underwent size normalization (e.g., scaling or cropping) to facilitate sparsity processing, the size of the optimized reconstructed image patch obtained at this point is the normalized preset size. To replace it back with the original video frame, an inverse transformation operation is required to restore it to its original size. This size restoration operation is performed inversely based on the algorithm parameters (e.g., scaling ratio, cropping position, or padding method) used during the initial size normalization process. If downsampling and scaling were performed during normalization, an upsampling interpolation algorithm (e.g., bicubic interpolation) is used for magnification; if cropping was performed, the repaired patch needs to be placed back at the original cropping position, and edge transition areas may need to be smoothed. After this step, a reconstructed image patch that perfectly matches the size of other areas in the original video frame and can be directly used to replace the damaged area, thus completing the error hiding, is obtained.

[0047] In summary, the embodiments of this application achieve high-quality reconstruction from sparse representation to the pixel domain through the aforementioned image reconstruction operations. This process not only utilizes the global structural prior provided by sparse coding, but also incorporates spatial context information from the local neighborhood through post-processing operations, and finally ensures compatibility between the reconstruction result and the original video frame through size restoration. The resulting reconstructed image patch can effectively replace the original damaged data, improving the overall viewing experience of the video after erroneous hiding.

[0048] In some instances, it also includes: The reconstructed image blocks replace the damaged image blocks in the video data to be processed, generating video data with the errors hidden.

[0049] For example, the reconstructed image blocks replace the damaged image blocks in the video data to be processed, generating the error-hidden video data. This process is the final output step of the error-hiding method, substantially repairing and replacing errors in the original video data to be processed based on the completed, high-quality reconstructed image blocks. Specifically, the system first accurately locates the exact coordinate range of the original damaged image block to be replaced in the corresponding video frame of the video data to be processed, based on the damaged area location information initially obtained from the decoding results. Subsequently, the reconstructed image blocks, whose size has been restored to match the original video frame after a series of steps including sparse coding, joint optimization, image reconstruction, and post-processing, are used to cover and replace all damaged pixel data within the coordinate range with pixel-level precision. This replacement operation ensures that the reconstructed area is spatially aligned with the original frame, achieving lossless updating. Finally, the system outputs a complete video data stream in which all damaged areas in the entire video data to be processed have been replaced by repaired image blocks, i.e., the error-hidden video data is generated. The generated video data maintains the original encoding format and timing structure while improving visual quality and eliminating visual defects caused by transmission or storage errors, providing a high-quality data foundation for subsequent video display, analysis, or storage applications.

[0050] Please see Figure 2 The diagram below illustrates a video data error concealment device based on sparse coding and dictionary learning, as provided in this application embodiment. The device includes: Damaged image acquisition unit 21 is used to acquire damaged image blocks in the video data to be processed; The sparse coefficient determination unit 22 is used to perform sparse coding operations on the damaged image blocks based on a preset sparse dictionary to determine the initial sparse coefficients. The weighted constraint construction unit 23 is used to construct adaptive weighted constraint terms based on the neighborhood information of the damaged image patch; The target coefficient determination unit 24 is used to add the adaptive weighted constraint terms to the optimization model built based on the initial sparse coefficients, perform joint optimization solution operations, and determine the target sparse coefficients. The image reconstruction generation unit 25 is used to perform image reconstruction operations based on the target sparse coefficients and a preset sparse dictionary, and to determine the reconstructed image blocks.

[0051] Please see Figure 3 This application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, it implements the steps of a video data error hiding method based on sparse coding and dictionary learning.

[0052] Since the electronic device described in this embodiment is the device used to implement the video data error hiding device based on sparse coding and dictionary learning in the embodiments of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiments of this application is within the scope of protection of this application.

[0053] In practice, when the computer program 311 is executed by the processor, it can implement any of the embodiments corresponding to the first aspect.

[0054] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0055] Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media containing computer-readable program code.

[0056] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] This application also provides a computer program product, which includes computer software instructions that, when executed on a processing device, cause the processing device to perform... Figure 1 The flowchart of the video data error concealment method based on sparse coding and dictionary learning in the corresponding embodiment.

[0060] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any usable medium that a computer can store or a data storage device such as a server or data center that integrates one or more usable media. The usable medium may be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0061] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0062] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Furthermore, the mutual couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0063] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0064] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in the form of hardware and / or software functional units.

[0065] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, magnetic disks, or optical disks.

[0066] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0067] Although preferred embodiments have been described in this specification, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications that fall outside the scope of this specification.

[0068] Obviously, those skilled in the art can make various modifications to this specification without departing from its spirit and scope. Therefore, this specification also intends to include any modifications that fall within the scope of the claims and their equivalents.

Claims

1. A video data error concealment method based on sparse coding and dictionary learning, characterized in that, The methods include: Retrieve damaged image blocks from the video data to be processed; Based on a preset sparse dictionary, sparse coding operations are performed on the damaged image blocks to determine the initial sparse coefficients; Based on the neighborhood information of the damaged image patch, an adaptive weighted constraint term is constructed; The adaptive weighted constraint term is added to the optimization model constructed based on the initial sparse coefficients, and a joint optimization solution operation is performed to determine the target sparse coefficients. Based on the target sparse coefficients and the preset sparse dictionary, an image reconstruction operation is performed to determine the reconstructed image blocks.

2. The method according to claim 1, characterized in that, The process of acquiring damaged image blocks in the video data to be processed includes: Based on the decoding results of the video data to be processed, determine the location information of the damaged areas in the video frame; Based on the location information of the damaged area, a set of candidate image patches containing the damaged area is extracted from the video frame; The size of each candidate image block in the candidate image block set is normalized based on a preset size to determine the damaged image block.

3. The method according to claim 1, characterized in that, The step of performing sparse coding on the damaged image blocks based on a preset sparse dictionary to determine initial sparse coefficients includes: Based on the damaged image patch, a matching preset sparse dictionary is selected from a preset dictionary set; Based on the preset sparse dictionary and the damaged image patch, the initial sparse coefficients are determined by performing a sparse decomposition operation using an orthogonal matching pursuit algorithm.

4. The method according to claim 1, characterized in that, The adaptive weighted constraint term is constructed based on the neighborhood information of the damaged image patch, including: Based on the spatial location of the damaged image block, multiple neighboring image blocks are extracted from the video data to be processed; Based on the damaged image block and each of the neighboring image blocks, calculate the similarity weight between each of the neighboring image blocks and the damaged image block; The adaptive weighted constraint term is constructed based on the similarity weights corresponding to each of the neighboring image blocks.

5. The method according to claim 1, characterized in that, The step of adding the adaptive weighted constraint term to the optimization model constructed based on the initial sparse coefficients and performing joint optimization to determine the target sparse coefficients includes: Based on the initial sparse coefficients and the adaptive weighted constraint term, a joint objective function is constructed that includes a data fidelity term, a sparse regularization term, and a weighted constraint term; Based on the iterative reweighted least squares method, the joint objective function is optimized and solved to determine the sparse coefficients after iterative update. Based on the preset convergence condition, determine whether the sparse coefficients after the iterative update meet the convergence requirements; If the convergence requirement is met, the iteratively updated sparse coefficients that meet the convergence requirement are determined as the target sparse coefficients.

6. The method according to claim 1, characterized in that, The step of performing image reconstruction based on the target sparse coefficients and the preset sparse dictionary to determine the reconstructed image blocks includes: Based on the target sparse coefficients and the preset sparse dictionary, a linear combination operation is performed to determine the initial reconstructed image blocks; Based on the neighborhood information of the damaged image block, post-processing operations are performed on the initial reconstructed image block to determine the optimized reconstructed image block; Based on the original frame size of the video data to be processed, the optimized reconstructed image block is subjected to a size restoration operation to determine the reconstructed image block.

7. The method according to claim 1, characterized in that, Also includes: The reconstructed image blocks replace the damaged image blocks in the video data to be processed, generating video data with the error hidden.

8. A video data error concealment device based on sparse coding and dictionary learning, characterized in that, The device includes: The damaged image acquisition unit is used to acquire damaged image blocks in the video data to be processed; The sparse coefficient determination unit is used to perform sparse coding operations on the damaged image block based on a preset sparse dictionary to determine the initial sparse coefficients. A weighted constraint construction unit is used to construct an adaptive weighted constraint term based on the neighborhood information of the damaged image patch; The target coefficient determination unit is used to add the adaptive weighted constraint term to the optimization model constructed based on the initial sparse coefficients, perform joint optimization solution operation, and determine the target sparse coefficients. The image reconstruction unit is used to perform image reconstruction operations based on the target sparse coefficients and the preset sparse dictionary, and to determine the reconstructed image blocks.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program stored in the memory, implements the steps of the video data error concealment method based on sparse coding and dictionary learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the video data error concealment method based on sparse coding and dictionary learning as described in any one of claims 1 to 7.

Images