A method, apparatus, and medium for inter-frame error concealment based on a stable diffusion model.
Through the stable diffusion model, the problem of video transmission quality and bandwidth consumption in rapidly changing network environments is solved, and high-quality and efficient inter-frame error hiding is achieved.
Patent Information
- Application Number
- CN202411166026.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-08-23
AI Technical Summary
The existing inter-frame image error hiding methods are difficult to ensure high-quality and low-latency video transmission in rapidly changing network environments, especially the bandwidth consumption based on the retransmission method, while the repair effect based on the prediction method is unstable.
Using the inter-frame error hiding method based on the stable diffusion model, the images of missing frames and front and back frames are extracted, and the space-time continuity characteristics are used to fill them. The energy function and gradient descent method are used to update the fill frames to ensure that the repaired image is consistent with the surrounding frames. Sex and stability.
It improves the image quality and overall consistency of video transmission, while reducing the amount of data transmission and saving bandwidth. It is suitable for scenarios with poor network conditions or real-time transmission.
Smart Images
Figure CN118694981B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an inter-frame error concealment method, device and medium based on a stable diffusion model. Background Art
[0002] Video compression and transmission play an important role in the medical industry, supporting remote medical diagnosis, education, and surgical guidance. Doctors and experts can participate in patient treatment remotely through real-time video consultations, or transmit high-definition medical images for remote diagnosis. In the field of education, video technology is used for remote training and surgical simulation to improve the skills of medical students and medical staff. In terms of demand, medical video transmission requires high quality, low latency, and security to ensure patient privacy and data security. The latest developments include the application of efficient coding standards such as VVC and AV1, as well as the promotion of 5G technology, which significantly improves the efficiency and reliability of video transmission. In addition, the integration of artificial intelligence technology makes image recognition and intelligent auxiliary diagnosis possible, further promoting the modernization and popularization of medical services.
[0003] Inter-frame image error concealment methods can generally be divided into two categories: retransmission-based and prediction-based. Retransmission-based methods repair errors by retransmitting damaged frames or areas. This method is simple and effective, but has high bandwidth overhead. Prediction-based methods use information from previous and subsequent frames to predict and repair damaged areas. This can reduce bandwidth usage without retransmitting data, but is sensitive to environmental changes. Advantages include the simplicity and ease of implementation of retransmission-based methods and obvious repair effects, which are suitable for demanding video transmission environments; while prediction-based methods save bandwidth resources and are suitable for scenarios with poor network conditions or requiring real-time transmission. However, the main disadvantage of retransmission-based methods is that they consume a lot of bandwidth and cannot adapt to rapidly changing network environments; prediction-based methods may be affected by prediction errors and environmental changes, resulting in unstable repair effects. Choosing an appropriate method requires a comprehensive consideration of network conditions, real-time requirements, and a balance between transmission quality. Summary of the invention
[0004] In order to cope with the rapidly changing network environment and complete the compensation of lost frames in the process of video compression transmission with higher timeliness and quality, the present invention proposes an inter-frame error concealment method based on a stable diffusion model, comprising the steps of:
[0005] S1: Extract the lost frames in the video compression transmission process and the inter-frame images before and after the lost frames;
[0006] S2: Based on the spatiotemporal continuity characteristics between corresponding pixels of images between consecutive frames, the filling frame is obtained based on the pixel values of the images between the previous and next frames through the stability diffusion model;
[0007] S3: Obtain the direction in which the gradient of the stability diffusion model grows fastest at the current filling frame, and update the filling frame based on the gradient direction;
[0008] S4: After the gradient change is less than a preset threshold, the filling frame is output to complete the inter-frame error concealment under the lost frame prediction.
[0009] Furthermore, the step S1 also includes an initialization operation on the lost frame, and the initialization operation is blank or random value processing.
[0010] Furthermore, in the step S2, the stability diffusion model uses an energy function to obtain the padded frame, and the formula is expressed as follows:
[0011]
[0012] In the formula, To evaluate the padded frames The energy function of the stability and continuity characteristics of and is the weight parameter used to balance the influence of data term and regularization term, is a balance data item used to measure the difference between the padded frame and the known frame, Regularization term to ensure that the padded frames remain stable both locally and globally.
[0013] Furthermore, the balance data item measures the difference between the padded frame and the known frame based on the difference at the pixel level, and the formula is as follows:
[0014]
[0015] In the formula, is the set of all pixels in the image, is the pixel position, is the previous inter-frame image of the lost frame, is the next inter-frame image of the lost frame, is the sum of squares of pixel-level differences expressed as squared Euclidean distance.
[0016] Furthermore, the regularization term uses the image gradient norm under total variation regularization, and the formula is expressed as follows:
[0017]
[0018] In the formula, To fill the gradient of the frame at position p, is the gradient L1 norm used to measure the gradient washing property of the image.
[0019] Furthermore, the gradient at the position p is obtained using a finite difference method.
[0020] Furthermore, in the step S3, the padded frame is updated by the gradient descent method, and the formula is expressed as follows:
[0021]
[0022] In the formula, To control the learning rate of each update step size, The energy function E is the padded frame gradient.
[0023] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the inter-frame error concealment method based on a stable diffusion model are implemented.
[0024] Also included is a device for processing data, comprising:
[0025] a memory having a computer program stored thereon;
[0026] The processor is used to execute the computer program in the memory to implement the steps of the inter-frame error concealment method based on the stable diffusion model.
[0027] Compared with the prior art, the present invention has at least the following beneficial effects:
[0028] (1) The inter-frame error concealment method, device and medium based on the stable diffusion model described in the present invention perform inter-frame error concealment based on the stable diffusion method, which can pay more attention to the visual effect and stability after the lost frame is filled;
[0029] (2) Pay more attention to the local and global continuity of the image, ensuring that the repaired image remains consistent with the surrounding frames, thereby improving the overall quality of the image;
[0030] (3) While maintaining high-quality images, the amount of data transmission is reduced, thereby saving bandwidth and improving transmission efficiency. It can be used in scenarios with poor network conditions or where real-time transmission is required. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A step diagram of an inter-frame error concealment method based on a stable diffusion model. DETAILED DESCRIPTION
[0032] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.
[0033] Inter-frame image compression transmission is a common technology in video coding, which uses inter-frame prediction and motion compensation to achieve data compression and transmission. First, video frames are divided into key frames (I frames) and non-key frames (P frames, B frames), where key frames can be decoded independently and non-key frames rely on the information of previous and next frames. Inter-frame prediction finds motion vectors by comparing the pixels of previous and next frames, thereby finding similar areas and reducing redundant information. Motion compensation adjusts pixel values according to the predicted motion vector, reducing the amount of data required for encoding. These technologies enable non-key frames to be further compressed through differential coding and motion compensation, rather than storing the complete information of each frame independently. The goal of inter-frame image compression transmission is to reduce the amount of data transmitted as much as possible while maintaining visual quality, thereby saving bandwidth and improving transmission efficiency. However, when network packet loss, delay, jitter, signal interference in wireless transmission, or quality problems in wired connections occur, video frames may be damaged or lost during transmission. In order to deal with this problem, such as Figure 1 As shown, the present invention proposes an inter-frame error concealment method based on a stable diffusion model, comprising the steps of:
[0034] S1: Extract the lost frames in the video compression transmission process and the inter-frame images before and after the lost frames;
[0035] S2: Based on the spatiotemporal continuity characteristics between corresponding pixels of images between consecutive frames, the filling frame is obtained based on the pixel values of the images between the previous and next frames through the stability diffusion model;
[0036] S3: Obtain the direction in which the gradient of the stability diffusion model grows fastest at the current filling frame, and update the filling frame based on the gradient direction;
[0037] S4: After the gradient change is less than a preset threshold, the filling frame is output to complete the inter-frame error concealment under the lost frame prediction.
[0038] First, we need to find the missing frames Surrounding known frames , Some feature information, such as pixel values or higher-level feature representations, is extracted from the image, and the possible pixel values of the corresponding pixel points of the lost frame are evaluated based on the coherence of the image changes in time and space between consecutive frames. Here, the present invention uses the stable diffusion method to infer the lost frame. The pixel value of .
[0039] Among them, Stability AI's Stable Diffusion is a deep learning technique that is mainly used for image generation. It is based on the diffusion process, an iterative random process that adds noise to the data at each step until the data becomes unrecognizable. Then, a neural network is trained to reverse the process, starting from a high noise level and gradually removing the noise to eventually generate a clear image. Therefore, we can learn the spatial and temporal continuity between pixels in an image through the Stability Diffusion model, and fill in the pixel values of the missing frames based on the information of the known frames to maintain the visual continuity and consistency of the image.
[0040] Here, we define an energy function , which consists of the following two parts: (1) Data Term, which is used to measure the difference between the padded frame and the known previous and next frames; (2) Regularization Term, which is used to ensure that the padded frame maintains stability and continuity locally and globally. The energy function is defined as:
[0041]
[0042] In the formula, To evaluate the padded frames The energy function of the stability and continuity characteristics of and is the weight parameter used to balance the influence of data term and regularization term, Used to measure the difference between the padded frame and the known frame, is a regularization term that ensures that the padded frames remain stable locally and globally.
[0043] Specifically, the Data Term data item is used to measure the filling frame With known previous and next frames and A common way to calculate this data item is to use pixel-level differences. Assume Represents the pixel value at position p in image I, then the data item can be expressed as:
[0044]
[0045] In the formula, is the set of all pixels in the image, is the pixel position, is the sum of squares of pixel-level differences expressed as squared Euclidean distance.
[0046] The regularization term is used to ensure that the padded frame Maintain stability and continuity locally and globally. The design of regularization terms can adopt various methods, such as total variation (TV), smoothness constraints, etc. A common regularization term is total variation regularization, which can be defined as the norm of the image gradient:
[0047]
[0048] In the formula, To fill the gradient of the frame at position p, is the gradient L1 norm used to measure the gradient washing property of the image.
[0049] The energy function designed in this way is minimized by To optimize the padded frame , to ensure that the padded frames are as consistent and stable as possible with the frames in the original sequence visually. Here, we use the gradient expression method to optimize the energy function.
[0050] Here, the gradient is a vector that represents the maximum growth direction of the energy function at a certain point. The modulus (size) of the gradient represents the growth rate in this direction. In the optimization problem, the gradient descent algorithm uses this property to update the parameters along the negative gradient direction to find the minimum value of the energy function. The padding frame update under the gradient descent algorithm is expressed as follows:
[0051]
[0052] In the formula, To control the learning rate of each update step size, The energy function E is the padded frame gradient.
[0053] Repeat the above steps until the gradient change is less than the preset threshold. This is the padded frame to be obtained, which is as consistent and stable as possible with the known frame by minimizing the energy function.
[0054] On the basis of the above, the present invention further proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the inter-frame error concealment method based on a stable diffusion model are implemented.
[0055] Also included is a device for processing data, comprising:
[0056] a memory having a computer program stored thereon;
[0057] The processor is used to execute the computer program in the memory to implement the steps of the inter-frame error concealment method based on the stable diffusion model.
[0058] In summary, the inter-frame error concealment method, device and medium based on the stable diffusion model described in the present invention can perform inter-frame error concealment based on the stability diffusion method, pay more attention to the visual effect and stability after the lost frame is filled, and pay more attention to the local and global continuity of the image, ensuring that the repaired image maintains consistency with the surrounding frames, thereby improving the overall quality of the image. While maintaining high-quality images, the amount of data transmission is reduced, bandwidth is saved and transmission efficiency is improved, and it can be applied to scenes with poor network conditions or requiring real-time transmission.
[0059] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0060] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0061] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0062] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
Claims
1. An inter-frame error concealment method based on a stable diffusion model, characterized in that: Includes steps: S1: Extract the lost frames in the video compression transmission process and the inter-frame images before and after the lost frames; S2: Based on the spatiotemporal continuity characteristics between corresponding pixels of images between consecutive frames, the filling frame is obtained based on the pixel values of the images between the previous and next frames through the stability diffusion model; S3: Obtain the direction in which the gradient of the stability diffusion model grows fastest at the current filling frame, and update the filling frame based on the gradient direction, and repeat this step until the gradient change is less than a preset threshold; S4: after the gradient change is less than a preset threshold, output the filling frame to complete the inter-frame error concealment under the lost frame prediction; In the step S2, the stability diffusion model uses an energy function to obtain the filling frame, and the formula is expressed as follows: In the formula, To evaluate the padded frames The energy function of the stability and continuity characteristics of and is the weight parameter used to balance the influence of data term and regularization term, is a balance data item used to measure the difference between the padded frame and the known frame, A regularization term to ensure that the padded frames remain stable both locally and globally; In the step S3, the padded frame is updated by the gradient descent method, and the formula is expressed as follows: In the formula, To control the learning rate of each update step size, The energy function E is the padded frame gradient.
2. The inter-frame error concealment method based on a stable diffusion model as claimed in claim 1, characterized in that: The step S1 also includes an initialization operation for the lost frame, and the initialization operation is blank or random value processing.
3. The inter-frame error concealment method based on a stable diffusion model as claimed in claim 1, characterized in that: The balance data item measures the difference between the padded frame and the known frame based on the difference at the pixel level, and the formula is as follows: In the formula, is the set of all pixels in the image, is the pixel position, is the previous inter-frame image of the lost frame, is the next inter-frame image of the lost frame, is the sum of squares of pixel-level differences expressed as squared Euclidean distance.
4. The inter-frame error concealment method based on a stable diffusion model as claimed in claim 3, characterized in that: The regularization term uses the image gradient norm under total variation regularization, and the formula is expressed as follows: In the formula, To fill the gradient of the frame at position p, is the gradient L1 norm used to measure the gradient washing property of the image.
5. The inter-frame error concealment method based on a stable diffusion model as claimed in claim 4, characterized in that: The gradient at the position p is obtained by using the finite difference method.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the error concealment method according to any one of claims 1 to 5 are implemented.
7. A device for processing data, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the error concealment method according to any one of claims 1 to 5.