A method for inter-frame image error concealment based on deep reinforcement learning

Abstract

A method for inter-frame image error hiding based on deep reinforcement learning belongs to the technical field of video image processing, including: step one: inter-frame image preprocessing and initialization: S101, inter-frame image preprocessing; S102, initializing the main network and target network of the deep Q network, and defining the network structure and parameters; creating and initializing the experience replay pool; setting hyperparameters, including learning rate, discount factor, #imgabs0# value, exploration probability, and experience replay pool size; step two: DQN model training: S201, defining state and action space; S202, defining reward function; S203, Q value calculation; S204, Q value update; S205, experience replay and parameter update; step three: inter-frame image error hiding operation: according to the selected action a t The present invention continuously optimizes the training of the Q network, and the model can gradually learn and optimize the restoration strategy in complex image scenes, thereby effectively reducing the impact of inter-frame image errors.

Description

Technical Field

[0001] The present invention belongs to the technical field of video image processing, and in particular relates to an inter-frame image error hiding method based on deep reinforcement learning. Background Art

[0002] Inter-frame image error concealment is a method of handling lost or damaged frames during video transmission or storage. It uses information from adjacent frames to reconstruct missing or damaged frames, thereby ensuring video continuity and picture quality. This technology can effectively reduce the impact of transmission errors on the video experience and ensure smooth visual effects. Inter-frame image error concealment technology is widely used in video communication, streaming media transmission, and storage. It ensures video continuity and visual quality by repairing lost or damaged parts in video frames.

[0003] Inter-frame image error concealment methods can be classified according to their implementation principles and application fields: (1) Inter-frame image error concealment methods based on replication and interpolation, which fill the missing areas of the damaged frame by copying or interpolating pixels from surrounding frames. This method is simple and direct, with high computational efficiency, suitable for static or simple dynamic scenes, and can quickly restore image information. However, it easily introduces image blur and artifacts, especially in scenes with complex backgrounds and large dynamic changes. (2) Inter-frame image error concealment methods based on motion compensation, which use inter-frame motion information to find the best matching block and compensate, can accurately restore the damaged area in motion and effectively avoid motion blur. However, it has limited adaptability to complex and nonlinear motion, and may cause block effects and artifacts, requiring strong motion prediction and compensation capabilities. (3) Inter-frame image error concealment methods based on deep learning, which use models such as convolutional neural networks to learn and predict the pixel values ​​of the missing areas in the damaged frame, can handle complex image features and motion patterns, and have strong generalization capabilities. This method performs well in dealing with complex scenes and large-scale motions, but the training and reasoning process is complex and requires a large amount of labeled data and high computing resources. Summary of the invention

[0004] The purpose of this invention is to propose an inter-frame image error hiding method based on deep reinforcement learning, and continuously optimize the training of the Q network. The model can gradually learn and optimize the restoration strategy in complex image scenes, thereby effectively reducing the impact of inter-frame image errors, and then use the Q network to learn the long-term rewards of different restoration operations to improve the accuracy and efficiency of image restoration.

[0005] The technical solution adopted by the present invention is:

[0006] A method for inter-frame image error concealment based on deep reinforcement learning comprises the following steps:

[0007] Step 1: Inter-frame image preprocessing and initialization:

[0008] S101, inter-frame image preprocessing: denoising, contrast enhancement and normalization operations to improve image quality and enhance the effect of restoration operations;

[0009] S102, initialize the main network and target network of the Deep Q-Network (DQN), define the network structure and parameters, and set the main network and target network to the same initial weights; create and initialize the experience replay pool to store experience samples during training; set hyperparameters, including learning rate ( α ), discount factor ( c )、 value, exploration probability, and experience replay pool size;

[0010] Step 2: DQN model training:

[0011] S201, define state and action space:

[0012] Defining states S t It is a combination of the information of the current damaged frame and its previous and next frames, usually represented as a spliced ​​image of three consecutive frames:

[0013] ;

[0014] in, I t is a damaged frame, indicating that the frame t image information, I t-1 and I t+1 They are the front and back frames respectively;

[0015] Define the action space A, where each action represents a restoration operation, including Gaussian filtering, block matching and motion compensation, and total variation image restoration;

[0016] ;

[0017] S202, defining a reward function;

[0018] S203, Q value calculation:

[0019] Q function approximation: Using the Q function Q( S t , a t ) to evaluate the state S t Execute an action a tThe value of , which is approximated by a deep neural network (DQN);

[0020] S204, Q value update;

[0021] S205, experience playback and parameter update;

[0022] Step 3: Inter-frame image error concealment operation:

[0023] Depending on the action selected a t Perform the corresponding image restoration operations: for Gaussian filtering, use the information of previous and next frames to restore the damaged area through weighted average filtering; for block matching and motion compensation, find the best matching block in the previous frame and perform compensation; for total variation image restoration, restore the pixel values ​​of the damaged area through optimization.

[0024] Furthermore, in step S202, the reward function r t ( S t , a t ) Based on the execution action a t The similarity and quality of the images are then defined as:

[0025] ;

[0026] in, is the restored image, Represents the structural similarity index, which measures the structural similarity between the restored image and the original image. Indicates edge clarity, which can be calculated by edge detection algorithms (such as Sobel operator). Represents color fidelity, which can be calculated by comparing the similarity of color histograms or color distributions; parameter a、b、c Adjust according to specific needs and optimization goals to balance the contribution of different indicators to the reward function. This reward function combines multiple image quality evaluation factors and can more comprehensively evaluate the effect of image restoration.

[0027] Furthermore, in step S204, the Q value updating method is:

[0028] Use the Bellman equation to calculate the target Q value, the formula is:

[0029] ;

[0030] Then update the current Q value according to the target Q value;

[0031] in, yt represents the calculated target Q value; r t Indicates the current status S t Next, take action a t Immediate rewards after S t+1 Indicates execution of an action a t The next state reached after l Represents the discount factor, which ranges from [0, 1] and is used to weigh future rewards and immediate rewards. l The value makes the model pay more attention to the immediate reward. l The value makes the model focus more on long-term returns; Indicates the target network Q In Status S t+1 and actions a t Next Q Value, parameter is i , represents the learning rate, which controls the step size of parameter update.

[0032] Furthermore, in step S205, the experience replay and parameter update method is:

[0033] The experience sample ( S t , a t , r t , S t+1 ) is stored in the experience replay pool D, small batches of samples are randomly extracted from the experience replay pool for training, the target Q value is calculated and the network parameters are updated by minimizing the loss function;

[0034] The loss function is expressed as:

[0035] ;

[0036] By minimizing the loss function, the parameters of the Q network are updated using the gradient descent method. i , so that the predicted value of the Q network is closer to the target Q value.

[0037] Further, in step three, the inter-frame image error concealment operation method is: according to Q Current state of network computing S t All possible actions Q value, and select QThe action with the largest value, and then based on the selected action a t Perform the corresponding image recovery operation:

[0038] (1) Gaussian filtering: For each pixel ( x,y ), using the information of the previous and next frames for weighted averaging:

[0039] ;

[0040] in, wt- 1 and wt +1 is the weight;

[0041] (2) Block matching and motion compensation

[0042] Block matching and motion compensation restore the damaged area by finding the best matching block in the previous and next frames and performing motion compensation:

[0043] Block matching: Assume the block size is N × N , in the previous frame I t−1 Find the current frame I t The damaged block's closest match:

[0044] ;

[0045] Among them, SSD is the difference measure between blocks, and the block with the smallest SSD is selected;

[0046] Motion compensation: The block corresponding to the smallest SSD is used for recovery:

[0047] ;

[0048] in and is the best offset found during block matching;

[0049] (3) Total Variation Image Inpainting

[0050] Image restoration based on total variation (TV) is a commonly used image restoration technique, which smoothes the image and preserves edge features by minimizing the total variation of image gradients.

[0051] Initialization: I t filled Initialized as the weighted average of the previous and next frames:

[0052] ;

[0053] Iterative optimization: Use gradient descent to minimize the objective function and update I t filled The value of ensures that the pixel values ​​in the known area remain unchanged after each iteration.

[0054] Beneficial effects of the present invention:

[0055] The present invention realizes efficient inter-frame image error hiding through a deep reinforcement learning model, significantly improving the accuracy and speed of image restoration. Based on the learning ability of the Q network, the system can dynamically adjust the restoration strategy according to different image damage degrees and complexities, thereby improving the system's adaptability and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a schematic diagram of DQN inter-frame image error recovery. DETAILED DESCRIPTION

[0057] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0058] A method for inter-frame image error concealment based on deep reinforcement learning comprises the following steps:

[0059] Step 1: Inter-frame image preprocessing and initialization:

[0060] S101, inter-frame image preprocessing: denoising, contrast enhancement and normalization operations to improve image quality and enhance the effect of restoration operations;

[0061] S102, initialize the main network and target network of the deep Q network, define the network structure and parameters, and set the main network and target network to the same initial weight; create and initialize the experience replay pool to store the experience samples during the training process; set hyperparameters, including the learning rate ( α ), discount factor ( c )、 value, exploration probability, and experience replay pool size;

[0062] Step 2: DQN model training

[0063] S201, define state and action space:

[0064] Defining states S t It is a combination of the information of the current damaged frame and its previous and next frames, usually represented as a spliced ​​image of three consecutive frames:

[0065] ;

[0066] in,I t Is a damaged frame, indicating that the frame t image information, I t-1 and I t+1 They are the front and back frames respectively;

[0067] Define the action space A, where each action represents a restoration operation, including Gaussian filtering, block matching and motion compensation, and total variation image restoration;

[0068] ;

[0069] S202, define the reward function:

[0070] Reward Function r t ( S t , a t ) Based on the execution action a t The similarity and quality of the images are then defined; it takes into account aspects such as SSIM, edge sharpness and color fidelity:

[0071] ;

[0072] in, is the restored image, Represents the structural similarity index, which measures the structural similarity between the restored image and the original image. Indicates edge clarity, which can be calculated by edge detection algorithms (such as Sobel operator). Represents color fidelity, which can be calculated by comparing the similarity of color histograms or color distributions; parameter a、b、c Adjust according to specific needs and optimization goals to balance the contribution of different indicators to the reward function. This reward function combines multiple image quality evaluation factors to more comprehensively evaluate the effect of image restoration; This reward function combines multiple image quality evaluation factors to more comprehensively evaluate the effect of image restoration.

[0073] S203, Q value calculation:

[0074] Q function approximation: Using the Q function Q( S t , a t ) to evaluate the state S t Execute an action a tThe value of , which is approximated by a deep neural network (DQN);

[0075] S204, Q value update:

[0076] Target Q value calculation: Use the Bellman equation to calculate the target Q value. The formula is:

[0077] ;

[0078] Then update the current Q value according to the target Q value;

[0079] in, y t represents the calculated target Q value; r t Indicates the current status S t Next, take action a t Immediate rewards after S t+1 Indicates execution of an action a t The next state reached after l Represents the discount factor, which ranges from [0, 1] and is used to weigh future rewards and immediate rewards. l The value makes the model pay more attention to the immediate reward. l The value makes the model focus more on long-term returns; Indicates the target network Q In Status S t+1 and actions a t Next Q Value, parameter is i , represents the learning rate, which controls the step size of parameter update.

[0080] S205, experience replay and parameter update:

[0081] The experience sample ( S t , a t , r t , S t+1 ) is stored in the experience replay pool D, small batches of samples are randomly extracted from the experience replay pool for training, the target Q value is calculated and the network parameters are updated by minimizing the loss function;

[0082] The loss function is expressed as:

[0083] ;

[0084] By minimizing the loss function, the parameters of the Q network are updated using the gradient descent method. i , so that the predicted value of the Q network is closer to the target Q value.

[0085] Through the above steps, the DQN algorithm gradually optimizes the Q network so that it can better predict the Q value of each state-action pair and thus select the optimal action strategy.

[0086] Step 3: Inter-frame image error concealment operation

[0087] according to Q Current state of network computing S t All possible actions Q value, and select Q The action with the largest value, and then based on the selected action a t Perform the corresponding image recovery operation:

[0088] (1) Gaussian filtering: For each pixel ( x,y ), using the information of the previous and next frames for weighted averaging:

[0089] ;

[0090] in, w t-1 and w t+1 is the weight;

[0091] (2) Block matching and motion compensation

[0092] Block matching and motion compensation restore the damaged area by finding the best matching block in the previous and next frames and performing motion compensation:

[0093] Block matching: Assume the block size is N × N , in the previous frame I t−1 Find the current frame I t The damaged block's closest match:

[0094] ;

[0095] Among them, SSD is the difference measure between blocks, and the block with the smallest SSD is selected;

[0096] Motion compensation: The block corresponding to the smallest SSD is used for recovery:

[0097] ;

[0098] in, and is the best offset found during block matching;

[0099] (3) Total Variation Image Inpainting

[0100] Image restoration based on total variation (TV) is a commonly used image restoration technique, which smoothes the image and preserves edge features by minimizing the total variation of image gradients.

[0101] Initialization: I t filled Initialized as the weighted average of the previous and next frames:

[0102] ;

[0103] Iterative optimization: Use gradient descent to minimize the objective function and update I t filled The value of ensures that the pixel values ​​in the known area remain unchanged after each iteration.

[0104] The image restoration method based on total variation can effectively restore the pixel values ​​of the damaged area and improve the restoration effect by combining the information of the previous and next frames.

[0105] In the VVC inter-frame image error hiding process of the present invention, (1) image preprocessing steps such as denoising, contrast enhancement and normalization are performed to improve image quality and the effect of the restoration operation; the main network and target network of the DQN are initialized, the experience replay pool is created, and hyperparameters such as learning rate, discount factor, exploration probability, etc. are set to define the state representation method and action space. (2) A more advanced reward function than SSIM is designed and used, which comprehensively considers factors such as image structure similarity, edge clarity and color fidelity. (3) According to the selected action, the corresponding image restoration operation is performed, such as three-dimensional filtering using the information of the previous and next frames to restore the damaged area through weighted average filtering; block matching and motion compensation find the best matching block in the previous frame for compensation; image restoration methods such as total variation restore the pixel values ​​of the damaged area through optimization. Combined with deep reinforcement learning, the optimal restoration strategy is selected to gradually improve the image restoration effect.

Claims

1. A method for inter-frame image error concealment based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Inter-frame image preprocessing and initialization: S101, inter-frame image preprocessing: denoising, contrast enhancement and normalization operations to improve image quality and enhance the effect of restoration operations; S102, initializing the main network and target network of the deep Q network, defining the network structure and parameters, and setting the main network and the target network to the same initial weights; Create and initialize the experience replay pool to store experience samples during training; set hyperparameters, including learning rate, discount factor, value, exploration probability, and experience replay pool size; Step 2: DQN model training: S201, define state and action space: Defining states S t It is the information combination of the current damaged frame and its previous and next frames: ； in, I t is a damaged frame, indicating that the frame t image information, I t-1 and I t+1 They are the front and back frames respectively; Define the action space A, where each action represents a restoration operation, including Gaussian filtering, block matching and motion compensation, and total variation image restoration; ； S202, defining a reward function; S203, Q value calculation: Q function approximation: Using the Q function Q( S t , a t ) to evaluate the state S t Execute an action a t The value of , which is approximated by a deep neural network; S204, Q value update; S205, experience playback and parameter update; Step 3: Inter-frame image error concealment operation: Depending on the action selected a t Perform the corresponding image restoration operations: for Gaussian filtering, use the information of previous and next frames to restore the damaged area through weighted average filtering; for block matching and motion compensation, find the best matching block in the previous frame and perform compensation; for total variation image restoration, optimize and restore the pixel values ​​of the damaged area.

2. The inter-frame image error concealment method based on deep reinforcement learning according to claim 1, characterized in that: In step S202, the reward function r t ( S t , a t ) Based on the execution action a t The similarity and quality of the images are then defined as: ； in, is the restored image, Represents the structural similarity index, which measures the structural similarity between the restored image and the original image. Indicates edge clarity, calculated by edge detection algorithm, Represents color fidelity, calculated by comparing the similarity of color histograms or color distributions; parameter α, β, γ Adjust according to specific needs and optimization goals to balance the contribution of different indicators to the reward function.

3. The inter-frame image error concealment method based on deep reinforcement learning according to claim 1, characterized in that: In step S204, the Q value updating method is: Use the Bellman equation to calculate the target Q value, the formula is: ； Then update the current Q value according to the target Q value; in, y t represents the calculated target Q value; r t Indicates the current status S t Next, take action a t Immediate rewards after S t+1 Indicates execution of an action a t The next state reached after λ Represents the discount factor, which ranges from [0, 1] and is used to weigh future rewards and immediate rewards. λ The value makes the model pay more attention to the immediate reward. λ The value makes the model focus more on long-term returns; Indicates the target network Q In Status S t+1 and actions a t Next Q Value, parameter is θ , represents the learning rate, which controls the step size of parameter update.

4. The inter-frame image error concealment method based on deep reinforcement learning according to claim 1, characterized in that: In step S205, the experience playback and parameter update method is: The experience sample ( S t , a t , r t , S t+1 ) is stored in the experience replay pool D, small batches of samples are randomly extracted from the experience replay pool for training, the target Q value is calculated and the network parameters are updated by minimizing the loss function; The loss function is expressed as: ； By minimizing the loss function, the parameters of the Q network are updated using the gradient descent method. θ , so that the predicted value of the Q network is closer to the target Q value.

5. The inter-frame image error concealment method based on deep reinforcement learning according to claim 1, characterized in that: In step 3, the inter-frame image error concealment operation method is: according to Q Current state of network computing S t All possible actions Q value, and select Q The action with the largest value, and then based on the selected action a t Perform the corresponding image recovery operation: (1) Gaussian filtering: For each pixel ( x,y ), using the information of the previous and next frames for weighted averaging: ； in, w t-1 and w t+1 is the weight; (2) Block matching and motion compensation Block matching and motion compensation restore the damaged area by finding the best matching block in the previous and next frames and performing motion compensation: Block matching: Assume the block size is N × N , in the previous frame I t−1 Find the current frame I t The damaged block's closest match: ； Among them, SSD is the difference measure between blocks, and the block with the smallest SSD is selected; Motion compensation: The block corresponding to the smallest SSD is used for recovery: ； in, and is the best offset found during block matching; (3) Total Variation Image Inpainting Initialization: I t filled Initialized as the weighted average of the previous and next frames: ； Iterative optimization: Use gradient descent to minimize the objective function and update I t filled The value of ensures that the pixel values ​​in the known area remain unchanged after each iteration.

Images