Dual network-based low-quality film image inpainting enhancement method and system

Abstract

The application discloses a low-quality film image repair and enhancement method and system based on a dual network, and the method comprises the following steps: using an arbitrary scene segmentation method to segment an original video to be repaired into a plurality of video clips; extracting key frames of each video clip to form a key frame set, and extracting video frames in the original video to form a video frame set; combining the key frames and corresponding key frames repaired by a manual picture to form a low-quality-high-quality paired data set; the paired data set is divided into a plurality of groups and sequentially input into a pre-established and pre-trained pre-training repair and enhancement model, a transfer learning strategy is used for parameter updating, and a trained repair and enhancement model is obtained; the video frame set is input into the repair and enhancement model to obtain a repaired and enhanced picture set; and the picture set is recombined into a video according to an original video frame rate; wherein the pre-training repair and enhancement model is a dual network, comprising a generation network and a degradation network; and the trained repair and enhancement model is the generation network.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and deep learning technology, and in particular to a method and system for restoring and enhancing low-quality film images based on dual networks. Background Technology

[0002] Currently, my country's ultra-high-definition video industry is facing problems such as "strong hardware but weak software" and "terminals first, content relatively lagging behind," which seriously hinders the development of the entire industry. On the one hand, high-quality high-definition content is in short supply; on the other hand, due to limitations in shooting technology and long periods of shelving, many excellent film and video works from a long time ago are difficult to release because their image quality cannot meet the current requirements of high-definition video. Low-quality film and video often suffers from problems such as blurred textures, color distortion, and image deterioration.

[0003] Image inpainting and enhancement are techniques for reconstructing lost or damaged parts of images or videos. Current image inpainting and enhancement methods can be categorized into traditional methods and deep learning-based methods. Traditional methods typically fill unknown regions with information from existing regions, such as iterative repair using variational methods, boundary priority-based filling, patch voting and spatiotemporal completion, and offset and graph segmentation-based methods. However, these algorithms can only fill unknown regions with information from existing regions and cannot predict the content to be filled. Furthermore, these methods are primarily limited to translation for repairing known regions; introducing scaling, rotation, or other variations exponentially increases computational complexity. With technological advancements, machine learning, represented by deep learning, is sweeping the field of computer vision research. Researchers have found that when traditional physical models encounter bottlenecks, machine learning methods can help explain these complex mathematical models. Inspired by this, researchers have shifted their focus to deep learning-based image inpainting and enhancement techniques. Deep learning-based methods demonstrate superior performance compared to traditional algorithms in image restoration and enhancement tasks due to their excellent encoding and mapping capabilities for image information, and have already achieved good results on ideal low-quality-high-quality paired datasets.

[0004] However, for the restoration and enhancement of low-quality film video images, there are no directly relevant low-quality-high-quality paired datasets available for training. To address this issue, most existing methods utilize artificial prior loss models to degrade high-quality datasets into low-quality datasets and then train the model based on these paired datasets. While this approach can achieve good results on training datasets, it is largely ineffective in real-world applications. This is because the transformation function between the input and output datasets learned by the model is merely a simple degradation function determined a priori, making it difficult to fit the complex degradation function between low-quality images and ideal high-quality images in practical applications. Therefore, exploring low-quality film video image restoration and enhancement methods suitable for real-world applications is essential. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and to propose a method and system for restoring and enhancing low-quality film images based on dual networks.

[0006] To achieve the above objectives, this invention proposes a method for restoring and enhancing low-quality film images based on dual networks, the method comprising:

[0007] Step 1) Use an arbitrary scene segmentation method to segment the original video to be repaired into several video segments according to the video scene;

[0008] Step 2) Extract the keyframes of each video segment to form a keyframe set, and extract all video frames from the original video to be repaired to form a video frame set;

[0009] Step 3) Combine the keyframes and the corresponding keyframes after manual image restoration into a low-quality-high-quality paired dataset;

[0010] Step 4) Divide the low-quality-high-quality pairwise datasets into several groups, input them sequentially into the pre-established and trained pre-trained repair and enhancement model, and use the transfer learning strategy to update the parameters to obtain the trained repair and enhancement model.

[0011] Step 5) Input the video frame set from Step 2) into the restoration and enhancement model from Step 4) to obtain the restored and enhanced image set;

[0012] Step 6) Reassemble the image set from Step 5) into a video at the original video frame rate to restore and enhance low-quality film images;

[0013] The pre-trained repair and enhancement model is a dual network, comprising a generator network and a degradation network; the trained repair and enhancement model is a generator network.

[0014] As an improvement to the above method, the low-quality film image includes: having scratches, having dirt, missing details and / or color distortion.

[0015] As an improvement to the above method, the keyframe in step 2) is the frame closest to the middle moment in each video segment.

[0016] As an improvement to the above method, the manual image restoration in step 3) includes sequentially performing scratch repair, stain repair, detail enhancement, and recoloring.

[0017] As an improvement to the above method, the pre-trained inpainting and enhancement model's generative network is used to inpaint and enhance low-quality images into high-quality images, and the pre-trained inpainting and enhancement model's degradation network is used to degrade high-quality images into low-quality images; wherein, both the generative network and the degradation network include: a hybrid attention module, a cross-semantic attention module, a feature fusion module, and a low-rank optimization module; wherein the input of the generative network is a low-quality-high-quality image pair.<X,Y> The low-quality image X is used as the input to the degradation network.<X,Y> High-quality image Y.

[0018] As an improvement to the above method, the processing of the pre-trained repair and enhancement model specifically includes:

[0019] Step S1) Generate an encoder with an l-layer structure in the network to perform hierarchical feature extraction on the low-quality image X, obtaining the hierarchical features f corresponding to each layer. i , i∈[1,l];

[0020] In the degradation network, an encoder with an l-layer structure performs hierarchical feature extraction on a high-quality image Y, obtaining the hierarchical features f′ corresponding to each layer. i , i∈[1,l];

[0021] Step S2) Generates the hybrid attention module of the network to process the hierarchical features f of the last layer. l Hybrid attention processing is performed to capture richer global high-level semantic context information, resulting in semantic fusion features f. l_f ;

[0022] The hybrid attention module of the degenerate network will use the hierarchical features f′ of the last layer. l Hybrid attention processing is performed to capture richer global high-level semantic context information, resulting in semantic fusion features f′. l_f ;

[0023] Step S3) Generates the network's cross-semantic attention module to process the hierarchical features f output from the i-th layer of the encoder. i , i∈[1,l-1], with semantic fusion feature f l_f Perform corresponding cross-semantic attention processing to optimize local spatial details and obtain the local spatial detail optimized feature f of the i-th layer. i_c ;

[0024] The cross-semantic attention module of the degenerate network will take the hierarchical features f′ output from the i-th layer of the encoder. i i∈[1,l-1], and semantic fusion feature f′ l_f Perform corresponding cross-semantic attention processing to optimize local spatial detail information, and obtain the local spatial detail optimized feature f′ of the i-th layer. i_c ;

[0025] Step S4) Generates the feature fusion module of the network by optimizing the local spatial details of the encoder layer li using the feature f. l-i_c With the output f of the corresponding i-th layer decoder i_d After processing with the corresponding feature fusion mechanism, the output features o of each layer of the decoder are obtained. i , i∈[1,l-1]; the low-rank optimization module optimizes o i Perform low-rank optimization to obtain the low-rank optimized output result O. i The input to the first layer of the decoder is the semantic fusion feature f. l_f Output f after low-rank optimization in_d ;

[0026] The feature fusion module of the degenerate network optimizes the local spatial details of the encoder layer li into feature f′. l-i_c The output f′ of the corresponding i-th layer decoder i_d After performing corresponding feature fusion processing, the output feature o′ of each layer of the decoder is obtained. i , i∈[1,l-1]; for o′ i Perform low-rank optimization to obtain the low-rank optimized output result O′. i The input to the first layer of the decoder is the semantic fusion feature f′. l_f The output f′ after low-rank optimization in_d ;

[0027] Step S5) Calculate the output f of the last layer of the decoder in the generator network. l_d Upsampling is performed to obtain the generator network output X. H ;

[0028] The output f′ of the last layer of the decoder of the degenerate network l_d Upsampling is performed to obtain the degenerate network output Y. L .

[0029] As an improvement to the above method, the method further includes a training step for pre-training the repair and enhancement model; specifically including:

[0030] Construct a training dataset; the training dataset includes several low-quality-high-quality pairs of data.

[0031] The training dataset is randomly divided into m groups according to each pair a;

[0032] The low-quality images of each pair of data are sequentially input into the generator network, while the corresponding high-quality images are input into the degradation network. The parameters in the generator and degradation networks are updated using gradient descent until the training requirements are met, resulting in a well-trained pre-trained repair and enhancement model.

[0033] As an improvement to the above method, the construction of the training dataset specifically includes:

[0034] Step T1) Collect x low-quality film videos, x≥100; for each low-quality film video, use an arbitrary scene segmentation method to segment it into n video segments according to the video scene; extract the frame closest to the middle moment in each video segment to obtain the corresponding set of low-quality film video images {I1,I2,…,I... n};

[0035] Step T2) for {I1,I2,…,I n Manual image restoration was performed, sequentially including scratch repair, smudge repair, detail enhancement, and recoloring, resulting in a restored image set {I′1,I′2,…,I′}. n This yields a low-quality-high-quality paired dataset corresponding to the low-quality film video.<I1,I′1> ,<I2,I′2> ,…, n ,I′ n >}; thus, we obtain a low-quality-high-quality paired dataset of w low-quality film videos;

[0036] Step T3) Collect z high-quality images with unique content {P1, P2, ..., P...} z}, satisfying z=25×n, degrades high-quality images to obtain a low-quality image set {P′1,P′2,…,P′}. z}, forming a pairwise dataset {<P1,P′1> ,<P2,P′2> ,…, <P z ,P′ z >};

[0037] Step T4) merges the datasets obtained in steps T2) and T3) to obtain the final training dataset {<I1,I′1>,…, n ,I′ n >,<P1,P′1> ,…, <P z ,P′ z >}.

[0038] As an improvement to the above method, in step T3),

[0039] ​​For image P z The degraded image P′ z The following calculations were performed: Among them, symbols This represents the convolution operation, where K is the degradation matrix and N is Gaussian noise. To ensure consistent image size after processing, zeros are appropriately padded around the image based on the kernel size. The degradation matrix K has five possible values:

[0040]

[0041] [0.15 0.2 0.3 0.2 0.15]、

[0042] The dataset {P1, P2, ..., P} is... z The images are randomly divided into 5 subsets, each containing 5×n images; different degradation processing matrices are used to process each of the five subsets.

[0043] On the other hand, this invention proposes a low-quality film image restoration and enhancement system based on dual networks, the system comprising:

[0044] The scene segmentation module is used to segment the original video to be repaired into several video segments according to the video scene using any scene segmentation method;

[0045] The extraction module is used to extract keyframes from each video segment to form a keyframe set, and to extract all video frames from the original video to be repaired to form a video frame set.

[0046] The dataset synthesis module is used to combine keyframes and corresponding keyframes that have been manually restored from images to form low-quality-high-quality paired datasets.

[0047] The transfer learning training module is used to divide the low-quality-high-quality pairwise datasets into several groups, input them sequentially into the pre-established and trained pre-trained repair and enhancement model, and use the transfer learning strategy to update the parameters to obtain the trained repair and enhancement model.

[0048] The restoration and enhancement module is used to input the set of video frames extracted from the module into the restoration and enhancement model to obtain a set of restored and enhanced images.

[0049] The video compositing module is used to re-synthesize the restored and enhanced image set into a video according to the original video frame rate, thereby realizing the restoration and enhancement of low-quality film images;

[0050] The pre-trained repair and enhancement model is a dual network, comprising a generator network and a degradation network; the trained repair and enhancement model is a generator network.

[0051] Compared with the prior art, the advantages of the present invention are:

[0052] This invention addresses practical applications by proposing a method for restoring and enhancing low-quality film images based on dual networks and transfer learning. By constructing a network architecture based on the dual-loop concept, the network fitting parameters can represent the mode transition between unpaired low-quality and high-quality datasets in real-world scenarios. The transfer of network parameters from the training data mode to the restored video mode is obtained by restoring keyframes. Furthermore, during dual network training, this invention first extracts multi-level features from the input image using an encoder to obtain corresponding hierarchical features. Then, the hierarchical features obtained by the encoder are processed using a hybrid attention mechanism to capture more complete high-level semantics and global contextual information. Subsequently, a cross-semantic attention mechanism is used to obtain reconstructed features across contexts. A simple feature fusion strategy is also employed to aggregate high-level contextual features and low-level spatial detail features, and low-rank optimization is used to further constrain the overall image features. The decoder is used to progressively aggregate high-level contextual features and low-level spatial details. This method can effectively fit the complex degradation function between low-quality film video images and ideal high-quality video images in practical applications. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the dual network structure provided by the present invention;

[0054] Figure 2 A schematic diagram of the hybrid attention module structure provided by the present invention;

[0055] Figure 3 This is a schematic diagram of the cross-semantic attention module structure provided by the present invention. Detailed Implementation

[0056] The method of this invention includes: constructing a network architecture based on the dual recurrence concept, enabling the network to effectively learn the complex degradation function between low-quality film video images and ideal high-quality images in practical applications; and then transferring network parameters from the pre-training data mode to the video data mode to be repaired by restoring keyframes. The dual network training specifically includes: performing hybrid attention processing on the hierarchical features extracted by the encoder to capture a more complete global context and high-level semantics; then obtaining cross-fusion features through a cross-semantic attention mechanism; and employing a feature fusion strategy to aggregate contextual features and spatial detail features. Low-rank optimization further enhances the overall constraint of the full-image features. The transfer learning specifically includes: manually restoring keyframes extracted from the video to be repaired; updating the parameters of the pre-trained model based on the transfer learning strategy; and using the updated model for image restoration and enhancement.

[0057] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0058] Example 1

[0059] The purpose of this invention is to overcome the technical defects of existing low-quality film image restoration and enhancement methods, and to propose a low-quality film image restoration and enhancement method based on dual networks and transfer learning. The method includes two parts: the construction of a pre-trained model for low-quality film video image restoration and enhancement based on dual networks and low-quality film video image restoration and enhancement based on transfer learning.

[0060] The construction of a pre-trained model for low-quality film video image inpainting and enhancement based on dual networks includes:

[0061] Step 1) Collect no fewer than w low-quality film videos, satisfying w ≥ 100. Assume all videos contain n scenes and u video frames. First, use an arbitrary scene segmentation method to segment the original video into n video segments {B1, V2, ..., V...}. n Each video segment contains a video scene. Assume the k-th video segment V... k The duration is t k Extract the I-frame closest to the middle moment of each video segment as training data, that is, for video segment V k Extract the closest to t k I-frame at time / 2 k As training data, the final image set {I1,I2,…,I...} is obtained. k ,…,I n}

[0062] Step 2) A professional graphic designer will work on the set {I1,I2,…,I…} n The images in the set {I′1,I′2,…,I′} are manually restored in sequence, including scratch repair, stain repair, detail enhancement, and recoloring, resulting in a restored image set {I′1,I′2,…,I′}. n This results in a low-quality-high-quality pairwise dataset tailored to real-world application scenarios.<I1,I′1> ,<I2,I′2> ,…, n ,I′ n >}.

[0063] Step 3) Collect z high-quality images with unique content {P1, P2, ..., P...} z}, satisfying z=25×n, degrades high-quality images to obtain a low-quality image set {P′1,P′2,…,P′}. z}, forming a pairwise dataset {<P1,P′1> ,<P2,P′2> ,…, <P z ,P′ z >}. For image P i The degraded image P′​i It can be obtained from the following calculations: Among them, symbols This represents a convolution operation, where K is the degradation processing matrix and N is Gaussian noise. To ensure consistent image size after processing, zeros are appropriately padded around the image based on the convolution kernel size. Matrix K has five possible values:

[0064]

[0065] [0.15 0.2 0.3 0.2 0.15]、

[0066] The dataset {P1, P2, ..., P} is... z The images are randomly divided into 5 subsets, each containing 5×n images. Different degradation processing matrices are applied to each of the five subsets to perform degradation processing.

[0067] Step 4) Merge the datasets obtained in Step 2) and Step 3) to obtain the final training dataset.<I1,I′1> ,…, n ,I′ n >,<P1,P′1> ,…, <P z ,P′ z The training dataset is randomly divided into m groups according to each pair of a.

[0068] Step 5) The dual network consists of a generator network and a degenerate network, such as... Figure 1 As shown. Figure 1 In the diagram, "H" represents the hybrid attention mechanism (module), "C" represents the cross-semantic attention mechanism (module), "F" represents the feature fusion mechanism (module), and "L" represents the low-rank optimization mechanism (module). The goal of the generative network is to inpaint and enhance low-quality images into high-quality images, while the goal of the degradation network is to degrade high-quality images into low-quality images. This mutual conversion between the two types of data allows the neural network to effectively infer and learn the transformation function between them. During training, a set of images is randomly selected, with low-quality images used as input to the generative network and high-quality images used as input to the degradation network. For simplicity, we will use low-quality-high-quality paired images in the following sections.<X,Y> The process is illustrated by taking the low-quality image X as an example, which is input into the generation model and the high-quality image Y as input into the degradation model.

[0069] Step 6) After inputting the low-quality image into the generation model, the encoder with an l-layer structure performs hierarchical feature extraction on the input image to obtain the hierarchical features f corresponding to each layer. i , i∈[1,l]. After inputting a high-quality image into the degradation model, an encoder with an l-layer structure performs hierarchical feature extraction on the input image to obtain the hierarchical feature f′ corresponding to each layer.​i , i∈[1,l];

[0070] Step 7) The hierarchical features f of the last layer in Step 6) l and f′ l Hybrid attention processing is performed separately to capture richer global high-level semantic context information, resulting in semantic fusion features f. l_f and f′ l_f Hybrid attention module structure such as Figure 2 As shown;

[0071] Step 8) The hierarchical features f output by the i-th layer of the generator network and the degenerate network encoder. i and f′ i , i∈[1,l-1], and the semantic fusion feature f l_f and f′ l_f Perform corresponding cross-semantic attention processing to optimize local spatial details, and obtain the local spatial detail optimized features f of the i-th layer. i_c and f′ i_c The cross-semantic attention module structure is as follows: Figure 3 As shown;

[0072] Step 9) Optimize the local spatial detail features f of the li-th layer of the generator network and the degenerate network encoder. l-i_c and f′ l-i_c With the output f of the corresponding i-th layer decoder i_d and f′ i_d After processing with the corresponding feature fusion mechanism, the output features o of each layer of the decoder are obtained. i and o′ i , i∈[1,l-1]. For o i and o′ i Perform low-rank optimization processing separately to obtain the low-rank optimized output result O. i and O′ i The input to the first layer of the decoder is the semantic fusion feature f. l_f and f′ l_f The output f after low-rank optimization in_d and f′ in_d ;

[0073] Step 10) The output f of the last layer of the decoder of the generator network and the degenerate network. l_d and f′ l_d Upsampling is performed on each image to obtain an output with the same resolution as the network input image. The output of the generator network is denoted as X. H The output of the degenerate network is denoted as Y. L ;

[0074] Step 11) Update the parameters in the generating and degenerate networks using gradient descent.

[0075] Step 12) Transfer the low-quality image Y L Input the generative model with a high-quality image X H Input the degradation model, repeat steps 6) to 10), and denot the output of the generator network as Y. L_H The output of the degenerate network is denoted as X. H_L ;

[0076] Step 13) Update the parameters in the generating and degenerate networks using gradient descent.

[0077] Step 14) Repeat steps 6) to 13) based on the training data, iterating repeatedly until the optimal combination of parameters is trained;

[0078] Step 7) specifically includes:

[0079] Step 7-1) For any input feature f, firstly, the feature is compressed using a convolutional layer to obtain a dense feature f′. The convolutional layer has a kernel size of 1*1, a stride of 1, and 128 channels. The dense feature f′ is then simultaneously input into the global context branch and the local context branch.

[0080] Step 7-2) The global context branch includes a global average pooling layer and a convolutional layer with a kernel size of 1*1. The global average pooling layer converts f′ into a semantic vector s. s and f′ are then multiplied by channels and convolved to obtain the global context information f. g The calculation formula is as follows:

[0081] s = GAP(f′)

[0082] f g =Sigmoid(conv(s⊙f′))

[0083] Wherein, “GAP” represents global average pooling, “⊙” represents channel multiplication, “conv” represents convolution operation with a kernel size of 1*1, a stride of 1, and 128 channels, and “Sigmoid” represents the Sigmoid activation function;

[0084] Step 7-3) The local context branch is composed of b convolutional layers with a kernel size of 3*3 and 128 channels stacked together. The local context branch converts f′ into local context weight w, and the calculation formula is as follows:

[0085] w=Sigmoid(stacked_conv(f′))

[0086] Where, stacked_conv represents stacked convolutional layers;

[0087] Step 7-4) Using w on f g Feature enhancement is performed to obtain the enhanced feature f. gw , will f gw Perform a convolution operation with a kernel size of 1*1, and fuse the output of the convolution operation with the dense feature f′ to obtain f″. The calculation formula is as follows:

[0088] f gw =w×f g

[0089] f″=Sigmoid(conv(f gw ))+f′

[0090] In this context, "×" represents dot product.

[0091] Furthermore, step 8) above specifically includes:

[0092] Step 8-1) The output f of the i-th layer of the encoder i and f′ i The features f are obtained by performing convolution operations separately. i_128 and f′ i_128 The kernel size is 1*1 and the number of channels is 128.

[0093] Step 8-2) f l_f and f′ l_f Convolution operations are performed separately, with a kernel size of 1*1 and 128 channels. Then, three interpolation upsampling processes are performed to obtain the feature f. l_f_128 and f′ l_f_128 ;

[0094] Step 8-3) For f i_128 and f l_f_128 Concatenating along the channel dimension yields features f that can represent spatial details and global context. i_cc , for f′ i_128 and f′ l_f_128 Concatenating along the channel dimension yields features f′ that can represent spatial details and global context. i_cc The calculation formula is as follows:

[0095] f i_cc =[f i_128 ,f l_f_128 ]

[0096] f′ i_cc =[f′ i_128 ,f′ l_f_128 ]

[0097] In this context, "[]" indicates cascading;

[0098] Step 8-4) for f i_cc and f′ i_cc Perform convolution operations separately, with a kernel size of 1*1 and 256 channels. Process the convolution results using an effective channel attention mechanism, and then compare the results with f. i and f′ i Perform dot product operations on each layer to obtain the local spatial detail optimization feature f of the i-th layer. i_c and f′ i_c The calculation formula is as follows:

[0099] f i_c =f i ×ECA(f i_cc )

[0100] f′ i_c =f′ i ×ECA(f′ i_cc )

[0101] In this context, "ECA" represents channel attention mechanism, and "×" represents dot product.

[0102] Furthermore, step 9) specifically includes:

[0103] Step 9-1) Use a feature fusion mechanism to optimize the local spatial details of the li-th layer of the generative network encoder, feature f. l-i_c With the output f of the corresponding i-th layer decoder i_d The concatenated feature maps are then stitched together, and a 1x1 convolutional layer is used to compress the stitched feature maps into 128 channels, yielding the output feature maps of each layer of the decoder. i A feature fusion mechanism is employed to optimize the local spatial details of the li-th layer of the degenerate network encoder, specifically feature f′. l-i_c The output f′ of the corresponding i-th layer decoder i_d The concatenated feature maps are then stitched together, and a 1x1 convolutional layer is used to compress the stitched feature maps into 128 channels, yielding the output feature o′ of each layer of the decoder. i .

[0104] Step 9-2) for o i and o′ i Perform low-rank optimization processing separately to obtain the low-rank optimized output result O. i and O′ i .

[0105] Furthermore, the low-rank optimization processing mechanism includes:

[0106] Step 9-2-1) For an input of any size C×H×W, where C represents the number of channels, H represents the height of the input feature map, and W represents the width of the input feature map, project the input features using M groups of projectors. Each group of projectors contains three types of projectors, used to project the input into a one-dimensional vector of dimensions C, H, and W, respectively. Each projector contains a pooling layer, a 1*1 convolutional layer, and a sigmoid activation function.

[0107] Step 9-2-2) After projection processing, three sets of feature matrices of size M×C×1×1, M×1×H×1, and M×1×1×W are obtained. The Kronecker product of the three matrices is then calculated to obtain a feature matrix of dimension M×C×H×W.

[0108] Step 9-2-3) Calculate the mean along dimension M to obtain a matrix of size C×H×W.

[0109] Furthermore, step 11) specifically includes:

[0110] Step 11-1) Calculate the error between the outputs of the generative and degenerate models and the true labels. This error calculation employs a combination of L1 loss, SSIM loss, and perceptual loss. The perceptual loss function, proposed by Johnson et al., is a feature extraction-based loss function commonly used in tasks such as image style transfer and image super-resolution. The perceptual loss utilizes a pre-trained VGG19 convolutional neural network to extract features from the input and target images, and uses the difference between their feature representations as the loss function to optimize the model and improve the expressive power of the neural network. The generative network error is calculated using the following formula:

[0111] Loss=αL1(X H ,Y)+βL SSIM (X H ,Y)+γL p (X H ,Y)

[0112] Where α, β, and γ represent the weighting coefficients of the three types of losses.

[0113] The error of the degenerate network is obtained by the following formula:

[0114] Loss′=αL1(Y L ,X)+βL SSIM (Y L ,X)+γL p (Y L ,X)

[0115] Where α, β, and γ represent the weighting coefficients of the three types of losses.

[0116] Step 11-2) Use the loss function defined in Step 11-1) to update the parameters in the generating network and the degenerate network using gradient descent.

[0117] Furthermore, step 13) specifically includes:

[0118] Step 13-1) Calculate the error between the outputs of the generative and degenerate models and the true labels. This error calculation employs a combination of L1 loss, SSIM loss, and perceptual loss. The perceptual loss function, proposed by Johnson et al., is a feature extraction-based loss function commonly used in tasks such as image style transfer and image super-resolution. The perceptual loss utilizes a pre-trained VGG19 convolutional neural network to extract features from the input and target images, and uses the difference between their feature representations as the loss function to optimize the model and improve the expressive power of the neural network. The generative network error is calculated using the following formula:

[0119] Loss=αL1(Y L_H ,Y)+βL SSIM (Y L_H ,Y)+γL p (Y L_H ,Y)

[0120] Where α, β, and γ represent the weighting coefficients of the three types of losses.

[0121] The error of the degenerate network is obtained by the following formula:

[0122] Loss′=αL1(X H_L ,X)+βL SSIM (X H_L ,X)+γL p (X H_L ,X)

[0123] Where α, β, and γ represent the weighting coefficients of the three types of losses.

[0124] Step 13-2) Use the loss function defined in Step 13-1) to update the parameters in the generating network and the degenerate network using gradient descent.

[0125] Low-quality film image inpainting and enhancement based on transfer learning specifically includes:

[0126] Step 1) Assuming the low-quality film video to be repaired and enhanced contains n scenes and u video frames, firstly, use an arbitrary scene segmentation method to segment the original video into n video segments {V1, V2, ..., V...}. n Each video segment contains a video scene. Assume the y-th video segment V... y The corresponding start and end times in the original video are respectively and Extract the I-frame closest to the midpoint of each video segment as the keyframe, i.e., for video segment V y Extract the closest I-frame at time I y As keyframes, the final set of keyframes {I1,I2,…,I...} is obtained. y ,…,I n Furthermore, extracting all video frames from the video yields a video frame set {F1, F2, ..., F}. y ,…,F u}

[0127] Step 2) A professional graphic designer manually modifies the set {I1, I2, ..., I... y ,…,I n The images in the set {I′1,I′2,…,I′} are repaired by sequentially performing scratch repair, smudge repair, detail restoration, and recoloring, resulting in a repaired image set {I′1,I′2,…,I′}. y ,…,I′ n This yields a low-quality-high-quality pairwise dataset for the current video.<I1,I′1> ,<I2,I′2> ,…, y ,I′ y >,…, n ,I′ n >}.

[0128] Step 3) Randomly divide the images required for training into m groups according to each pair a;

[0129] Step 4) Update the parameters of the pre-trained model obtained above and continue training for 50 epochs.

[0130] Step 5) Use the model trained in Step 4) on the set {F1, F2, ..., F...} y ,…,F u All images in} are repaired and enhanced to obtain a collection of repaired and enhanced images {F}. 1_o ,F 2_o ,…,F y_o ,…,F u_o}

[0131] Step 6) Combine the image collection {F} 1_o ,F 2_o ,…,F y_o ,…,F u_o The video is reassembled according to the original video frame rate.

[0132] Example 2

[0133] ​​Embodiment 2 of the present invention proposes a low-quality film image restoration and enhancement system based on dual networks, implemented based on the method of Embodiment 1. The system includes:

[0134] The scene segmentation module is used to segment the original video to be repaired into several video segments according to the video scene using any scene segmentation method;

[0135] The extraction module is used to extract keyframes from each video segment to form a keyframe set, and to extract all video frames from the original video to be repaired to form a video frame set.

[0136] The dataset synthesis module is used to combine keyframes and corresponding keyframes that have been manually restored from images to form low-quality-high-quality paired datasets.

[0137] The transfer learning training module is used to divide the low-quality-high-quality pairwise datasets into several groups, input them sequentially into the pre-established and trained pre-trained repair and enhancement model, and use the transfer learning strategy to update the parameters to obtain the trained repair and enhancement model.

[0138] The restoration and enhancement module is used to input the set of video frames extracted from the module into the restoration and enhancement model to obtain a set of restored and enhanced images.

[0139] The video compositing module is used to re-synthesize the restored and enhanced image set into a video according to the original video frame rate, thereby realizing the restoration and enhancement of low-quality film images;

[0140] The pre-trained insulation and enhancement model is a dual network, comprising a generator network and a degradation network; the trained insulation and enhancement model is a generator network.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for restoring and enhancing low-quality film images based on dual networks, the method comprising: Step 1) Use an arbitrary scene segmentation method to segment the original video to be repaired into several video segments according to the video scene; Step 2) Extract the keyframes of each video segment to form a keyframe set, and extract all video frames from the original video to be repaired to form a video frame set; Step 3) Combine the keyframes and the corresponding keyframes after manual image restoration into a low-quality-high-quality paired dataset; Step 4) Divide the low-quality-high-quality pairwise datasets into several groups, input them sequentially into the pre-established and trained pre-trained repair and enhancement model, and use the transfer learning strategy to update the parameters to obtain the trained repair and enhancement model. Step 5) Input the video frame set from Step 2) into the restoration and enhancement model from Step 4) to obtain the restored and enhanced image set; Step 6) Reassemble the image set from Step 5) into a video at the original video frame rate to restore and enhance low-quality film images; The pre-trained inpainting and enhancement model is a dual network, comprising a generator network and a degradation network; the trained inpainting and enhancement model is a generator network; the generator network of the pre-trained inpainting and enhancement model is used to inpaint and enhance low-quality images into high-quality images, and the degradation network of the pre-trained inpainting and enhancement model is used to degrade high-quality images into low-quality images; both the generator network and the degradation network include: a hybrid attention module, a cross-semantic attention module, a feature fusion module, and a low-rank optimization module; the input to the generator network is a low-quality-high-quality image pair.<X,Y> The low-quality image X is used as the input to the degradation network.<X,Y> High-quality image Y; The processing steps of the pre-trained repair and enhancement model specifically include: Step S1) Generate an encoder with an l-layer structure in the network to perform hierarchical feature extraction on the low-quality image X, obtaining the hierarchical features f corresponding to each layer. i , i∈[1,l]; In a degraded network, an encoder with an l-layer structure performs hierarchical feature extraction on a high-quality image Y, obtaining the hierarchical features f corresponding to each layer. i ′,i∈[1,l]; Step S2) Generates the hybrid attention module of the network to process the hierarchical features f of the last layer. l Hybrid attention processing is performed to capture richer global high-level semantic context information, resulting in semantic fusion features f. l_f ; The hybrid attention module of the degenerate network will apply the hierarchical features f of the last layer. l By performing hybrid attention processing, richer global high-level semantic context information is captured, resulting in semantic fusion features f′. l_f ; Step S3) Generates the network's cross-semantic attention module to process the hierarchical features f output from the i-th layer of the encoder. i , i∈[1,l-1], with semantic fusion feature f l_f Perform corresponding cross-semantic attention processing to optimize local spatial details and obtain the local spatial detail optimized feature f of the i-th layer. i_c ; The cross-semantic attention module of the degenerate network will take the hierarchical features f output from the i-th layer of the encoder. i ′, i∈[1,l-1], and semantic fusion feature f′ l_f Perform corresponding cross-semantic attention processing to optimize local spatial detail information, and obtain the local spatial detail optimized feature f′ of the i-th layer. i_c ; Step S4) Generates the feature fusion module of the network by optimizing the local spatial details of the encoder layer li using the feature f. l-i_c With the output f of the corresponding i-th layer decoder i_d After processing with the corresponding feature fusion mechanism, the output features o of each layer of the decoder are obtained. i , i∈[1,l-1]; the low-rank optimization module optimizes o i Perform low-rank optimization to obtain the low-rank optimized output result O. i The input to the first layer of the decoder is the semantic fusion feature f. l_f Output f after low-rank optimization in_d ; The feature fusion module of the degenerate network optimizes the local spatial details of the encoder layer li into feature f′. l-i_c The output f′ of the corresponding i-th layer decoder i_d After performing corresponding feature fusion processing, the output feature o′ of each layer of the decoder is obtained. i , i∈[1,l-1]; for o′ i Perform low-rank optimization to obtain the low-rank optimized output result O′. i The input to the first layer of the decoder is the semantic fusion feature f′. l_f The output f′ after low-rank optimization in_d ; Step S5) Calculate the output f of the last layer of the decoder in the generator network. l_d Upsampling is performed to obtain the generator network output X. H ; The output f′ of the last layer of the decoder of the degenerate network l_d Upsampling is performed to obtain the degenerate network output Y. L .

2. The method for restoring and enhancing low-quality film images based on dual networks according to claim 1, characterized in that, The low-quality film images include those with scratches, dirt, missing details, and / or color distortion.

3. The method for restoring and enhancing low-quality film images based on dual networks according to claim 1, characterized in that, The keyframe in step 2) is the frame closest to the middle moment in each video segment.

4. The method for restoring and enhancing low-quality film images based on dual networks according to claim 1, characterized in that, The manual image restoration in step 3) includes sequentially performing scratch repair, stain repair, detail enhancement, and recoloring.

5. The method for restoring and enhancing low-quality film images based on dual networks according to claim 1, characterized in that, The method also includes a training step for a pre-trained repair and enhancement model; specifically including: Construct a training dataset; the training dataset includes several low-quality-high-quality pairs of data. The training dataset is randomly divided into m groups according to each pair a; The low-quality images of each pair of data are sequentially input into the generator network, while the corresponding high-quality images are input into the degradation network. The parameters in the generator and degradation networks are updated using gradient descent until the training requirements are met, resulting in a well-trained pre-trained repair and enhancement model.

6. The method for restoring and enhancing low-quality film images based on dual networks according to claim 5, characterized in that, The construction of the training dataset specifically includes: Step T1) Collect x low-quality film videos, x≥100; for each low-quality film video, use an arbitrary scene segmentation method to segment it into n video segments according to the video scene; extract the frame closest to the middle moment in each video segment to obtain the corresponding set of low-quality film video images {I1,I2,…,I... n }; Step T2) for {I1,I2,…,I n Manual image restoration was performed, sequentially including scratch repair, smudge repair, detail enhancement, and recoloring, resulting in a restored image set {I′1,I′2,…,I′}. n This yields a low-quality-high-quality paired dataset corresponding to the low-quality film video.<I1,I′1> ,<I2,I′2> ,…, n ,I′ n }; This leads to a low-quality-high-quality paired dataset of w low-quality film videos;​ Step T3) Collect z high-quality images with unique content {P1, P2, ..., P...} z }, satisfying z=25×n, degrades high-quality images to obtain a low-quality image set {P′1,P′2,…,P′}. z }, forming a pairwise dataset {<P1,P′1> ,<P2,P′2> ,…, <P z ,P′ z >}; Step T4) merges the datasets obtained in steps T2) and T3) to obtain the final training dataset.<I1,I′1> ,…, n ,I′ n >,<P1,P′1> ,…, <P z ,P′ z >}.​ 7. The method for restoring and enhancing low-quality film images based on dual networks according to claim 6, characterized in that, In step T3), For image P z The degraded image P′ z The following calculations were performed: Among them, symbols This represents the convolution operation, where K is the degradation matrix and N is Gaussian noise. To ensure consistent image size after processing, zeros are appropriately padded around the image based on the kernel size. The degradation matrix K has five possible values: [0.15 0.2 0.3 0.2 0.15]、 The dataset {P1, P2, ..., P} is... z The images are randomly divided into 5 subsets, each containing 5×n images; different degradation processing matrices are used to process each of the five subsets.

8. A system for restoring and enhancing low-quality film images based on dual networks as described in claim 1, characterized in that, The system includes: The scene segmentation module is used to segment the original video to be repaired into several video segments according to the video scene using any scene segmentation method; The extraction module is used to extract keyframes from each video segment to form a keyframe set, and to extract all video frames from the original video to be repaired to form a video frame set. The dataset synthesis module is used to combine keyframes and corresponding keyframes that have been manually restored from images to form low-quality-high-quality paired datasets. The transfer learning training module is used to divide the low-quality-high-quality pairwise datasets into several groups, input them sequentially into the pre-established and trained pre-trained repair and enhancement model, and use the transfer learning strategy to update the parameters to obtain the trained repair and enhancement model. The restoration and enhancement module is used to input the video frame set from the extraction module into the restoration and enhancement model to obtain a set of restored and enhanced images; and The video compositing module is used to re-synthesize the restored and enhanced image set into a video according to the original video frame rate, thereby realizing the restoration and enhancement of low-quality film images; The pre-trained repair and enhancement model is a dual network, comprising a generator network and a degradation network; the trained repair and enhancement model is a generator network.

Images