Content repair method, device, equipment and computer program product
Through a generative adversarial network based on noise-containing images, using artificial intelligence and machine learning technology, the problem of poor repair results of various noise types in the existing technology has been solved, and the repair quality of old movies and other videos has been significantly improved.
Patent Information
- Application Number
- CN202210265786.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-17
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-03-17
AI Technical Summary
Existing content repair methods are not effective in content that contains multiple types of noise, especially for old movies and other videos, which are difficult to effectively repair picture quality problems caused by multiple types of noise.
Using a repair model based on noise-containing image training, a high-definition image is combined with noise-containing images through a generative adversarial network, and a generative adversarial network is trained to repair videos containing multiple types of noise, and image quality repair is achieved using artificial intelligence and machine learning technology.
It improves the generalization ability of the repair model, can effectively repair content containing many different noise types, significantly improves the repair quality of old movies and other videos, and solves the problem of poor applicability of a single noise type model.
Smart Images

Figure CN114627019B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a content repair method, device, equipment, computer program product and storage medium. Background Art
[0002] At present, the technology for content restoration usually uses the prior knowledge of specific types of noise to repair and process content with specific types of noise. The content referred to here can be videos, movies, etc. For example, the DnCNN algorithm adds Gaussian noise to clean pictures, trains a neural network that can process pictures containing Gaussian noise based on constructed paired data, decomposes the video containing Gaussian noise into frames containing Gaussian noise pictures, and processes them frame by frame using the trained neural network. However, the existing content restoration methods have poor restoration effects for content that may contain a variety of different types of noise, such as snowflakes, scratches, vertical lines, etc., such as old movies. This is because the neural network is trained based on the prior knowledge of a specific type of noise, and has poor applicability in restoration for content containing a variety of different types of noise. It can be seen that the restoration model obtained by the above existing technologies has poor generalization ability, which also affects the restoration effect of the content. Summary of the invention
[0003] The embodiments of the present application provide a content repair method, apparatus, device and computer program product to solve the problem in the prior art that the image quality of content containing various types of noise cannot be well restored and the applicability is poor.
[0004] The first aspect of the present application provides a content repair method, comprising:
[0005] Acquire an image to be repaired from the content to be repaired, wherein the image to be repaired contains noise;
[0006] Taking the image to be repaired as an input of a repair model, and using the repair model to repair the image to be repaired, to obtain a repaired image corresponding to the image to be repaired;
[0007] The restored content corresponding to the content to be restored is obtained by stitching the restored image corresponding to the image to be restored;
[0008] Among them, the restoration model is a model obtained by training with high-definition images and noisy images; the high-definition images are images whose image quality meets preset image quality requirements, and the noisy images are extracted from noisy content of the same type as the content to be restored.
[0009] A second aspect of the present application provides a content repair device, comprising:
[0010] An image acquisition unit, used for acquiring an image to be restored from the content to be restored, wherein the image to be restored contains noise;
[0011] An image restoration unit, configured to use the image to be restored as an input of a restoration model, and use the restoration model to restore the image quality of the image to be restored, so as to obtain a restored image corresponding to the image to be restored;
[0012] An image stitching unit, configured to stitch the restored image corresponding to the image to be repaired to obtain the restored content corresponding to the content to be repaired;
[0013] Among them, the restoration model is a model obtained by training with high-definition images and noisy images; the high-definition images are images whose image quality meets preset image quality requirements, and the noisy images are extracted from noisy content of the same type as the content to be restored.
[0014] A third aspect of the present application provides a content repair device, the device comprising a processor and a memory:
[0015] The memory is used to store program code and transmit the program code to the processor;
[0016] The processor is used to execute the steps of the content repair method provided by the first aspect of the present invention according to the instructions in the program code.
[0017] The fourth aspect of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of the content repair method provided in the first aspect of the present invention.
[0018] A fifth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the content repair method provided by the first aspect of the present invention.
[0019] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0020] The repair model used in the content repair method of the embodiment of the present application is obtained based on noisy image training. The noisy images are taken from original real noisy content. The noise types are real and rich, which improves the effectiveness of the repair model in repair. The repair model has strong generalization ability and can repair content to be repaired that contains a variety of different noise types, thereby improving the repair quality of content that may contain a variety of different noises. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 A flowchart of a content repair method provided in an embodiment of the present application;
[0022] Figure 2 A noisy image obtained from a real noisy image provided in an embodiment of the present application;
[0023] Figure 3 A flowchart of a repair model training method provided in an embodiment of the present application;
[0024] Figure 4 A flowchart of the specific steps of a repair model training method provided in an embodiment of the present application;
[0025] Figure 5 A flowchart of the specific steps of another repair model training method provided in an embodiment of the present application;
[0026] Figure 6 A flowchart of the specific steps of another repair model training method provided in an embodiment of the present application;
[0027] Figure 7 A flowchart of the specific steps of another repair model training method provided in an embodiment of the present application;
[0028] Figure 8 A schematic diagram of the old movie restoration model training process provided in an embodiment of the present application;
[0029] Fig. 9 A schematic diagram of an old movie restoration process provided in an embodiment of the present application;
[0030] Fig.10 A schematic diagram of the structure of a content repair device provided in an embodiment of the present application;
[0031] Fig.11 A schematic diagram of the structure of the server in the embodiment of the present application;
[0032] Fig.12 A schematic diagram of the structure of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION
[0033] As mentioned in the background art, the existing restoration methods for content containing a specific type of noise are not suitable for restoring the image quality of content containing multiple types of noise (such as old movies), and the restoration effect is not good.
[0034] The embodiments of the present application provide a content repair method, apparatus, device and computer program product. The repair model used in the content repair method is obtained based on noisy image training. The noisy image is taken from the original real noisy content. The noise type is real and rich, which improves the effectiveness of the repair model in repair. The repair model has strong generalization ability and can repair content to be repaired that contains a variety of different noise types.
[0035] The repair model can be obtained through artificial intelligence. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0036] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0037] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.
[0038] The solutions provided in the embodiments of the present application involve technologies such as machine learning of artificial intelligence, which are specifically described below through embodiments.
[0039] Below, the content repair method of the present application is specifically described by taking the restoration of old movies as an example.
[0040] The embodiment of the present application utilizes noisy images obtained from real old movie videos and multiple frames of continuous high-definition images obtained from high-definition videos to synthesize multiple frames of continuous degraded images. The multiple frames of continuous degraded images are used to simulate multiple frames of continuous degraded images in real old movie videos. The synthesized multiple frames of continuous degraded images and multiple frames of continuous high-definition images are used to train a generative adversarial network. The trained generative adversarial network can be used to repair old movie videos containing various types of noise.
[0041] First, some terms that may appear in the embodiments of the present application are explained.
[0042] Scratches, vertical lines, snowflake noise: During the production, recording, processing and transmission of old movies, the video quality is degraded due to the imperfections in the imaging system, recording equipment, transmission media and processing methods. Noise such as scratches, vertical lines and snowflake points appear, which seriously affects the quality of the video.
[0043] Video degradation: refers to the process of generating a low-quality video containing noise through several steps from a clean and clear HD video.
[0044] Video denoising: refers to the process of converting a noisy video into a clean / clear video through image or video processing algorithms or neural network generation.
[0045] Old movie restoration: For old movie-type videos, remove the noise contained in the video.
[0046] Neural network: refers to a network structure that uses a constructed operation process to operate on input data. It can be trained to fit specific data to obtain the ideal output.
[0047] The processing of noisy images can also be applied to computer vision technology. Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, find and measure targets, and further performs graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection.
[0048] The content repair method provided in this application can be applied to devices with data transmission functions, such as servers or terminals. Among them, the terminal device can specifically be a smart phone, a desktop computer, a laptop computer, a tablet computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc. The server can be an independent physical server, or it can be a server cluster or a distributed system composed of multiple physical servers. In addition, the server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. In the method described below, the executor of the method can be either a terminal device or a server.
[0049] Figure 1 A flow chart of a content repair method provided in an embodiment of the present application, the content repair method comprising:
[0050] S010: Obtain an image to be repaired from the content to be repaired, where the image to be repaired contains noise.
[0051] The image to be restored can be one or more images.
[0052] S020. Using the image to be repaired as an input of the repair model, and using the repair model to repair the image to be repaired, to obtain a repaired image corresponding to the image to be repaired.
[0053] Among them, the restoration model is a model obtained by training with high-definition images and noisy images; the high-definition image is an image whose quality meets the preset quality requirements, and the noisy image is extracted from the noisy content of the same type as the content to be restored.
[0054] Figure 2 The noise image obtained from the real noisy image provided in the embodiment of the present application. The noise type may include snowflakes, vertical lines, scratches, etc. The present application does not limit the noise type and number of types.
[0055] S030, obtaining restored content corresponding to the content to be restored by stitching restored images corresponding to the image to be restored.
[0056] In the embodiment of the present application, the content to be repaired generally includes more than one image, for example, the content may be a movie, a video, etc. Therefore, after repairing the image to be repaired, the complete repaired content may be obtained by splicing.
[0057] The repair model used in the content repair method of the embodiment of the present application is obtained based on noisy image training. The noisy images are taken from original real noisy content. The noise types are real and rich, which improves the effectiveness of the repair model in repair and makes the repair model have strong generalization ability. It can repair content to be repaired that contains a variety of different noise types, and improve the repair quality of content that may contain a variety of different noises.
[0058] As mentioned earlier, content restoration is achieved through the restoration model. To help you understand how the model can restore image quality, the following describes how the model is trained.
[0059] Figure 3 The training method of the repair model provided in this embodiment includes:
[0060] S110, obtaining a noisy image and a high-definition image.
[0061] Figure 4 The flowchart of step S110 provided in the embodiment of the present application includes:
[0062] S1101. Acquire N images from noisy content, and acquire N consecutive frames of images from the same high-definition content; all the N images contain noise, and the image quality of the N frames of images meets preset image quality requirements.
[0063] Wherein, N is an integer greater than 1. The noise positions in the N images are independent of each other.
[0064] In a specific example, for the training of an old movie restoration model, the noisy content can be a real old movie. The old movie is decomposed into multiple frames of old movie images containing noise, and a noise image is obtained from N old movie images. Since the noise position in each frame of the old movie image is independent of each other, the noise positions in the N noise images are independent of each other.
[0065] S1102 , cropping images of preset sizes containing noise from the N images as N noisy images, and cropping images of preset sizes at the same position of the N frames of images as N frames of high-definition images.
[0066] In a specific example, a region of size s*s is cropped in N images, and the cropping positions are independent of each other; a region of size s*s is cropped at the same position in N frames of images.
[0067] S1103, assigning serial numbers 1 to N to the N noisy images and the N high-definition images respectively, and establishing a one-to-one mapping relationship between the N noisy images and the N high-definition images based on the serial numbers; the mapping relationship is applied to the degradation process of the high-definition images and the training process of the restoration model.
[0068] In a specific example, N noisy images are marked as {m1, m2, ..., mN}, and N high-definition images are marked as {h1, h2, ... hN}. The one-to-one mapping relationship between N noisy images and N high-definition images is: m1 maps to h1, m2 maps to h2, ..., mN maps to hN.
[0069] S120: Perform image quality degradation processing on the high-definition image using the noisy image to obtain a degraded image corresponding to the high-definition image.
[0070] Figure 5 The flowchart of step S120 provided in the embodiment of the present application includes:
[0071] S1201, obtaining the mean value of the R channel, the mean value of the G channel, and the mean value of the B channel of the noisy image;
[0072] S1202 , subtract the R channel mean, the G channel mean, and the B channel mean from the color channel corresponding to the noisy image to obtain a centralized image corresponding to the noisy image.
[0073] The embodiment of the present application centralizes the three RGB channels of the noisy image to reduce the adverse effect of the noise brightness on the brightness of the high-definition image during the degradation process.
[0074] In a specific example, the centralization operation is expressed as mi*=mi-[mi r,mi g ,mi b ].
[0075] Where mi represents the i-th noisy image among N noisy images {m1,m2,…,mN}; mi r ,mi g ,mi b They respectively represent the pixel mean of the channel images of the three channels R, G, and B corresponding to the i-th noisy image; the centralized images of N noisy images {m1, m2, …, mN} are {m1*, m2*, …, mN*}, and mi* represents the centralized image of the i-th noisy image mi.
[0076] S1203. Randomly add or subtract a centered image corresponding to the noisy image from the high-definition image to obtain a degraded image corresponding to the high-definition image, wherein the degraded image is used to be input into the generator.
[0077] Figure 6 The flowchart of step S1203 provided in the embodiment of the present application includes:
[0078] S12031. Randomly determine a target high-definition image from N frames of high-definition images according to a preset probability, and use other images in the N frames of high-definition images as non-target high-definition images.
[0079] The preset probability value interval is (0,1).
[0080] S12032. Add the target high-definition image to the centered image corresponding to the noisy image with a mapping relationship, and limit the pixel values of the obtained image result to obtain a degraded image corresponding to the target high-definition image; and subtract the centered image corresponding to the noisy image with a mapping relationship from the non-target high-definition image, and limit the pixel values of the obtained image result to obtain a degraded image corresponding to the target high-definition image.
[0081] The pixel value is limited to the range [0,255].
[0082] In a specific example, p represents the preset probability, and p=0.5 is taken, that is, 50% of the images are randomly determined from N frames of high-definition images as target high-definition images, and the remaining 50% of the images are used as non-target high-definition images. The obtained degraded image is expressed as:
[0083]
[0084] Wherein, hi represents the i-th HD image in N HD images {h1, h2, …, hN}, and mi* represents the centralized image of the i-th noisy image mi. clip represents the addition of the target HD image to the centralized image corresponding to the noisy image with a mapping relationship, and the pixel values greater than 255 in the obtained image result are truncated to the maximum value 255; the subtraction of the non-target HD image from the centralized image corresponding to the noisy image with a mapping relationship, and the pixel values less than 0 in the obtained image result are truncated to the minimum value 0. Of course, the preset probability can also be set to other values according to actual needs, such as 0.3, 0.4, etc., and the value of the preset probability is not specifically limited here.
[0085] S130, using the high-definition image and the output of the generator to train the discriminator and the generator until the training is completed, and using the trained generator as the restoration model.
[0086] Figure 7 The flowchart of training the discriminator and the generator in step S130 provided in the embodiment of the present application, using the high-definition image and the output of the generator to train the discriminator and the generator includes:
[0087] S1301. Input the degraded images {l1, l2, ..., lN} corresponding to N frames of high-definition images {h1, h2, ..., hN} into the generator to obtain N frames of output images {G(l1), G(l2), ..., G(lN)} of the generator.
[0088] S1302. Obtain a reconstruction loss L1 based on N frames of high-definition images {h1, h2, ..., hN} and N frames of output images {G(l1), G(l2), ..., G(lN)}.
[0089] In this embodiment, the reconstruction loss is the average of the absolute values of the errors.
[0090] In a specific example, the reconstruction loss is expressed as:
[0091]
[0092] Wherein, hi represents the i-th frame in N high-definition image frames {h1, h2, …, hN}, and G(li) represents the i-th frame in N output image frames {G(l1), G(l2), …, G(lN)}.
[0093] S1303. Extract features of N frames of high-definition images {h1, h2, ..., hN} and features of N frames of output images {G(l1), G(l2), ..., G(lN)} through a pre-trained feature extraction network, and obtain the perceptual loss Lp based on the features of the N frames of high-definition images {h1, h2, ..., hN} and features of the N frames of output images {G(l1), G(l2), ..., G(lN)}.
[0094] In a specific example, the feature extraction network may be a deep neural network VGG, and the perceptual loss is expressed as:
[0095]
[0096] Among them, VGG(hi) represents the output of the feature extraction network corresponding to the i-th frame in N frames of high-definition images {h1,h2,…,hN}, and VGG(li) represents the output of the feature extraction network corresponding to the i-th frame in N frames of degraded images {l1,l2,…,lN}.
[0097] S1304. Based on the discriminator, the N frames of output images {G(l1), G(l2), …, G(lN)} are discriminated, and the discrimination loss Lg is obtained according to the discrimination results {D(G(l1)), D(G(l2)), …, D(G(lN))} of the N frames of output images {G(l1), G(l2), …, G(lN)}.
[0098] In a specific example, the discriminative loss is expressed as:
[0099]
[0100] Wherein, D(G(li)) represents the discrimination result of the discriminator on the i-th frame output image among N frames of output images {G(l1), G(l2),…, G(lN)}.
[0101] S1305. Adjust the generator according to at least one of the reconstruction loss L1, the perception loss Lp, and the discrimination loss Lg.
[0102] In a specific example, the loss function provided to the generator is L all =λ1L1+λ2Lp+λ3Lg
[0103] Among them, λ1, λ2, λ3 are weight coefficients for adjusting the proportion of the loss function, L1 is the reconstruction loss, Lp is the perceptual loss, and Lg is the discriminative loss.
[0104] S1306. Based on the discriminator, N frames of high-definition images {h1, h2, …, hN} are discriminated, and the loss of the discriminator in the collaborative training process of the generator and the discriminator is obtained according to the discrimination results {D(h1), D(h2), …, D(hN)} of the N frames of high-definition images {h1, h2, …, hN} and the discrimination results {D(G(l1)), D(G(l2)), …, D(G(lN))} of the N frames of output images {G(l1), G(l2), …, G(lN)}.
[0105] S1307, adjusting the discriminator according to the loss of the discriminator during the collaborative training process.
[0106] In a specific example, the loss of the discriminator during co-training is expressed as:
[0107]
[0108] Among them, D(hi) represents the discrimination result of the i-th frame of high-definition image in N frames of high-definition images {h1, h2, …, hN}, and D(G(li)) represents the discrimination result of the discriminator for the i-th frame output image in N frames of output images {G(l1), G(l2), …, G(lN)}.
[0109] Steps S110 - S130 are repeated until the performance of the repaired model reaches the expected level.
[0110] The repair model used for training in this application can adopt any network based on multi-frame input and multi-frame output, and this application does not limit this.
[0111] The embodiment of the present application uses continuous multi-frame images to train the restoration model, so that the restoration model can effectively utilize the different noise positions between different frame images based on the complementarity of the temporal information of the continuous frames of the images to be restored, and can handle the noise that a single-frame model cannot solve and the problem of unclear distinction of object elements contained in the content to be restored.
[0112] In a possible implementation, the content repair method repairs the content to be repaired by simultaneously repairing multiple consecutive frames of images to be repaired:
[0113] Acquiring an image to be restored from the content to be restored, including: decomposing the content to be restored frame by frame according to a time sequence to obtain a plurality of frames of images to be restored, and sending them into a queue to be restored in sequence.
[0114] The image to be repaired is used as the input of the repair model, and the repair model is used to repair the image to be repaired to obtain a repaired image corresponding to the image to be repaired. Specifically, the method includes: according to the order of images in the queue to be repaired, every N frames of the image to be repaired are sent to the repair model, and the repair model is used to obtain N frames of repaired images.
[0115] The restored content corresponding to the content to be restored is obtained by splicing the restored images corresponding to the image to be restored, specifically including: after all the images to be restored in the queue to be restored have been restored by the restoration model and have corresponding restored images, all the restored images are spliced frame by frame in time sequence to obtain the restored content corresponding to the content to be restored.
[0116] The embodiment of the present application can repair multiple consecutive frames at the same time. It can effectively utilize the different noise positions between different frames based on the complementarity of video timing information. It can handle the noise that cannot be solved by a single-frame model and the problem of unclear object elements contained in the video, thereby alleviating the flickering problem of old movies and making the repaired video more stable.
[0117] Figure 8 A schematic diagram of a repair model training process provided in an embodiment of the present application, wherein the repair model training process is as follows: Figure 8 The following steps and their details are not shown in the figure):
[0118] Step 1: Get a high-definition video without noise, and obtain multiple images with scratches, vertical lines, snowflake noise from real old movies.
[0119] Step 2: Disassemble the HD video frame by frame, randomly select N consecutive HD frames, and crop an s×s area, with each frame having the same cropping position.
[0120] Step 3: Randomly select N images containing scratched vertical lines and snowflake noise, and crop an s×s area. The cropping position of each image is independent of each other, and obtain N s×s noisy images.
[0121] Step 4: Center N noisy images of size s×s. For each noisy image, subtract the pixel mean of the corresponding channel image from the channel image of the three RGB channels to obtain N centralized images.
[0122] Step 5: For N frames of s×s high-definition images and N centralized images, obtain N frames of degraded images by adding noise.
[0123] Step 6: Input N frames of degraded images into the generator. The N frames of images output by the generator and N frames of high-definition images are input into the discriminator. The discriminator determines the probability that the input image is a high-definition image, and obtains the reconstruction loss, perceptual loss, and discriminator loss. The three losses are weighted and summed, and the gradient is transmitted back to the generator layer by layer through the back propagation algorithm. The generator is adjusted to continuously improve the performance of the generator (i.e., the restoration model). The discriminator is adjusted according to the loss of the discriminator during the collaborative training process.
[0124] Step 7. Repeat steps 2-6 until the performance of the repair model (i.e., the generator) reaches the expected level. At this time, the model output should be noise-free, clear, and stable.
[0125] The embodiments of this application use multiple noisy images and consecutive multiple frames of high-definition images obtained from real old movies to synthesize consecutive multiple frames of degraded images to simulate real old movie frames to be restored. The synthesized degraded images and high-definition images are used to train a restoration model. Since the types of noises taken from real old movies are real and rich, the utility of the restoration model in restoration can be improved during the training process, making the trained restoration model have strong generalization ability, capable of restoring the content to be restored containing multiple different types of noises and improving the restoration quality of the content that may contain multiple different noises.
[0126] Fig. 9 The following is a schematic diagram of the old movie restoration process provided by the embodiments of this application. The old movie restoration process is as follows:
[0127] Step a: Frame by frame, disassemble the old movie containing scratch vertical lines and snow noise into multiple frames of images to be restored in chronological order and send them into the queue of images to be restored in sequence.
[0128] Step b: According to the order of the images in the queue of images to be restored, every N frames of images to be restored are sent into the trained generator (i.e., the restoration model). The generator outputs N frames of restored images. If the remaining number of frames n < N at the end, the last frame is filled (N - n) times to make up N frames and then sent into the generator. In the output stage, the first n frames are taken.
[0129] Step c: Merge all the restored high-definition images frame by frame in chronological order to obtain the restored movie video with scratch vertical lines and snow noise removed.
[0130] The embodiments of this application can restore multiple consecutive frames simultaneously. Based on the complementarity of video chronological information, it can effectively utilize the characteristic that the noise positions are different between different frames, and can handle the problems of noises that cannot be solved by single-frame models and the unclear resolution of object elements contained in the video. Furthermore, it can alleviate the flickering problem of old movies and make the restored video more stable.
[0131] The content restoration method of the embodiments of this application uses a restoration model trained based on noisy images. The noisy images are taken from the original real noisy content. The types of noises are real and rich, which improves the utility of the restoration model in restoration. The restoration model has strong generalization ability and can restore the content to be restored containing multiple different types of noises.
[0132] Fig.10 The following is a schematic diagram of the structure of a content restoration device provided by the embodiments of this application, including:
[0133] An image acquisition unit, configured to acquire images to be restored from the content to be restored, and the images to be restored contain noises.
[0134] The image restoration unit is used to use the image to be restored as the input of the restoration model, and use the restoration model to restore the image quality of the image to be restored to obtain a restored image corresponding to the image to be restored.
[0135] The image stitching unit is used to stitch the restored image corresponding to the image to be repaired to obtain the restored content corresponding to the content to be repaired.
[0136] Among them, the restoration model is a model obtained by training with high-definition images and noisy images; the high-definition image is an image whose quality meets the preset quality requirements, and the noisy image is extracted from the noisy content of the same type as the content to be restored.
[0137] In a possible implementation, the content repair device further includes a repair model training unit, and the repair model training unit includes:
[0138] A first image acquisition unit, used for acquiring a noisy image and a high-definition image;
[0139] A degraded image generating unit is used to perform image quality degradation processing on a high-definition image using a noisy image to obtain a degraded image corresponding to the high-definition image; the degraded image is used to be input into a generator;
[0140] The training subunit is used to train the discriminator and the generator using the high-definition image and the output of the generator until the training is completed, and the trained generator is used as the restoration model.
[0141] In a possible implementation, the degraded image generating unit includes:
[0142] A noise image centralization unit is used to obtain the R channel mean, the G channel mean and the B channel mean of the noisy image; subtract the R channel mean, the G channel mean and the B channel mean from the color channel corresponding to the noisy image to obtain a centralized image corresponding to the noisy image;
[0143] The degraded image synthesis unit is used to randomly add or subtract the centralized image corresponding to the noisy image from the high-definition image to obtain the degraded image corresponding to the high-definition image.
[0144] In a possible implementation, the first image acquisition unit further includes:
[0145] The first image acquisition subunit is used to acquire N images from the noisy content and acquire N consecutive frames of images from the same high-definition content; the N images all contain noise, and the image quality of the N frames of images all meets the preset image quality requirements; N is an integer greater than 1;
[0146] An image cropping unit, used to crop images of preset sizes containing noise from N images as N noisy images, and crop images of preset sizes at the same position of N frames of images as N frames of high-definition images;
[0147] A mapping relationship establishing unit is used to assign serial numbers 1 to N to N noisy images and N frames of high-definition images respectively, and establish a one-to-one mapping relationship between the N noisy images and the N frames of high-definition images based on the serial numbers; the mapping relationship is applied to the degradation process of the high-definition images and the training process of the restoration model.
[0148] In a possible implementation, the degraded image synthesis unit further includes:
[0149] A target high-definition image determination unit is used to randomly determine a target high-definition image from N frames of high-definition images according to a preset probability, and other images in the N frames of high-definition images are regarded as non-target high-definition images; the value interval of the preset probability is (0,1);
[0150] The degraded image synthesis subunit is used to add the target high-definition image to the centered image corresponding to the noisy image with a mapping relationship, and limit the pixel values of the obtained image result to obtain the degraded image corresponding to the target high-definition image; and to subtract the centered image corresponding to the noisy image with a mapping relationship from the non-target high-definition image, and limit the pixel values of the obtained image result to obtain the degraded image corresponding to the target high-definition image.
[0151] In a possible implementation, the training subunit further includes
[0152] An input unit is used to input the degraded images {l1,l2,…,lN} corresponding to N frames of high-definition images {h1,h2,…,hN} into the generator to obtain N frames of output images {G(l1),G(l2),…,G(lN)} of the generator;
[0153] The adjustment unit is used to obtain a reconstruction loss L1 according to N frames of high-definition images {h1, h2, ..., hN} and N frames of output images {G(l1), G(l2), ..., G(lN)}; extract features of the N frames of high-definition images {h1, h2, ..., hN} and features of the N frames of output images {G(l1), G(l2), ..., G(lN)} through a pre-trained feature extraction network, and obtain a perceptual loss Lp according to the features of the N frames of high-definition images {h1, h2, ..., hN} and features of the N frames of output images {G(l1), G(l2), ..., G(lN)}; discriminate the N frames of output images {G(l1), G(l2), ..., G(lN)} based on a discriminator, and obtain a perceptual loss Lp according to the features of the N frames of high-definition images {h1, h2, ..., hN} and features of the N frames of output images {G(l1), G(l2), ..., G(lN)}. The discriminant loss Lg is obtained based on the discrimination results {D(G(l1)), D(G(l2)), …, D(G(lN))}; the generator is adjusted according to at least one of the reconstruction loss L1, the perceptual loss Lp and the discriminant loss Lg; N frames of high-definition images {h1, h2, …, hN} are discriminated based on the discriminator, and the loss of the discriminator in the collaborative training process of the generator and the discriminator is obtained according to the discrimination results {D(h1), D(h2), …, D(hN)} of the N frames of high-definition images {h1, h2, …, hN} and the discrimination results {D(G(l1)), D(G(l2)), …, D(G(lN))} of the N frames of output images {G(l1), G(l2), …, G(lN)}; the discriminator is adjusted according to the loss of the discriminator in the collaborative training process.
[0154] In one possible implementation, when the content repair device repairs the content to be repaired by simultaneously repairing N consecutive frames of images to be repaired, the image acquisition unit is also used to decompose the content to be repaired frame by frame according to the time sequence to obtain multiple frames of images to be repaired, and send them into the queue to be repaired in sequence.
[0155] The image restoration unit is further used to send every N frames of the images to be restored into the restoration model according to the sequence of the images in the restoration queue, and obtain N frames of restored images by using the restoration model;
[0156] The image stitching unit is also used to stitch all the repaired images frame by frame in time sequence after all the images to be repaired in the repair queue have been repaired by the repair model to obtain corresponding repaired images, so as to obtain the repaired content corresponding to the content to be repaired.
[0157] Those skilled in the art will appreciate that the content repair device of this embodiment may also execute other methods in the content repair method of the embodiment of the present application, which will not be elaborated in this embodiment.
[0158] The repair model used in the embodiment of the present application is obtained based on noisy image training. The noisy image is taken from the original real noisy content. The noise types are real and rich, which improves the effectiveness of the repair model in repair. The repair model has strong generalization ability and can repair content to be repaired that contains a variety of different noise types.
[0159] The structures of the control devices for implementing the above repair methods are introduced in the following in the server form and the terminal device form respectively.
[0160] Fig.11 : This is a schematic diagram of a server structure provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 922 (for example, one or more processors) and a memory 932, and one or more storage media 930 (for example, one or more mass storage devices) storing application programs 942 or data 944. Among them, the memory 932 and the storage medium 930 can be short-term storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 922 can be configured to communicate with the storage medium 930 and execute a series of instruction operations in the storage medium 930 on the server 900.
[0161] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems 941, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.
[0162] The steps performed by the server in the above embodiment can be based on the Fig.11 The server structure shown.
[0163] The CPU 922 is used to execute the following steps:
[0164] Acquire an image to be repaired from the content to be repaired, wherein the image to be repaired contains noise;
[0165] Taking the image to be repaired as an input of a repair model, and using the repair model to repair the image to be repaired, to obtain a repaired image corresponding to the image to be repaired;
[0166] The restored content corresponding to the content to be restored is obtained by stitching the restored image corresponding to the image to be restored;
[0167] Among them, the restoration model is a model obtained by training with high-definition images and noisy images; the high-definition images are images whose image quality meets preset image quality requirements, and the noisy images are extracted from noisy content of the same type as the content to be restored.
[0168] The present application also provides another control device, such as Fig.12 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:
[0169] Fig.12 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Fig.12 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Fig.12 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0170] Combine the following Fig.12 A detailed introduction to the various components of the mobile phone:
[0171] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: LowNoiseAmplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications may use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.
[0172] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0173] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
[0174] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Fig.12 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.
[0175] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.
[0176] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.
[0177] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Fig.12 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.
[0178] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby collecting overall data and information of the mobile phone. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.
[0179] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.
[0180] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.
[0181] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:
[0182] Acquire an image to be repaired from the content to be repaired, wherein the image to be repaired contains noise;
[0183] Taking the image to be repaired as an input of a repair model, and using the repair model to repair the image to be repaired, to obtain a repaired image corresponding to the image to be repaired;
[0184] The restored content corresponding to the content to be restored is obtained by stitching the restored image corresponding to the image to be restored;
[0185] Among them, the restoration model is a model obtained by training with high-definition images and noisy images; the high-definition images are images whose image quality meets preset image quality requirements, and the noisy images are extracted from noisy content of the same type as the content to be restored.
[0186] The embodiment of the present application also provides a computer-readable storage medium for storing program code, where the program code is used to execute any implementation of a content repair method described in the aforementioned embodiments.
[0187] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any one of the implementations of the content repair method described in the aforementioned embodiments.
[0188] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0189] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0190] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0191] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.
[0193] The repair model used in the embodiment of the present application is obtained based on noisy image training. The noisy image is taken from the original real noisy content. The noise types are real and rich, which improves the effectiveness of the repair model in repair. The repair model has strong generalization ability and can repair content to be repaired that contains a variety of different noise types.
[0194] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A content repair method, characterized in that: include: Acquire an image to be repaired from the content to be repaired, wherein the image to be repaired contains noise; Taking the image to be repaired as an input of a repair model, and using the repair model to repair the image to be repaired, to obtain a repaired image corresponding to the image to be repaired; The restored content corresponding to the content to be restored is obtained by stitching the restored image corresponding to the image to be restored; The training to obtain the repair model includes the following steps: Acquire a noisy image and a high-definition image; the high-definition image is an image whose image quality meets preset image quality requirements, and the noisy image is extracted from noisy content of the same type as the content to be repaired; Obtaining an R channel mean, a G channel mean, and a B channel mean of the noisy image; Subtract the R channel mean, the G channel mean, and the B channel mean from the color channel corresponding to the noisy image to obtain a centralized image corresponding to the noisy image; Randomly adding or subtracting the centered image corresponding to the noisy image from the high-definition image to obtain a degraded image corresponding to the high-definition image; the degraded image is used to be input into the generator; The discriminator and the generator are trained using the high-definition image and the output of the generator until the training is completed, and the trained generator is used as the restoration model.
2. The method according to claim 1, characterized in that: The step of acquiring the noisy image and the high-definition image comprises: Obtaining N images from the noisy content, and obtaining N consecutive frames of images from the same high-definition content; the N images all contain noise, and the image quality of the N frames of images all meets the preset image quality requirements; N is an integer greater than 1; Cropping images of preset sizes containing noise from the N images as N noisy images, and cropping images of preset sizes at the same position of the N frames of images as N frames of high-definition images; Serial numbers 1 to N are assigned to the N noisy images and the N frames of the high-definition images respectively, and a one-to-one mapping relationship between the N noisy images and the N frames of the high-definition images is established based on the serial numbers; the mapping relationship is applied to the degradation process of the high-definition images and the training process of the restoration model.
3. The method according to claim 2, characterized in that The step of randomly adding or subtracting the centralized image corresponding to the noisy image from the high-definition image to obtain a degraded image corresponding to the high-definition image includes: A target high-definition image is randomly determined from the N frames of high-definition images according to a preset probability, and other images in the N frames of high-definition images are used as non-target high-definition images; the value interval of the preset probability is (0,1); The target high-definition image is added with the centered image corresponding to the noisy image having the mapping relationship, and the pixel values of the obtained image result are restricted to obtain a degraded image corresponding to the target high-definition image; and the centered image corresponding to the noisy image having the mapping relationship is subtracted from the non-target high-definition image, and the pixel values of the obtained image result are restricted to obtain a degraded image corresponding to the target high-definition image.
4. The method according to claim 2 or 3, characterized in that: The method of using the high-definition image and the output of the generator to train the discriminator and the generator includes: Inputting the degraded images {l1, l2, ..., lN} corresponding to N frames of high-definition images {h1, h2, ..., hN} into the generator to obtain N frames of output images {G(l1), G(l2), ..., G(lN)} of the generator; Obtain the reconstruction loss L1 based on N frames of high-definition images {h1, h2, …, hN} and N frames of output images {G(l1), G(l2), …, G(lN)}; The features of N frames of high-definition images {h1, h2, ..., hN} and the features of N frames of output images {G(l1), G(l2), ..., G(lN)} are extracted through a pre-trained feature extraction network, and the perceptual loss Lp is obtained based on the features of N frames of high-definition images {h1, h2, ..., hN} and the features of N frames of output images {G(l1), G(l2), ..., G(lN)}; Based on the discriminator, the N frames of output images {G(l1), G(l2), ..., G(lN)} are discriminated, and the discrimination loss Lg is obtained according to the discrimination results {D(G(l1)), D(G(l2)), ..., D(G(lN))} of the N frames of output images {G(l1), G(l2), ..., G(lN)}; Adjusting the generator according to at least one of the reconstruction loss L1, the perceptual loss Lp, and the discriminative loss Lg; Based on the discriminator, N frames of high-definition images {h1, h2, ..., hN} are discriminated, and according to the discrimination results {D(h1), D(h2), ..., D(hN)} of the N frames of high-definition images {h1, h2, ..., hN} and the discrimination results {D(G(l1)), D(G(l2)), ..., D(G(lN))} of the N frames of output images {G(l1), G(l2), ..., G(lN)}; The discriminator is adjusted according to the loss of the discriminator during the collaborative training process.
5. The method according to claim 2 or 3, characterized in that: The step of obtaining the image to be restored from the content to be restored comprises: The content to be repaired is disassembled frame by frame according to the time sequence to obtain multiple frames of images to be repaired, and sent to a queue to be repaired in sequence; The method uses the image to be repaired as an input of a repair model, performs image quality repair on the image to be repaired using the repair model, and obtains a repaired image corresponding to the image to be repaired, including: According to the sequence of images in the queue to be repaired, every N frames of images to be repaired are sent to the repair model, and N frames of repaired images are obtained by using the repair model; The step of obtaining the restored content corresponding to the content to be restored by stitching the restored image corresponding to the image to be restored includes: After all the images to be repaired in the queue to be repaired have been repaired by the repair model and have corresponding repaired images, all the repaired images are spliced frame by frame in time sequence to obtain the repaired content corresponding to the content to be repaired.
6. A content repair device, characterized in that: include: An image acquisition unit, used for acquiring an image to be restored from the content to be restored, wherein the image to be restored contains noise; An image restoration unit, configured to use the image to be restored as an input of a restoration model, and use the restoration model to restore the image quality of the image to be restored, so as to obtain a restored image corresponding to the image to be restored; An image stitching unit, configured to stitch the restored image corresponding to the image to be repaired to obtain the restored content corresponding to the content to be repaired; Repair model training unit; The repair model training unit comprises: A first image acquisition unit is used to acquire a noisy image and a high-definition image; the high-definition image is an image whose image quality meets a preset image quality requirement, and the noisy image is extracted from the noisy content of the same type as the content to be repaired; A noise image centralization unit is used to obtain the R channel mean, the G channel mean and the B channel mean of the noisy image; subtract the R channel mean, the G channel mean and the B channel mean from the color channel corresponding to the noisy image to obtain a centralized image corresponding to the noisy image; A degraded image synthesis unit, used for randomly adding or subtracting the centralized image corresponding to the noisy image from the high-definition image to obtain a degraded image corresponding to the high-definition image; the degraded image is used for inputting into the generator; A training subunit is used to train the discriminator and the generator using the high-definition image and the output of the generator until the training is completed, and the trained generator is used as the restoration model.
7. The device according to claim 6, characterized in that The first image acquisition unit comprises: A first image acquisition subunit is used to acquire N images from the noisy content and acquire N consecutive frames of images from the same high-definition content; the N images all contain noise, and the image quality of the N frames of images all meets the preset image quality requirements; N is an integer greater than 1; An image cropping unit, used for cropping images of preset sizes containing noise from the N images as the N noisy images, and cropping images of preset sizes at the same position of the N frames of images as the N frames of high-definition images; A mapping relationship establishing unit is used to assign serial numbers 1 to N to the N noisy images and the N frames of the high-definition images, respectively, and establish a one-to-one mapping relationship between the N noisy images and the N frames of the high-definition images based on the serial numbers; the mapping relationship is applied to the degradation process of the high-definition images and the training process of the restoration model.
8. A content repair device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the steps of the content repair method according to any one of claims 1 to 5 according to the instructions in the program code.
9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the content repair method described in any one of claims 1 to 5 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to enable a computer device to execute the steps of the content repair method according to any one of claims 1 to 5.
