A random window pixel alignment training method and system for video restoration

Abstract

The application provides a random window pixel alignment training method and system for video restoration, which comprises: processing an original video through a generation subnetwork to output a latent representation of a video restoration result in a latent space; selecting a local area on the latent representation as a latent domain window; mapping the latent domain window to a pixel domain coordinate system according to the up-sampling scale relationship between the latent space and the pixel domain to determine a corresponding pixel decoding subdomain; performing local pixel-level decoding processing on the pixel decoding subdomain through a decoder to obtain a local predicted image block; calculating a reconstruction loss and a perception loss based on the comparison between the local predicted image block and a corresponding area of a true value video, and calculating a local temporal difference constraint on the pixel decoding subdomains of adjacent frames in the time dimension to form a local joint loss; and obtaining a global joint loss based on the local joint loss. The application significantly reduces the peak memory occupation and the calculation time delay, and improves the training throughput in a single machine or multi-machine environment.

Description

Technical Field

[0001] This application relates to computer vision and multimedia processing technology, and more specifically, to a random window pixel alignment training method and system for video restoration. Background Technology

[0002] As video applications continue to expand from mobile devices to large-screen scenarios, generative video restoration models are gradually becoming a core technology for improving viewing quality. However, the joint training of high resolution and long duration brings dual pressure on GPU memory and computing power: on the one hand, generative models often model in the latent space, and ultimately still need to decode the results back to the pixel domain to accept pixel-level constraints; on the other hand, directly decoding and supervising the entire image will lead to a sharp increase in GPU memory peaks and create unbearable computational overhead in the spatiotemporal dimension. In engineering practice, a common approach is to cut the image or video into several spatial sub-blocks for training, but if the block positions do not strictly correspond to the sampling stride of the latent space, there will be misalignment or boundary effects between the supervision region generated in the pixel domain and the generation region in the latent space, resulting in problems such as unstable gradient feedback, increased edge artifacts, and difficulty in training convergence. In addition, the temporal dimension of the video sequence further amplifies the above contradictions: if the video is fully decoded in the temporal dimension and then the pixel loss is calculated frame by frame, the GPU memory and latency will accumulate approximately linearly with the sequence length, severely restricting training throughput; if supervision is only applied to a small number of frames, the coverage and representativeness of pixel constraints will be weakened.

[0003] A search revealed a Chinese patent application with application number 202310061184.1, which discloses a blind face recovery method based on domain-aligned GAN prior. This method constructs a network containing a domain-aligned GAN inversion branch and a feature extraction fusion branch, and trains it using image domain and latent spatial domain loss functions. It can recover detailed high-definition images from low-resolution face images. However, this method is designed for static images and does not consider the training throughput constraints of the time dimension of video sequences. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the purpose of this application is to provide a random window pixel alignment training method and system for video restoration.

[0005] A first aspect of this application provides a random window pixel alignment training method for video restoration, comprising: The original video and the ground truth video are obtained, along with a video restoration model consisting of a generator sub-network and a decoder. The original video is processed using the generative sub-network, and a latent representation of the video restoration result is output in the latent space; One or more random or semi-random local regions are selected on the latent representation as latent windows; Based on the upsampling scale relationship between the latent space and the pixel domain, the latent window is mapped to the pixel domain coordinate system to determine the corresponding pixel decoding subdomain; For the pixel decoding subdomain, a decoder is used to perform local pixel-level decoding processing to obtain local predicted image blocks; Based on the comparison between the local predicted image patch and the corresponding region of the ground truth video, the reconstruction loss and perceptual loss are calculated, and local temporal difference constraints are calculated on the pixel decoding subdomain of adjacent frames in the temporal dimension to form a local joint loss. The local joint loss is processed to obtain the global joint loss, and the training of the video restoration model is completed.

[0006] Optionally, the step of mapping the latent window to the pixel domain coordinate system based on the upsampling scale relationship between the latent space and the pixel domain to determine the corresponding pixel decoding subdomain includes: Obtain the upsampling factor from the latent space to the pixel domain; Based on the upsampling factor, the latent window is mapped to the pixel domain coordinate system to obtain the corresponding pixel decoding subdomain.

[0007] Optionally, before mapping, the periphery of the latent window is expanded by several pixels to form an expanded latent window; Perform mapping on the expanded latent window; After mapping is completed, the region corresponding to the expanded pixel is cropped to obtain a restoration result with the same size as the latent window, which is used as the pixel decoding subdomain.

[0008] Optionally, during the mapping of the latent window to the pixel domain along the time dimension, each time segment uses the same upsampling factor and the same method for calculating the mapping coordinates.

[0009] Optionally, the local joint loss is: L represents the local joint loss. Indicates the reconstruction loss. Indicates perceived loss. Indicates time difference constraints; The reconstruction loss is achieved by using the mean absolute loss L1, the mean square error loss L2, or the Charbonnier loss. The perceptual loss employs a loss function based on a feature space distance metric; The temporal difference constraint uses the L2 distance between adjacent frames of the restored frame and the reference frame in the ground truth video as the loss.

[0010] Optionally, processing the local joint loss to obtain the global joint loss includes any of the following cases: When the latent window uses random uniform sampling, the local joint loss is scaled according to the ratio of the latent window area to the full area of ​​the latent representation to obtain the global joint loss; When the latent window uses semi-random non-uniform sampling, a sampling importance weight is applied to the local joint loss to obtain the global joint loss.

[0011] A second aspect of this application provides a random window pixel alignment training system for video restoration, comprising: Preparation module: Acquires the original video and ground truth video, as well as a video restoration model consisting of a generator sub-network and a decoder; Latent space module: The generator subnetwork is used to process the original video, and the latent representation of the video restoration result is output in the latent space; Latent window sampling module: Selects one or more random or semi-random local regions on the latent representation as latent windows; Alignment mapping module: Based on the upsampling scale relationship between the latent space and the pixel domain, the latent window is mapped to the pixel domain coordinate system to determine the corresponding pixel decoding subdomain; Local decoding module: For the pixel decoding subdomain, a decoder is used to perform local pixel-level decoding processing to obtain local predicted image blocks; Local pixel supervision module: Based on the comparison between the local predicted image patch and the corresponding region of the ground truth video, it calculates the reconstruction loss and the perceptual loss, and calculates the local temporal difference constraint on the pixel decoding subdomain of adjacent frames in the temporal dimension to form a local joint loss; Loss aggregation and training update module: Processes the local joint loss to obtain the global joint loss and completes the training of the video restoration model.

[0012] A third aspect of this application provides a video restoration method, comprising: Obtain the video to be restored; The video to be restored is input into the video restoration model trained by any of the random window pixel alignment training methods or training systems described above, to obtain the restored video.

[0013] A third aspect of this application provides a video restoration method, comprising: Obtain the video to be restored; The video to be restored is input into the video restoration generation network trained using the random window pixel alignment training method or the training system described above, to obtain the restored video.

[0014] A fourth aspect of this application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform any of the methods described above, or to run the system described above.

[0015] A fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described therein or to run the system described therein.

[0016] The random window pixel alignment training method for video restoration provided in this application selects a random latent window in the latent space and performs local decoding and pixel supervision only on the pixel decoding subdomain that corresponds uniquely to it. This significantly reduces the GPU memory and computational overhead while ensuring the expected equivalence of pixel-level constraints. Geometrically consistent alignment mapping avoids block misalignment and boundary artifacts. Local temporal difference constraints are added to the video sequence to suppress flicker and improve stability in a local range.

[0017] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a random window pixel alignment training method for video restoration according to an exemplary embodiment; Figure 2 This is a framework diagram illustrating a random window pixel alignment training method for video restoration according to an exemplary embodiment.

[0019] Figure 3 This is a schematic diagram illustrating the structure of a random window pixel alignment training system for video restoration according to an exemplary embodiment. Detailed Implementation

[0020] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0021] Terminology Explanation: Latent window: In video data processing tasks, a spatial or spatiotemporal window within a certain range is defined for the latent space representation of the input (e.g., extracted via VAE, Diffusion, or other latent encoders). This window is used to select local regions for sampling, loss calculation, or feature operations during the training or inference phases to improve the model's sensitivity to or efficiency in response to local changes. The size, shape, and position of the latent window can be flexibly set based on factors such as the original image resolution, latent space size, and motion estimation, and it is a key mechanism for achieving local perception and constraint control.

[0022] In existing technologies, joint training with high resolution and long duration faces pressure on GPU memory and computing power. Therefore, images or videos are often segmented into spatial sub-blocks for training. However, if the segmentation position does not match the latent space sampling stride, it can lead to misalignment between the pixel domain supervision region and the latent space generation region or generate boundary effects, thereby causing problems such as unstable gradient feedback, increased edge artifacts, and difficulty in training convergence. Moreover, the time dimension exacerbates these contradictions. Based on the above problems, this application provides a random window pixel alignment training method for video restoration to solve the aforementioned problems.

[0023] Reference Figure 1 and Figure 2 As shown in one embodiment of this application, a random window pixel alignment training method for video restoration includes: S100, acquires the original video and the ground truth video, as well as a video restoration model consisting of a generator sub-network and a decoder; Specifically, the original video clip is a low-resolution video clip, while the true video clip is a high-resolution video clip.

[0024] S200 processes the original video through a generative subnetwork and outputs a latent representation of the video restoration result in the latent space.

[0025] Specifically, the latent representation refers to the low-dimensional continuous representation space obtained after encoding the original video frame by the generator subnetwork. In this latent representation, the spatiotemporal structure of the video content is compressed and fully preserved, thus facilitating the modeling and optimization of downstream tasks (such as video restoration, video enhancement, etc.).

[0026] S300, select one or more random or semi-random local regions on the latent representation as latent window; S400: Based on the upsampling scale relationship between the latent space and the pixel domain, the latent window is mapped to the pixel domain coordinate system to determine the corresponding pixel decoding subdomain. S500 performs local pixel-level decoding processing on only the pixel decoding subdomain to obtain local predicted image blocks; S600 calculates reconstruction loss and perceptual loss based on the comparison between the local predicted image patch and the corresponding region of the ground truth video, and calculates local temporal difference constraints on the pixel decoding subdomain of adjacent frames in the temporal dimension to form a local joint loss. S700, based on local joint loss, obtains global joint loss and completes the training of video restoration model.

[0027] The embodiments described above in this application, through a combination of latent space clipping and pixel domain alignment, enable pixel-level supervision during training without relying on full-frame decoding. This significantly reduces peak memory usage and computational latency, improving training throughput in single-machine or multi-machine environments. For video sequences, the introduced local temporal constraints suppress flicker and promote smoothness across frames within the decoding subdomain. The design of expected-equivalent supervision, from local joint loss to global joint loss, ensures that local supervision is statistically representative of the full frame, thereby achieving convergence quality and visual performance comparable to full-frame training with lower resource consumption.

[0028] This method is adaptable to various generator network and decoder architectures, and is easy to integrate and maintain in a modular manner within existing engineering systems. The generator network includes diffusion generators based on UNet, DiT, cascaded structures, or those incorporating motion estimation modules, and the decoder includes decoders based on VAE, patch-wise mapping, or streaming convolution.

[0029] To make model training more accurate and standardized, in some specific embodiments of this application, step S100 includes: acquiring the original video, the ground truth video, and the video restoration model composed of a generator subnetwork and a decoder.

[0030] Specifically, after the original video is processed by the generative sub-network, the latent representation of the restoration result is output in the latent space; the ground truth video is used to calculate the loss function with local high-definition predicted image patches to train the generative video restoration model.

[0031] The embodiments described above in this application provide a data foundation and a model foundation for subsequent model training.

[0032] In some specific embodiments of this application, the generative subnetwork used refers to a neural network used to recover a high-quality image from an input video clip or its latent space representation, particularly a generator based on a diffusion model.

[0033] To reduce computational overhead, the training process employs a latent window. In some specific embodiments of this application, step S300 includes: selecting one or more random or semi-random local regions on the latent representation as latent windows.

[0034] Specifically, random selection refers to selecting a latent space block region corresponding to a certain time frame as the latent window in a completely random manner within the entire latent space representation. Semi-random selection refers to first screening out important regions, such as regions with drastic spatiotemporal changes or containing edge or foreground objects, and then randomly selecting local regions from these regions as the latent window. This selection method balances computational efficiency and the representativeness of the training signal.

[0035] The above embodiments of this application allow for flexible selection of the above selection method according to the needs of different video content.

[0036] Directly decoding and monitoring the entire image leads to a sharp increase in peak memory usage and creates unbearable computational overhead in the spatiotemporal dimension. To avoid this problem, this application employs an aligned mapping method. S400, based on the upsampling scale relationship between the latent space and the pixel domain, the latent window is mapped to the pixel domain coordinate system to determine the corresponding pixel decoding subdomain. This can be achieved through the following steps: S401, obtain the upsampling factor s from the latent space to the pixel domain; Specifically, the alignment mapping follows an integer scale relationship from the latent space to the pixel domain in the spatial dimension, which is usually determined by the latent space downsampling step; that is, the upsampling factor is equal to the scaling value corresponding to this integer scale.

[0037] S402, map the latent window [u,u+h)×[v,v+w) to the pixel sub-window [su,su+sh)×[sv,sv+sw), which is the pixel decoding sub-domain.

[0038] The above embodiments of this application, through the combination of "latent space clipping - pixel domain alignment", enable the training process to obtain pixel-level supervision without relying on full-frame decoding, thereby significantly reducing peak memory usage and computation latency, and improving training throughput in single-machine or multi-machine environments.

[0039] When performing restoration in the latent space, insufficient edge receptive field may occur due to the presence of stride, padding, and deconvolution kernel. Therefore, in some specific embodiments of this application, before performing the mapping from the latent space to the pixel domain, several pixels are added around the latent window to form an expanded latent window; mapping is then performed on the expanded latent window; after mapping, the region in the pixel domain corresponding to the expanded pixels is cropped to avoid boundary artifacts, thereby obtaining a restoration result consistent with the size of the latent window, which is the pixel decoding subdomain.

[0040] Specifically, the extended pixels usually refer to the filling area required in receptive field calculations such as convolution. The size can be 1 or 2 pixels, and the number should be consistent with the network's receptive field requirements to effectively avoid edge artifacts in the decoded image.

[0041] In the technical solutions of the above embodiments of this application, the latent window and the pixel decoding subdomain are strictly consistent in geometry, thus avoiding the problem of block misalignment; at the same time, the problem of insufficient edge receptive field is solved, the pixel gradient feedback is more stable, and the edge artifacts are significantly reduced.

[0042] In some specific embodiments of this application, time compression or grouping modeling with latent space in the time dimension can map the time index of the latent window to the corresponding frame or corresponding frame group in the pixel domain through the same alignment rules, so as to ensure the registration consistency of the time difference loss calculation.

[0043] Specifically, the same alignment rule means that when mapping the latent window along the time dimension to the pixel domain, all time segments should use a consistent upsampling factor and mapping coordinate calculation method.

[0044] For example: if the time index of a certain latent domain segment is t, and its spatial window is [u,u+h)×[v,v+w), then no matter which time point t is, its decoding area mapped to the pixel domain should be [su,su+sh)×[sv,sv+sw), that is, using a uniform s for magnification.

[0045] The alignment rules in the above embodiments of this application ensure that the pixel positions correspond precisely across time steps, thereby guaranteeing the consistency of coordinate alignment in time difference or inter-frame registration calculations.

[0046] If pixel loss is calculated frame-by-frame after complete decoding in the time dimension, memory usage and latency will accumulate approximately linearly with the sequence length, severely limiting training throughput. If supervision is applied only to a small number of frames, the coverage and representativeness of pixel constraints will be weakened. To address this issue, this application designs local joint loss and global joint loss. In the following specific embodiments of this application, the local joint loss is: ; L represents the local joint loss. Indicates the reconstruction loss. Indicates perceived loss. This indicates a time difference constraint.

[0047] Among them, the reconstruction loss adopts the mean absolute loss L1, the mean square error loss L2, or the Charbonnier loss; The perceptual loss uses a loss function based on the feature space distance metric; the temporal difference constraint uses the L2 distance between adjacent frames of the restored frame and the reference frame in the ground truth video as the loss.

[0048] For example: Mean squared error loss L2, the formula is: ; in The weighting coefficients for reconstructing the loss. This represents the restored frame generated from the clipped area of ​​the window. This represents a high-resolution reference frame for the corresponding region; The perceptual loss is extracted by the DISTS model, which measures the structural and textural similarity between the restored frame and the real frame at the perceptual level. Compared to traditional perceptual losses (such as VGG feature distance), DISTS is more structure-sensitive and better aligns with human perceptual judgments. Its formula is: ; in For distance perception computing networks, This is the weighting factor.

[0049] The temporal difference constraint requires that the motion changes between restored frames should closely match the motion changes between real frames to avoid obvious jumps in the restored video. Therefore, the L2 distance between adjacent frames of the generated restored frame and the high-resolution reference frame (the corresponding frame in the ground truth video) is used as the loss, and its specific formula is as follows: ; in and Let these represent the high-resolution reference image and the restored image of frame t, respectively. This is the weighting factor.

[0050] In the embodiments described above, for video sequences, the introduced local temporal constraints can suppress flicker and promote cross-frame smoothing within the decoding subdomain.

[0051] To ensure statistical consistency between the supervision and the full-frame pixel loss, in some specific embodiments of this application, S700, based on the local joint loss, the global joint loss is obtained to complete the training of the video restoration model, which can be achieved in any of the following two ways.

[0052] The first type, when the latent window uses random uniform sampling, the local loss is sampled from each region of the latent space with equal probability, and statistically it is not biased towards any region. Therefore, the local loss is consistent with the full-frame pixel loss in the sense of sampling expectation.

[0053] For example, assuming a latent space resolution of 480×270, and selecting a latent window of size 32×32, its area ratio b = (32×32) / (480×270) ≈ 0.008. If the local joint loss of this window is a = 0.08, then it is magnified by 1 / b ≈ 125 times to obtain an estimate of the global loss of 10.0, which is used for parameter updates.

[0054] Of course, when collecting data from multiple potential windows, the average of the multiple global loss estimates can be taken.

[0055] The second approach, when using semi-random, non-uniform sampling, is to apply sampling importance weights to the local loss. This method balances computational efficiency with the representativeness of the training signal.

[0056] For example: During image inpainting model training, the latent space feature map is 64×64, and the loss is calculated by semi-randomly selecting three 16×16 local regions: The first sample is taken at the edge of the damage (important area), with a sampling probability of 0.4 (higher than the probability of uniform sampling), and the local joint loss is 0.8. The second one is in the normal texture area, with a sampling probability of 0.3 and a local joint loss of 0.6; The third one is in the smooth region, with a sampling probability of 0.2 and a local joint loss of 0.2.

[0057] To prevent losses in high-probability areas from excessively impacting the overall loss, an importance weight is assigned to the loss in each area. Dividing the local loss by the sampling probability, the weighted contribution of the first region is 0.8 / 0.4=2, the second is 0.6 / 0.3=2, and the third is 0.2 / 0.2=1, totaling 5. Calculating a normalization factor, we get 1 / 0.4 + 1 / 0.3 + 1 / 0.2 ≈ 2.5 + 3.33 + 5 = 10.83. Finally, the global loss is 5 / 10.83 ≈ 0.46. This approach avoids calculating the full image loss (saving computational resources), focuses on important regions, and balances the influence of different regions through weights, achieving a balance between efficiency and training effectiveness.

[0058] In the embodiments described above, the local joint loss is normalized and scaled to obtain the global joint loss, making its expected value under the sampling distribution close to or equivalent to the full-frame supervision loss. Therefore, the entire training process does not require decoding the entire image in each iteration, effectively reducing peak memory usage and computational consumption during training. Simultaneously, it ensures that local supervision is statistically representative of the full image, thus achieving model convergence quality and visual performance comparable to full-frame training with lower resource consumption. Based on the same technical concept, other embodiments of this application also provide a random window pixel alignment training system 100 for video restoration, such as... Figure 3 As shown, system 100 includes: Preparation module 110: Acquire the original video and the ground truth video, as well as the video restoration model consisting of a generator subnetwork and a decoder; Latent space module 120: It uses a generative subnetwork to process the original video and outputs a latent representation of the video restoration result in the latent space; Latent window sampling module 130: Selects one or more random or semi-random local regions on the latent representation as latent windows; Alignment mapping module 140: Based on the upsampling scale relationship between the latent space and the pixel domain, the latent window is mapped to the pixel domain coordinate system to determine the corresponding pixel decoding subdomain; Local decoding module 150: For the pixel decoding subdomain, the decoder performs local pixel-level decoding processing to obtain local predicted image blocks; Local pixel supervision module 160: Based on the comparison between the local predicted image patch and the corresponding region of the ground truth video, it calculates the reconstruction loss and the perceptual loss, and calculates the local temporal difference constraint on the pixel decoding subdomain of adjacent frames in the temporal dimension to form a local joint loss; Loss aggregation and training update module 170: Processes the local joint loss to obtain the global joint loss and completes the training of the video restoration model.

[0059] The random window pixel alignment training system for video restoration described in the above embodiments of this application can be implemented by referring to the steps of the random window pixel alignment training method for video restoration described in the above embodiments, and will not be repeated here.

[0060] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0061] Based on the random window pixel alignment training method or training system for video restoration described in the above embodiments, other embodiments of this application also provide a video restoration method, including: Step 1: Obtain the video to be restored; Step 2: Input the video to be restored into the video restoration model trained using the random window pixel alignment training method or training system for video restoration described in any of the above embodiments to obtain the restored video.

[0062] To verify the universality and effectiveness of the training method proposed in this application in various video super-resolution scenarios, some specific embodiments were experimentally compared on public datasets UDM10, SPMCS, MVSR4x, and VideoLQ after the video restoration model was trained. This experiment covered several mainstream methods, including multi-frame enhancement methods (such as RealBasicVSR, RealViFormer), generative methods (such as Upscale-A-Video, MGLD-VSR, SeedVR), and the method of this application. Three types of metrics were used for evaluation: reconstruction metrics (PSNR, SSIM) to measure pixel restoration effect; perceptual metrics (LPIPS, DISTS, MUSIQ, CLIP-IQA, DOVER) to evaluate the consistency between the generated result and human perception, particularly suitable for subjective quality evaluation under real degraded input; and temporal consistency metrics (…). The ) is used to measure the degree of motion distortion, reflecting the model's performance in video coherence modeling. Table 1 uses arrows to indicate the direction of the indicators, where ↑ indicates a higher value is better, and ↓ indicates a lower value is better.

[0063]

[0064] As shown in Table 1, the embodiments of this application demonstrate stable performance on multiple public datasets, achieving leading results in most metrics. High scores on perception-related metrics such as LPIPS, MUSIQ, and CLIP-IQA indicate that the model has a certain advantage in subjective quality restoration; high scores on DOVER and other metrics reflecting cross-frame consistency and temporal stationarity also demonstrate this advantage. The proposed method also achieved good results in terms of metrics, demonstrating its ability to maintain video continuity. Furthermore, it maintains strong competitiveness in reconstruction metrics such as PSNR and SSIM, balancing clarity and consistency. Overall, the proposed method performs well across multiple metrics and data scenarios, exhibiting adaptability and practical application potential. Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-mentioned random window pixel alignment training method or video restoration method for video restoration, or to run the above-mentioned random window pixel alignment training system for video restoration.

[0065] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the program can be used to execute the above-described random window pixel alignment training method or video restoration method for video restoration, or to run the above-described random window pixel alignment training system for video restoration.

[0066] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0067] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0068] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0069] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0070] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0072] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A random window pixel alignment training method for video restoration, characterized in that, include: The original video and the ground truth video are obtained, along with a video restoration model consisting of a generator sub-network and a decoder. The original video is processed using the generative sub-network, and a latent representation of the video restoration result is output in the latent space; One or more random or semi-random local regions are selected on the latent representation as latent windows; Based on the upsampling scale relationship between the latent space and the pixel domain, the latent window is mapped to the pixel domain coordinate system to determine the corresponding pixel decoding subdomain; For the pixel decoding subdomain, a decoder is used to perform local pixel-level decoding processing to obtain local predicted image blocks; Based on the comparison between the local predicted image patch and the corresponding region of the ground truth video, the reconstruction loss and perceptual loss are calculated, and local temporal difference constraints are calculated on the pixel decoding subdomain of adjacent frames in the temporal dimension to form a local joint loss. The local joint loss is processed to obtain the global joint loss, and the training of the video restoration model is completed.

2. The random window pixel alignment training method for video restoration according to claim 1, characterized in that, The step of mapping the latent space window to the pixel domain coordinate system based on the upsampling scale relationship between the latent space and the pixel domain, and determining the corresponding pixel decoding subdomain, includes: Obtain the upsampling factor from the latent space to the pixel domain; Based on the upsampling factor, the latent window is mapped to the pixel domain coordinate system to obtain the corresponding pixel decoding subdomain.

3. The random window pixel alignment training method for video restoration according to claim 2, characterized in that, Before mapping, the perimeter of the latent window is expanded by several pixels to form an expanded latent window; Perform mapping on the expanded latent window; After mapping is completed, the region corresponding to the expanded pixel is cropped to obtain a restoration result with the same size as the latent window, which is used as the pixel decoding subdomain.

4. The random window pixel alignment training method for video restoration according to claim 2, characterized in that, During the mapping process from the latent window to the pixel domain along the time dimension, each time segment uses the same upsampling factor and the same method for calculating the mapping coordinates.

5. The random window pixel alignment training method for video restoration according to claim 1, characterized in that, The local joint loss is: L represents the local joint loss. Indicates the reconstruction loss. Indicates perceived loss. Indicates time difference constraints; The reconstruction loss is achieved by using the mean absolute loss L1, the mean square error loss L2, or the Charbonnier loss. The perceptual loss employs a loss function based on a feature space distance metric; The temporal difference constraint uses the L2 distance between adjacent frames of the restored frame and the reference frame in the ground truth video as the loss.

6. The random window pixel alignment training method for video restoration according to claim 1, characterized in that, The process of processing the local joint loss to obtain the global joint loss includes any of the following cases: When the latent window uses random uniform sampling, the local joint loss is scaled according to the ratio of the latent window area to the full area of ​​the latent representation to obtain the global joint loss; When the latent window uses semi-random non-uniform sampling, a sampling importance weight is applied to the local joint loss to obtain the global joint loss.

7. A random window pixel alignment training system for video restoration, characterized in that, include: Preparation module: Acquires the original video and ground truth video, as well as a video restoration model consisting of a generator sub-network and a decoder; Latent space module: The generator subnetwork is used to process the original video, and the latent representation of the video restoration result is output in the latent space; Latent window sampling module: Selects one or more random or semi-random local regions on the latent representation as latent windows; Alignment mapping module: Based on the upsampling scale relationship between the latent space and the pixel domain, the latent window is mapped to the pixel domain coordinate system to determine the corresponding pixel decoding subdomain; Local decoding module: For the pixel decoding subdomain, a decoder is used to perform local pixel-level decoding processing to obtain local predicted image blocks; Local pixel supervision module: Based on the comparison between the local predicted image patch and the corresponding region of the ground truth video, it calculates the reconstruction loss and the perceptual loss, and calculates the local temporal difference constraint on the pixel decoding subdomain of adjacent frames in the temporal dimension to form a local joint loss; Loss aggregation and training update module: Processes the local joint loss to obtain the global joint loss and completes the training of the video restoration model.

8. A video restoration method, characterized in that, include: Obtain the video to be restored; The video to be restored is input into the video restoration model trained by the random window pixel alignment training method for video restoration as described in any one of claims 1-6 or the training system as described in claim 7, to obtain the restored video.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-6 and 8, or to run the system of claim 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the methods described in claims 1-6 and 8, or to run the system described in claim 7.

Images