Video noise AI generation processing method

Abstract

The invention relates to the field of video processing, and aims to provide a video noise AI (Artificial Intelligence) generation processing method which can be used for improving the denoising performance, reserving details, reducing the processing cost and improving the processing efficiency. According to the technical scheme, the method comprises the steps that S1, data preprocessing is conducted, a video file is decoded into image frames, the format of the image frames is converted, a data set is obtained after standardization processing and data enhancement are conducted, and the data set is divided into a training set, a verification set and a test set; s2, establishing a model: establishing a noise detection model, a generative adversarial network and a detail enhancement model; s3, model fusion: combining the noise detection model, the generative adversarial network, the detail enhancement model and the self-attention module into a joint model; training, verifying and testing the joint model; and S4, video noisy point generation processing is carried out, the collected video is subjected to data preprocessing, the joint model is input to obtain a noiseless image frame sequence, and a noiseless video is output after merging.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a method for AI-generated processing of video noise. Background Art

[0002] With the rapid development and widespread application of video technology, the improvement of video quality and the removal of noise have become important research directions in the field of video processing. Traditional video denoising and generation technologies have solved the problems of video noise and quality to a certain extent, mainly relying on basic image processing algorithms and a small number of machine learning methods, but there are still many shortcomings, such as limited noise removal effect, insufficient detail retention, slow processing speed, etc. The details are as follows:

[0003] 1. Insufficient denoising performance: Traditional video denoising technologies are mostly based on basic filtering algorithms, such as median filtering, mean filtering, and Gaussian filtering. These methods work well when dealing with low-frequency noise, but when faced with complex random noise, sensor noise, and compression artifacts, the denoising effect is obviously insufficient, which can easily lead to blurred video images and serious loss of details.

[0004] 2. Insufficient detail preservation: Existing video denoising technologies often lose a lot of detail information while removing noise, resulting in distorted video images that are not natural and realistic enough. This is because traditional methods find it difficult to balance the relationship between denoising and detail preservation, and often sacrifice image details during the denoising process.

[0005] 3. Slow processing speed: Traditional video denoising methods have high computational complexity and slow processing speed when processing high-resolution videos, making it difficult to meet the needs of real-time processing. This is a major limitation for video surveillance and real-time video processing applications that require fast response.

[0006] 4. Lack of adaptability: Existing technologies are mostly targeted at specific types of videos and noise, lacking versatility and adaptability. When faced with different video formats, noise patterns, and application scenarios, traditional methods often require complex parameter adjustments, making it difficult to achieve one-click adaptation. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the above-mentioned background technology and provide a video noise AI generative processing method, which should be able to improve denoising performance and detail retention, and can reduce processing costs and improve processing efficiency.

[0008] The technical solution of the present invention is:

[0009] The video noise AI processing method includes the following steps:

[0010] S1: Data preprocessing

[0011] Decode the video file into image frames, convert the image frame format, perform standardization and data enhancement to obtain a data set, and divide the data set into a training set, a validation set, and a test set;

[0012] S2: Building the model

[0013] Establish noise detection model, generative adversarial network, and detail enhancement model;

[0014] S3: Model Fusion

[0015] Combine the noise detection model, generative adversarial network, detail enhancement model, and self-attention module into a joint model; train, verify, and test the joint model;

[0016] S4: Video noise generation processing

[0017] The video is collected for data preprocessing and input into the joint model to obtain a noise-free image frame sequence, which is then merged to output a noise-free video.

[0018] The step S1 comprises:

[0019] S1-1: Decoding

[0020] Decode the video file using the C++ native codec and convert it into a sequence of image frames;

[0021] S1-2: Format Conversion

[0022] Convert image frames to tensor format;

[0023] S1-3: Standardization

[0024] Normalize the pixel values ​​of the image frame:

[0025] S1-4: Data Augmentation

[0026] Randomly crop, randomly flip, randomly rotate, Gaussian noise, motion blur, underexposure / overexposure, and color jitter on image frames;

[0027] S1-5: Dividing the Dataset

[0028] The image frame data is divided into training set, validation set and test set.

[0029] In the noise detection model,

[0030] Using multi-classification cross entropy loss function:

[0031]

[0032] Among them, y i is the true label, is the predicted probability;

[0033] The optimization algorithm is:

[0034] m t =β1m t-1 +(1-β1)g t

[0035]

[0036]

[0037] Among them, α is the learning rate, β1 and β2 are momentum tops, and g t is the gradient.

[0038] In the generative adversarial network,

[0039] The generator loss function is:

[0040]

[0041] The discriminator loss function is:

[0042]

[0043] in, is the gradient penalty top.

[0044] The detail enhancement model is an improved U-NET model:

[0045] Use Swin Transformer encoder to extract image features;

[0046] Add the attention mechanism module and feature pyramid module to the fusion splicing layer of the decoder;

[0047] The loss function includes pixel loss function and perceptual loss function;

[0048] The pixel loss function is:

[0049]

[0050] The perceptual loss function is:

[0051]

[0052] The step S3 comprises:

[0053] S3-1: Fine-tuning

[0054] Use the dataset to fine-tune the noise detection model, generative adversarial network, and detail enhancement model respectively;

[0055] S3-2: End-to-end joint training

[0056] S3-2-1: Building a joint model

[0057] The noise detection model, the generative adversarial network, the detail enhancement model, and the self-attention module are sequentially connected to obtain a joint model;

[0058] S3-2-2: Define the joint loss function

[0059] The joint loss function is:

[0060]

[0061] in, is the loss of the noise detection model, is the loss of the generative adversarial network, is the loss of the detail enhancement model, α, β, γ are weight coefficients;

[0062] S3-2-3: Train the joint model and optimize the parameters of the joint model.

[0063] The step S4 comprises:

[0064] S4-1: Preprocessing Video

[0065] Capturing a video, and processing the video through steps S1-1 and S1-2 to obtain preprocessed data;

[0066] S4-2: Noise processing

[0067] The preprocessed data is input into the joint model to obtain a noise-free image frame sequence;

[0068] S4-3: Post-processing

[0069] S4-3-1: Format Conversion

[0070] Generate a noise-free video from a noise-free image frame sequence;

[0071] S4-3-2: Parameter settings

[0072] Set the output video resolution, bit rate, and frame rate;

[0073] S4-3-3: Post-processing

[0074] Automatically adjust the exposure and stabilize the video to output noise-free video.

[0075] The beneficial effects of the present invention are:

[0076] 1. Excellent denoising performance: This method uses advanced machine learning technology and image processing algorithms to effectively remove various types of noise, including random noise, sensor noise, and compression artifacts, thereby significantly improving the clarity and quality of the video;

[0077] 2. Enhanced detail preservation: This method carefully preserves the fine details and textures of the video during the denoising process, ensuring that the generated video maintains a natural and realistic appearance. Through a comprehensive learning mechanism, combining a variety of video data including clean and noisy examples, a balance of details and denoising effects is achieved;

[0078] 3. Realistic frame generation: Using advanced generative AI technology, this method can generate realistic and consistent video frames, effectively filling in the missing or damaged parts of the original video, thereby improving the overall integrity and coherence of the video content;

[0079] 4. Wide range of applications: This method has wide adaptability and can be seamlessly applied to various video formats, noise patterns and application scenarios, including video restoration and enhancement, video surveillance and security, video editing and production, and video compression and transmission.

[0080] 5. Efficiency and adaptability: This method can quickly adapt to different video processing requirements by adopting universal configuration and flexible learning mechanism, greatly reducing the time and complexity of manual adjustment, and improving processing efficiency and accuracy;

[0081] In summary, the present invention significantly improves the performance of video denoising and generation by utilizing advanced artificial intelligence and machine learning technologies, overcomes many shortcomings of the prior art, and provides an efficient, accurate and high-quality solution for the field of video processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a flow chart of the present invention.

[0083] Figure 2 is a schematic diagram of the joint model of the present invention.

[0084] Figure 3 Schematic diagram of the detail enhancement model of the present invention.

[0085] Figure 4 is the noisy image tensor that is input to the detail enhancement model.

[0086] Figure 5 It is the noise-free image tensor after detail enhancement output by the detail enhancement model. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0088] The video noise AI processing method includes the following steps:

[0089] S1: Data preprocessing

[0090] Decode the video file into image frames, convert the image frame format, standardize and enhance the pixel values ​​of the image frames, and then divide them into training set, validation set and test set;

[0091] S1-1: Decoding

[0092] Use C++ native codec to decode the video file and convert it into a sequence of image frames. Use the OpenCV library to open the video file and read the data frame by frame. Specifically include:

[0093] S1-1-1: Initialize the video capture object and open the input video file;

[0094] S1-1-2: Use a loop structure to read the video frame by frame until the entire video file is read;

[0095] S1-1-3: Each frame of the image is saved in a container (database) for subsequent processing.

[0096] S1-2: Format Conversion

[0097] Convert the decoded image frame to a format acceptable to the joint model, such as a tensor format (torch.Tensor).

[0098] S1-2-1: Convert the image frame read by OpenCV from BGR format to RGB format;

[0099] S1-2-2: Convert the image data type to floating point type and perform normalization (scale to the [0,1] interval).

[0100] S1-2-3: Adjust the order of image frame dimensions to adapt to the input requirements of the joint model (e.g., from HWC to CHW).

[0101] S1-3: Standardization

[0102] Normalize the image pixel values ​​to meet the input requirements of the pre-trained model. Usually, the pixel values ​​are scaled to the [0,1] interval and the mean and standard deviation are normalized.

[0103] S1-3-1: Calculate the mean and standard deviation, or use predefined mean and standard deviation;

[0104] S1-3-2: Normalize each channel of each image frame by subtracting the mean and dividing by the standard deviation.

[0105] S1-4: Data Augmentation

[0106] In order to improve the generalization ability of the model, the image frames are randomly cropped, randomly flipped, randomly rotated, Gaussian noise, motion blurred, underexposed / overexposed, and color jittered to further enhance the robustness of the model.

[0107] Random Crop: Crop fixed-size regions from random locations of the image.

[0108] Random Flip: Flip the image horizontally or vertically with a certain probability.

[0109] Random Rotation: Randomly rotate the image by a certain angle and fill the blank area at the edges.

[0110] Gaussian Noise: Adds random Gaussian noise to the image.

[0111] Motion Blur: Simulates image blur caused by camera motion.

[0112] Under / Over Exposure: Adjusts the brightness of the image to simulate under- or over-exposure.

[0113] Color jittering: Randomly changes the color properties of an image, such as brightness, contrast, and saturation.

[0114] Each image frame randomly selects one or more of the above methods for processing to improve the robustness of the model to different noises and image transformations. Each image frame can generate at least one data-enhanced image frame, increasing the number of image frames.

[0115] S1-5: Dividing the Dataset

[0116] The image frame data is divided into training set, validation set and test set to facilitate the training, validation and testing of the model.

[0117] First, the order of the image frames is randomly disrupted to ensure that the data is evenly distributed, and then the data is divided according to the predetermined ratio: 80% as the training set, 10% as the validation set, and 10% as the test set.

[0118] S2: Building the model

[0119] Establish noise detection model, generative adversarial network, and detail enhancement model;

[0120] (1) Noise Detection Model

[0121] The noise detection model is a deep convolutional neural network (CNN) that is used to detect and classify noise types. The noise detection model can use the ResNet model or the Inception model. Transformer and ViT (Vision Transformer) can also be used to improve model performance.

[0122] The noise detection model includes input layer, feature extraction layer and classification layer.

[0123] Input layer that accepts the normalized image tensor.

[0124] Feature extraction layer, using convolutional layer or Transformer encoder to extract image features.

[0125] The classification layer maps the features to the classification space and outputs the probability distribution of the noise type.

[0126] In the noise detection model:

[0127] The input is a normalized image tensor with shape batchsize, channels, height, width.

[0128] The output is a one-hot encoding vector of noise type, including:

[0129] Gaussian noise [1,0,0,0,0,0,0,0], Impulse noise [0,1,0,0,0,0,0,0], Poisson noise [0,0,1,0,0,0,0,0], Salt and pepper noise [0,0,0,1,0,0,0,0], Uniform noise [0,0,0,0,1,0,0,0], Periodic noise [0,0,0,0,0,1,0,0], Random noise [0,0,0,0,0,0,1,0], Speckle noise [0,0,0,0,0,0,0,1].

[0130] Loss function: Multi-classification cross entropy loss function is used to measure the difference between the model output and the true label:

[0131]

[0132] Among them, y i is the true label, is the predicted probability.

[0133] Optimization algorithm: Use Adam optimizer for model training, set the initial learning rate to 1e-4, and hyperparameters such as learning rate decay need to be adjusted through the validation set; the formula is:

[0134] m t =β1m t-1 +(1-β1)gt

[0135]

[0136]

[0137] Among them, α is the learning rate, β1 and β2 are momentum tops, and g t is the gradient; β1=0.9, β2=0.999, β1=0.9,∈=1e-8.

[0138] (2) Generative Adversarial Network (GAN)

[0139] The generative adversarial network adopts the encoder-decoder CNN structure. The generative adversarial network can adopt the U-Net model or the Pix2Pix model. It can also adopt the ViT model or the Transformer-based generative model.

[0140] The generative adversarial network consists of an encoder and a decoder.

[0141] The encoder, including multiple convolutional layers, gradually reduces the image spatial dimension and increases the number of feature channels, or uses the Transformer encoder to extract features.

[0142] The decoder restores the image spatial dimensions through multiple deconvolution layers and is connected to the corresponding layers of the encoder through skip connections.

[0143] In Generative Adversarial Networks:

[0144] The input to the generator is the noise image tensor and the feature vector of the noise image (obtained from the noise detection model).

[0145] The output of the generator is the generated noise-free image tensor.

[0146] The discriminator is a fully convolutional discriminator based on PatchGAN, which scores the authenticity of the image blocks. The Transformer discriminator can also be used to improve the discrimination ability. The structure includes: multiple convolutional layers or Transformer blocks, which gradually reduce the image space dimension and increase the number of feature channels; the last layer outputs the authenticity score of the image block.

[0147] The input of the discriminator is the noise-free image output by the generator and the original noisy image tensor.

[0148] The discriminator outputs a vector of authenticity scores for each image patch.

[0149] Generator loss function: Minimize the loss of generated images judged as fake, using the adversarial loss function, the formula is

[0150]

[0151] Discriminator loss function:

[0152] Maximize the loss of distinguishing real / fake images, and introduce optimization methods such as WGAN-GP to stabilize GAN. The formula is

[0153]

[0154] in, It is the gradient penalty top, which is used to improve training stability.

[0155] Optimization algorithm: Use Adam optimizer to optimize the generator and discriminator, and set the initial learning rate to 1e-4, β1=0.9, β2=0.999, β1=0.9,∈=1e-8.

[0156] The optimization methods include: WGAN-GP (Wasserstein GAN with Gradient Penalty), Spectral Normalization, R1 / R2 Regularization, Hinge Loss, and Self-Attention Mechanism. They can be used alone or in combination to maximize the stability of GAN training and the quality of generated images. In practical applications, suitable optimization methods can be selected and debugged according to the characteristics of specific problems and data sets.

[0157] (3) Detail Enhancement Model

[0158] The detail enhancement model is an improved U-NET model, including: encoder, decoder, attention mechanism module, and feature pyramid module.

[0159] Encoder: Use Swin Transformer to extract image features and replace the convolution in the original encoder;

[0160] Decoder: Use multiple upsampling convolutional layers to maintain the original structure and improve detail recovery capabilities; add attention mechanism modules and feature pyramid modules to the fusion splicing layer.

[0161] Swin Transformer performs feature extraction and downsampling in a layered manner. Swin Transformer includes: (1) layered processing, Swin Transformer performs feature extraction and downsampling by splitting and merging windows layer by layer; (2) shift window mechanism, windows are shifted between different layers to ensure feature fusion across windows; (3) patch merging, at the end of each stage, adjacent patches are merged to reduce the resolution of the feature map, which is similar to the downsampling operation in traditional CNN, but does not use an explicit pooling layer.

[0162] The attention mechanism module performs weighted processing on important features, enhancing the model's attention to key information, improving segmentation accuracy, and enhancing feature selectivity.

[0163] The feature pyramid module is used for multi-scale feature fusion, fusing decoder feature maps from different scales to provide more comprehensive contextual information and improve detail processing capabilities.

[0164] In the detail enhancement model:

[0165] The input is a noisy image tensor, such as Figure 4 ; The output is a noise-free image tensor with enhanced details, such as Figure 5 ;

[0166] Loss function, combining pixel loss function (L1 or Charbonnier) with perceptual loss function (feature reconstruction loss based on VGG network), adding perceptual loss to improve the quality and details of generated images;

[0167] The pixel loss function is:

[0168]

[0169] The perceptual loss function is:

[0170]

[0171] Optimization algorithm: Adam optimizer is used to optimize the detail enhancement model, and the initial learning rate is set to 1e-4, β1=0.9, β2=0.999, β1=0.9,∈=1e-8.

[0172] like Figure 3 As shown, the structure of the detail enhancement model is as follows:

[0173] The encoder includes a first hierarchical visual transformer, a second hierarchical visual transformer, a third hierarchical visual transformer, and a fourth hierarchical visual transformer which are connected in sequence.

[0174] The decoder includes a first upsampling convolution module, a first fusion splicing layer, a first convolution layer, a second upsampling convolution module, a second fusion splicing layer, a second convolution layer, a third upsampling convolution module, a third fusion splicing layer, a third convolution layer, a fourth upsampling convolution module, a fourth fusion splicing layer, and a fourth convolution layer, which are connected in sequence.

[0175] The first layered visual transformer is directly connected to the fourth fusion splicing layer through a jump connection, the second layered visual transformer is directly connected to the third fusion splicing layer through a jump connection, the third layered visual transformer is directly connected to the second fusion splicing layer through a jump connection, the fourth layered visual transformer is directly connected to the first fusion splicing layer through a jump connection, and the fourth layered visual transformer is connected to the first upsampling convolution module through a bottleneck layer.

[0176] The first fusion splicing layer, the second fusion splicing layer, the third fusion splicing layer, and the fourth fusion splicing layer all include attention mechanism modules and feature pyramid modules. These splicing layers splice the feature maps of the corresponding positions of the encoder and decoder, fusing high-level semantics and low-level detail information.

[0177] S3: Model fusion and fine-tuning

[0178] S3-1: Fine-tuning

[0179] The dataset is used to fine-tune the noise detection model, generative adversarial network, and detail enhancement model respectively.

[0180] Fine-tuning refers to performing a small amount of training on the basis of a pre-trained model to adapt to a specific target data set or a specific task and improve the model performance in a specific scenario. During the fine-tuning process, the parameters of some layers are locked and only specific layers are trained, thereby reducing the risk of overfitting and speeding up training.

[0181] When fine-tuning, train each model separately on the dataset, adjust the model parameters, and use the validation set to monitor the model performance to prevent overfitting.

[0182] S3-2: End-to-end joint training

[0183] End-to-end joint training is to train all models (noise detection, generative adversarial network, detail enhancement model) at the same time, and optimize them through an overall loss function, which can better coordinate the interactions between various models and optimize the global performance rather than just the performance of each individual module.

[0184] The specific strategy of end-to-end joint training is as follows:

[0185] S3-2-1: Building a joint model

[0186] The noise detection model, generative adversarial network, detail enhancement model, and self-attention module are connected in sequence to obtain a joint model.

[0187] In the joint model: each model is cascaded sequentially, the noise detection output is used as the input of the generative adversarial network, the output of the generative adversarial network is used as the input of the detail enhancement model, and the output of the detail enhancement model is used as the input of the self-attention module.

[0188] The noise detection model is used to obtain the noise type. The noisy image is input into the generative adversarial network to obtain a noise-free image. The noise-free image output by the generative adversarial network is input into the detail enhancement model to improve the image quality.

[0189] The self-attention module is added after the detail enhancement model to perform weighted fusion on the outputs of the generative adversarial network and the detail enhancement model, including: calculating the feature representation of the outputs of the two models, using the self-attention mechanism to calculate the feature weights, and performing weighted fusion on the output features, comprehensively considering the contributions of the two models to further improve the image quality.

[0190] S3-2-2: Define a joint loss function to combine the loss functions of each model.

[0191] The joint loss function is:

[0192]

[0193] in, is the loss of the noise detection model, is the loss of the generative adversarial network, is the loss of the detail enhancement model, α, β, γ are weight coefficients;

[0194] S3-2-3: Train the joint model and optimize the parameters of the joint model (loss function of each module, self-attention module).

[0195] Use the same optimizer and set appropriate learning rate and decay strategy.

[0196] Initial training: Train the joint model starting from random initialization.

[0197] Training is done in stages: first train the noise detection model and fix its parameters, then train the generative adversarial network and detail enhancement model, and finally jointly fine-tune the entire joint model.

[0198] Dynamic weight adjustment: During the training process, the weight coefficients in the joint loss function are dynamically adjusted according to the performance of the validation set to balance the contribution of each module.

[0199] S4: Noise generation processing

[0200] The video is collected for data processing and input into the joint model to obtain a noise-free image frame sequence, which is then merged to output a noise-free video.

[0201] S4-1: Preprocessing Video

[0202] Capture a video, decode the video first (step S1-1) to obtain an image frame sequence, then perform format conversion (step S1-2), convert the image frame sequence format (tensor format), and then input the joint model for processing to obtain preprocessed data;

[0203] S4-2: Noise processing

[0204] The preprocessed data is input into the joint model to obtain a noise-free image frame sequence;

[0205] S4-3: Post-processing

[0206] S4-3-1: Format Conversion

[0207] Encode the generated noise-free image sequence frames into a compressed video format (such as H.264 / H.265) using OpenCV for video encoding.

[0208] Initialize the video writing object, set the format and parameters of the output video; write the processed image frame by frame to generate the output video file.

[0209] S4-3-2: Parameter settings

[0210] Set the output video parameters (resolution, bit rate, frame rate) according to user needs.

[0211] Set the resolution, bit rate, and frame rate of the video writing object; then adjust the size and quality of the image frame as needed.

[0212] S4-3-3: Post-processing

[0213] Automatically adjust the exposure and stabilize the video to output noise-free video, further improving the video quality.

[0214] Through the video stabilization algorithm (existing algorithm), the impact of camera shaking on video quality is reduced.

[0215] Through the automatic exposure adjustment algorithm (existing algorithm), the video brightness is ensured to be moderate, improving the video viewing experience.

[0216] Embodiment 1 (Video Restoration and Enhancement)

[0217] This method is used to repair and enhance old, damaged or low-quality videos, including:

[0218] 1) Collect old and low-quality video data;

[0219] 2) Perform preprocessing and noise feature extraction; train a joint model and perform video restoration processing to make the video clearer and more natural by removing noise and generating missing parts, thereby restoring the original visual effect of the video.

[0220] 3) Output the restored video with multiple resolution options.

[0221] Example 2 (Video Surveillance and Security)

[0222] This method can be used to effectively remove noise and generate clear video images, improving the accuracy of monitoring and event analysis, including:

[0223] 1) Collect surveillance video data, including videos in high-noise and low-light environments;

[0224] 2) Perform preprocessing and noise detection; train a joint model to perform real-time denoising on surveillance videos;

[0225] 3) Output the processed video to improve the monitoring image quality.

[0226] Embodiment 3 (Video Editing and Production)

[0227] This method is used in video editing and production workflows to remove noise from videos and generate creative effects to improve video quality, including:

[0228] 1) Collect video materials that need to be edited and produced;

[0229] 2) Perform preprocessing, noise feature extraction, and detail-preserving annotation; train a joint model and perform video processing to generate natural and realistic effects;

[0230] 3) Output the processed video for post-processing with creative effects.

[0231] Embodiment 4 (Video Compression and Transmission)

[0232] Integrate this method into video compression and transmission systems to reduce noise and improve overall video quality, including:

[0233] 1) Collect video data for compression and transmission, especially in low-bandwidth environments;

[0234] 2) Perform preprocessing and noise feature extraction; train the joint model and perform video processing before compression;

[0235] 3) Output the processed video, compress it and transmit it to reduce bandwidth usage and improve transmission quality.

[0236] Embodiment 5 (Hardware Acceleration Optimization)

[0237] Hardware acceleration (such as NVIDIA NGX) is used to improve the processing efficiency of this method and enhance its real-time and high efficiency, including: using the CUDA library to accelerate the model and image processing process; using the GPU for calculations during model initialization and processing; using model quantization technology to reduce the model size and improve computing efficiency; and deploying on a dedicated AI chip to achieve efficient edge computing.

[0238] The preferred embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

Claims

1. The video noise AI generative processing method includes the following steps: S1: Data preprocessing Decode the video file into image frames, convert the image frame format, perform standardization and data enhancement to obtain a data set, and divide the data set into a training set, a validation set, and a test set; S2: Building the model Establish noise detection model, generative adversarial network, and detail enhancement model; S3: Model Fusion Combine the noise detection model, generative adversarial network, detail enhancement model, and self-attention module into a joint model; train, verify, and test the joint model; S4: Video noise generation processing The video is collected for data preprocessing and input into the joint model to obtain a noise-free image frame sequence, which is then merged to output a noise-free video.

2. The video noise AI generation processing method according to claim 1, characterized in that: The step S1 comprises: S1-1: Decoding Decode the video file using the C++ native codec and convert it into a sequence of image frames; S1-2: Format Conversion Convert image frames to tensor format; S1-3: Standardization Normalize the pixel values ​​of the image frame: S1-4: Data Augmentation Randomly crop, randomly flip, randomly rotate, Gaussian noise, motion blur, underexposure / overexposure, and color jitter on image frames; S1-5: Dividing the Dataset The image frame data is divided into training set, validation set and test set.

3. The video noise AI generation processing method according to claim 1, characterized in that: In the noise detection model, Using multi-classification cross entropy loss function: Among them, y i is the true label, is the predicted probability; The optimization algorithm is: m t =β1m t-1 +(1-β1)g t Among them, α is the learning rate, β1 and β2 are momentum tops, and g t is the gradient.

4. The video noise AI generation processing method according to claim 1, characterized in that: In the generative adversarial network, The generator loss function is: The discriminator loss function is: in, is the gradient penalty top.

5. The video noise AI generation processing method according to claim 1, characterized in that: The detail enhancement model is an improved U-NET model: Use Swin Transformer encoder to extract image features; Add the attention mechanism module and feature pyramid module to the fusion splicing layer of the decoder; The loss function includes pixel loss function and perceptual loss function; The pixel loss function is: The perceptual loss function is:

6. The video noise AI generation processing method according to claim 1, characterized in that: The step S3 comprises: S3-1: Fine-tuning Use the dataset to fine-tune the noise detection model, generative adversarial network, and detail enhancement model respectively; S3-2: End-to-end joint training S3-2-1: Building a joint model The noise detection model, the generative adversarial network, the detail enhancement model, and the self-attention module are sequentially connected to obtain a joint model; S3-2-2: Define the joint loss function The joint loss function is: in, is the loss of the noise detection model, is the loss of the generative adversarial network, is the loss of the detail enhancement model, α, β, γ are weight coefficients; S3-2-3: Train the joint model and optimize the parameters of the joint model.

7. The video noise AI generation processing method according to claim 1, characterized in that: The step S4 comprises: S4-1: Preprocessing Video Capturing a video, and performing step S1-1 and step S1-2 on the video to obtain preprocessed data; S4-2: Noise processing The preprocessed data is input into the joint model to obtain a noise-free image frame sequence; S4-3: Post-processing S4-3-1: Format Conversion Generate a noise-free video from a noise-free image frame sequence; S4-3-2: Parameter settings Set the output video resolution, bit rate, and frame rate; S4-3-3: Post-processing Automatically adjust the exposure and stabilize the video to output noise-free video.

Images