Real-time enhancement method for rebroadcast picture

Abstract

The invention provides a method for enhancing a rebroadcast picture in real time. The method comprises the following steps: acquiring an original video and a target high-quality video as training samples; inputting the original video into the generation model for enhancement processing to obtain a corresponding simulated high-quality video, comparing the simulated high-quality video with a target high-quality video by using a discrimination model, and optimizing the generator according to a comparison result; in the optimization process, a generator is adjusted in combination with an L2 loss function based on pixel difference and a perception loss function, and supervised learning is performed by using a high-resolution video and a low-resolution version of the high-resolution video; and placing the generation model into a director station system, receiving a real-time video stream by the generation model, performing enhancement processing on each frame, and outputting a high-quality rebroadcasting picture. The method has the beneficial effects that the real-time video stream is processed through the neural network, the picture definition and the visual effect are improved, and the picture quality output by the director is guaranteed.

Description

Technical Field

[0001] The invention belongs to the technical field of broadcast directing, and in particular relates to a real-time enhancement method for broadcast pictures. Background Art

[0002] Video is a mainstream entertainment medium widely accepted by the public. At present, some live events are not fully and fully watched by viewers in different regions due to time zone issues. Therefore, event broadcasting has become the main compensation measure. As a secondary broadcasting solution, broadcasting often has the problem of pixel and image quality degradation due to transcription. Therefore, it is necessary to enhance the image of the broadcast video. However, the existing technology has the following defects: Traditional video enhancement technologies usually rely on fixed image processing algorithms, such as sharpening filters and contrast adjustment, which are difficult to adapt to the complexity of different video content and have limited enhancement effects. Many traditional image enhancement algorithms have bottlenecks in processing speed and cannot meet the needs of real-time broadcasting, which may cause delays and freezes. Existing technologies often require manual parameter adjustment and optimization, which is not only time-consuming and labor-intensive, but also easily leads to inconsistent enhancement effects. In low-resolution videos, detail information is often lost. Traditional methods are insufficient in detail recovery and it is difficult to significantly improve video clarity. Some enhancement technologies may introduce artifacts or noise while improving clarity, resulting in unnatural visual effects and affecting the audience's viewing experience.

[0003] In summary, traditional video enhancement technology has shortcomings in terms of quality improvement, real-time performance, detail restoration, visual naturalness and adaptability. Summary of the invention

[0004] In view of this, the present invention aims to provide a real-time enhancement method for broadcast images, in order to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows: A method for real-time enhancement of a broadcast picture, the method comprising: Get the original video and the target high-quality video; The original video is input into the generative model for enhancement processing to obtain the corresponding simulated high-quality video. The discriminant model is used to compare the simulated high-quality video with the target high-quality video, and the generator is optimized according to the comparison results. In the optimization process, the generator is adjusted by combining the L2 loss function based on pixel difference and the perceptual loss function, and supervised learning is performed using the high-resolution video and its low-resolution version; The generative model is placed in the director's console system. The generative model receives the real-time video stream and enhances each frame to output high-quality broadcast images.

[0006] Furthermore, the original video and the target high-quality video are classified by a classification strategy, and the classification strategy specifically includes: Calculate the color histogram of the video frame and analyze the color distribution and saturation to determine whether the video is colorful or dull. Use filters to extract texture features of the video frame to determine whether the video is an animation or a real-life video. Calculate the optical flow to capture motion information in the video and determine whether the video is an action scene or a static scene. Calculate the average brightness and contrast to determine whether the video is a night scene or has sufficient light.

[0007] Furthermore, after classification, multiple generators are constructed according to the classification results, each generator corresponds to a type, and the corresponding generator is used to enhance the corresponding type of video.

[0008] Furthermore, the model designs of the generative model and the discriminative model are respectively: Generative model, using convolutional neural network architecture, combined with upsampling layers to improve resolution, and using residual blocks to improve the performance of the generator; The discriminant model uses a convolutional neural network architecture to extract features layer by layer, outputs true and false discrimination through a fully connected layer, and uses a PatchGAN architecture to divide the input into small blocks and discriminate them separately.

[0009] Furthermore, the training process of the generative model and the discriminative model includes: Initialize the weights of the generator and discriminator; Use the generator to generate simulated high-quality videos, use the discriminator to calculate the discriminant loss of the target high-quality video and the simulated high-quality video, update the weight of the discriminator and calculate the generation loss of the generator, and update the weight of the generator; The calculation process of discriminative loss and generative loss is repeated until the loss converges.

[0010] Furthermore, the discriminant loss is used to maximize the recognition of real samples and minimize the misjudgment of generated samples. The specific formula is: ; Where x is a real sample from the real data distribution pdata; z is a random noise vector from the noise distribution p z ; G(z) is the sample generated by the generator; D(x) is the output of the discriminator for the real sample, which is close to 1; D(G(z)) is the output of the discriminator for the generated sample, which is close to 0.

[0011] Furthermore, the generation loss is used to maximize the discriminator's misjudgment of the generated samples, and its specific formula is: .

[0012] Furthermore, the process of adjusting the generator by combining the L2 loss function based on pixel difference with the perceptual loss function includes: Input low-quality video z to generator G, generate high-quality video G(z), and calculate L2 loss L L2 , extract the features of the generated video and the target video, and calculate the perceptual loss L perceptual , calculate the total content loss L content , calculate the gradient of the loss, and use the Adam algorithm to update the parameters of the generator to minimize the total content loss.

[0013] Compared with the prior art, the real-time enhancement method of broadcast images described in the present invention has the following beneficial effects: Through enhanced processing of the generative model, the clarity and details of the video can be significantly improved, making the broadcast image closer to the target high-quality video effect; This method can process video streams in real time, ensuring that there will be no delay or freeze during the broadcast process, thus meeting the timeliness requirements in practical applications; Combined with the perceptual loss function, the output video is visually more realistic and natural, improving the audience's viewing experience; By utilizing the adversarial training mechanism of the generative adversarial network, the generative model can continuously optimize itself, adapt to different video content and environments, and improve the robustness and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 The present invention is a flowchart of a method for real-time enhancement of broadcast images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0016] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0017] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0018] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0019] See also Figure 1 As shown, this embodiment provides a method for real-time enhancement of broadcast images, and the method specifically includes the following steps: S1, obtain the original video and the target high-quality video; S2, input the original video into the generative model for enhancement processing to obtain the corresponding simulated high-quality video, use the discriminant model to compare the simulated high-quality video with the target high-quality video, and optimize the generator according to the comparison results; S3, in the optimization process, the generator is adjusted by combining the L2 loss function based on pixel difference and the perceptual loss function, and supervised learning is performed using the high-resolution video and its low-resolution version; S4. Place the generated model into the director's console system. The generated model receives the real-time video stream and enhances each frame to output high-quality broadcast images.

[0020] After step S1 is executed, the original video and the target high-quality video are classified by a classification strategy, and the classification strategy specifically includes: Calculate the color histogram of the video frame and analyze the distribution and saturation of the color to determine whether the video is colorful or dull; Use filters to extract texture features of video frames and determine whether the video is animation or real-life; By calculating the optical flow, the motion information in the video is captured and the video is judged as an action scene or a static scene. By calculating the average brightness and contrast, it is determined that the video is a night scene video or has sufficient light.

[0021] After classification, multiple generators are constructed according to the classification results. Each generator corresponds to a type, and the corresponding generator is used to enhance the corresponding type of video.

[0022] In some embodiments, a specific execution process of the above classification according to the classification strategy is as follows: Collect various types of video data, including cartoons, live-action videos, colorful videos (such as natural scenery), dark-colored videos (such as black-and-white movies), night scene videos, etc. Label each video sample to identify its category. This will be used to train the classification model; Calculate the color histogram for each video frame, obtain color distribution information, and calculate the average color saturation to distinguish between videos with bright colors and dull colors; Use Gabor filters or other texture analysis methods to extract texture features of the frames. These features can help distinguish animation from live-action videos, as animations usually have more uniform and obvious texture patterns. Use optical flow technology to calculate inter-frame motion information and identify dynamic scenes. These features are particularly important for action movies or fast-moving videos. Calculate the average brightness and contrast of the frame to identify night scenes or low-light videos, and use principal component analysis (PCA) to reduce the dimensionality of the extracted high-dimensional features and simplify the data structure; Select a convolutional neural network (CNN) as the classification model and train it using a labeled video dataset to ensure that the model can recognize different types of videos; Based on the classification results, we select a suitable generator for the video. For example, for animation videos, we use a generator that focuses on edge sharpening and color enhancement; for night scene videos, we use a generator that focuses on noise reduction and brightness enhancement.

[0023] The model designs of the generation model and the discrimination model in step S2 are respectively: Generative model, using convolutional neural network architecture, combined with upsampling layers to improve resolution, and using residual blocks to improve the performance of the generator; The discriminant model uses a convolutional neural network architecture to extract features layer by layer, outputs true and false discrimination through a fully connected layer, and uses a PatchGAN architecture to divide the input into small blocks and discriminate them separately.

[0024] In some embodiments, a specific model design of the generation model is as follows: The input layer receives low-quality video frames, which is a 3D tensor (number of frames, width, height, number of channels). The input data is normalized to meet the input requirements of the network. Convolutional layer: Use several convolutional layers to extract low-level features, use small convolution kernels (such as 3x3) and more filters (such as 64 or 128) to capture rich feature information, and use ReLU (RectifiedLinearUnit) as the activation function after each convolutional layer to introduce nonlinearity; Residual Block,Several residual blocks are introduced, each residual block contains two convolutional layers and a short-circuit connection.,Each residual block includes a convolutional layer, batch normalization, ReLU activation,,convolution and batch normalization again, followed by a short-circuit connection.

[0025] The pixel shuffle layer increases the resolution by rearranging the pixels in the feature map; for example, to increase the resolution of the feature map by a factor of s, first increase the number of channels of the feature map to s through a convolutional layer. , and then apply the pixel shuffle operation to reorganize the channel dimension into the spatial dimension to achieve resolution improvement. For an input feature map of size , converted to size by pixel shuffling .

[0026] The output layer uses a convolutional layer to convert the feature map into the number of channels of the target video frame (3 channels of RGB images), and uses the tanh or sigmoid function to limit the output pixel values ​​to an appropriate range.

[0027] In some embodiments, a specific model design of the discriminant model is as follows: Convolutional layer, extract features layer by layer, use smaller convolution kernels (such as 3x3), and reduce the size of feature maps through multi-layer convolution and downsampling (such as pooling or stride convolution); Fully connected layer, a scalar is output through the fully connected layer, indicating whether the input video frame is real or generated; Activation function, use LeakyReLU to avoid the dead neuron problem.

[0028] The output layer uses the sigmoid activation function to output the true and false probabilities.

[0029] The training process of the generative model and the discriminative model includes: Initialize the weights of the generator and discriminator; Use the generator to generate simulated high-quality videos, use the discriminator to calculate the discriminant loss of the target high-quality video and the simulated high-quality video, update the weight of the discriminator and calculate the generation loss of the generator, and update the weight of the generator; The calculation process of discriminative loss and generative loss is repeated until the loss converges.

[0030] The discriminant loss is used to maximize the recognition of real samples and minimize the misjudgment of generated samples. The specific formula is: ; Among them, x is a real sample from the real data distribution pdata; z is a random noise vector from the noise distribution pz; G(z) is the sample generated by the generator; D(x) is the output of the discriminator for the real sample, which is close to 1; D(G(z)) is the output of the discriminator for the generated sample, which is close to 0.

[0031] The generation loss is used to maximize the discriminator's misjudgment of the generated samples, and its specific formula is: .

[0032] The process of adjusting the generator by combining the L2 loss function based on pixel difference and the perceptual loss function in step S3 includes: Input low-quality video z to generator G, generate high-quality video G(z), and calculate L2 loss L L2 , extract the features of the generated video and the target video, and calculate the perceptual loss L perceptual , calculate the total content loss L content , calculate the gradient of the loss, and use the Adam algorithm to update the parameters of the generator to minimize the total content loss.

[0033] Adjust λ according to the relative size and trend of the loss L2 and λ perceptual The value of If the L2 loss is high, increase the weight of the L2 loss, reduce the size of the convolution kernel, and reduce the number of convolution layers; If the perceptual loss is high, increase the weight of the perceptual loss, increase the size of the convolution kernel, and increase the number of convolution layers.

[0034] In some embodiments, a specific execution process of step S3 is as follows: Define the loss function First, we define the total content loss which combines the L2 loss and the perceptual loss; The L2 loss focuses on the difference between the generated video and the target video at the pixel level: ; Perceptual loss focuses on the difference between the generated video and the target video in the high-level feature space. Suppose we use a layer of feature representation from a pre-trained VGG network: ; Combining L2 loss and perceptual loss yields: .

[0035] In the above loss function formula, the meanings of the various parameters and symbols are as follows: x represents a real high-quality video sample sampled from the real data distribution; z represents a low-quality video sample input to the generator, or a noise vector sampled from a noise distribution; y represents the target high-quality video sample, that is, the real video to which we expect the video generated by the generator should be close; G(z) represents the output of the generator G, that is, the simulated high-quality video generated by the generator based on the input z; D(x) represents the output of the discriminator D for the real video sample x, which is usually a value between 0 and 1, indicating the probability that the sample is a real video; D(G(z)) represents the output of the discriminator D for the video G(z) generated by the generator, indicating the probability that the sample is a generated video; L perceptual is the perceptual loss function, which measures the difference between the generated video and the target video in the high-level feature space; Represents a specific layer of a network (such as a VGG network) used to extract features. It maps the input video to the feature space; L content The total content loss function combines L2 loss and perceptual loss to comprehensively measure the quality of the generated video; and Weight coefficient used to balance the contribution of L2 loss and perceptual loss in the total content loss.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0037] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for real-time enhancement of broadcast images, characterized in that: The method comprises: Get the original video and the target high-quality video; The original video is input into the generative model for enhancement processing to obtain the corresponding simulated high-quality video. The discriminant model is used to compare the simulated high-quality video with the target high-quality video, and the generator is optimized according to the comparison results. In the optimization process, the generator is adjusted by combining the L2 loss function based on pixel difference and the perceptual loss function, and supervised learning is performed using the high-resolution video and its low-resolution version; The generative model is placed in the director's console system. The generative model receives the real-time video stream and enhances each frame to output high-quality broadcast images.

2. A method for real-time enhancement of broadcast images according to claim 1, characterized in that: The original video and the target high-quality video are classified by a classification strategy, wherein the classification strategy specifically includes: Calculate the color histogram of the video frame and analyze the color distribution and saturation to determine whether the video is colorful or dull. Use filters to extract texture features of the video frame to determine whether the video is an animation or a real-life video. Calculate the optical flow to capture motion information in the video and determine whether the video is an action scene or a static scene. Calculate the average brightness and contrast to determine whether the video is a night scene or has sufficient light.

3. A method for real-time enhancement of broadcast images according to claim 2, characterized in that: After classification, multiple generators are constructed according to the classification results. Each generator corresponds to a type, and the corresponding generator is used to enhance the corresponding type of video.

4. A method for real-time enhancement of broadcast images according to claim 1, characterized in that: The model designs of the generative model and the discriminative model are respectively: Generative model, using convolutional neural network architecture, combined with upsampling layers to improve resolution, and using residual blocks to improve the performance of the generator; The discriminant model uses a convolutional neural network architecture to extract features layer by layer, outputs true and false discrimination through a fully connected layer, and uses a PatchGAN architecture to divide the input into small blocks and discriminate them separately.

5. A method for real-time enhancement of broadcast images according to claim 1, characterized in that: The training process of the generative model and the discriminative model includes: Initialize the weights of the generator and discriminator; Use the generator to generate simulated high-quality videos, use the discriminator to calculate the discriminant loss of the target high-quality video and the simulated high-quality video, update the weight of the discriminator and calculate the generation loss of the generator, and update the weight of the generator; The calculation process of discriminative loss and generative loss is repeated until the loss converges.

6. A method for real-time enhancement of broadcast images according to claim 5, characterized in that: The discriminant loss is used to maximize the recognition of real samples and minimize the misjudgment of generated samples. The specific formula is: ； Where x is a real sample from the real data distribution pdata; z is a random noise vector from the noise distribution p z ; G(z) is the sample generated by the generator; D(x) is the output of the discriminator for the real sample, which is close to 1; D(G(z)) is the output of the discriminator for the generated sample, which is close to 0.

7. A method for real-time enhancement of broadcast images according to claim 6, characterized in that: The generation loss is used to maximize the discriminator's misjudgment of the generated samples, and its specific formula is: .

8. A method for real-time enhancement of broadcast images according to claim 1, characterized in that: The process of adjusting the generator by combining the L2 loss function based on pixel differences with the perceptual loss function includes: Input low-quality video z to generator G, generate high-quality video G(z), and calculate L2 loss L L2 , extract the features of the generated video and the target video, and calculate the perceptual loss L respectively perceptual , total content loss L content , and the gradient of the loss, and update the parameters of the generator through the Adam algorithm to minimize the total content loss.

Images