Real-time enhancement processing method and device for video frame image

Abstract

The embodiment of the application discloses a kind of real-time enhancement processing method and device of video frame image, wherein the method comprises: obtaining the image sample set containing image sample pair;Low-quality image in image sample pair is input to initial enhancement model and is trained, enhanced image is generated, and the image magnification layer set at the tail of model is used to magnify and process enhanced image;According to the weight parameter of the model adjusted by the magnified enhanced image and high-quality image, until meeting the iteration end condition, obtain the initial enhancement model trained, and remove the image magnification layer from the initial enhancement model trained to obtain target enhancement model;Enhanced frame image is obtained by using target enhancement model to enhance each frame image in the video to be processed.The application adds image magnification layer at the tail of model, improves the model training requirement, effectively improves the image enhancement effect of model, and can efficiently realize real-time enhancement processing to video picture.

Description

Technical Field

[0001] This application relates to the field of Internet technology, specifically to a real-time enhancement processing method and apparatus for video frame images. Background Technology

[0002] Video quality is a key factor affecting the user's viewing experience. Due to various factors, especially in live streaming scenarios, video quality is often not guaranteed, impacting the user's viewing experience. To improve image quality, image enhancement algorithms can be used. However, existing image enhancement algorithms typically require complex calculations for each pixel, resulting in high computational load and slow processing speed, making them unsuitable for scenarios with high real-time requirements. Summary of the Invention

[0003] In view of the above problems, this application proposes a real-time enhancement processing method, apparatus, computing device and computer storage medium for video frame images, to solve the following problems: existing image enhancement algorithms have a large amount of computation and slow processing speed, making them difficult to apply to scenarios with high real-time requirements.

[0004] According to one aspect of the embodiments of this application, a real-time enhancement processing method for video frame images is provided, comprising:

[0005] Obtain an image sample set containing image sample pairs, where each image sample pair includes a low-quality image and a high-quality image corresponding to the low-quality image;

[0006] The low-quality image from the image sample pair is input into the initial enhancement model for training to generate the corresponding enhanced image. The enhanced image is then magnified by an image magnification layer set at the tail of the initial enhancement model.

[0007] Based on the magnified enhanced image and the high-quality image in the image sample pair, adjust the weight parameters of the initial enhancement model until the iteration termination condition is met to obtain the trained initial enhancement model, and remove the image magnification layer from the trained initial enhancement model to obtain the target enhancement model.

[0008] The target enhancement model is used to enhance each frame of the video to be processed, resulting in the enhanced frame image corresponding to each frame.

[0009] Furthermore, the process of inputting the low-quality image from the image sample pair into the initial enhancement model for training to generate the corresponding enhanced image further includes:

[0010] In the initial enhancement model, feature maps are obtained by extracting features from low-quality images. The feature maps are then downsampled using pixel inverse recombination. Finally, the downsampled feature maps are processed through convolutional layers and network base blocks to obtain the processed feature maps.

[0011] A pixel recombination operation is used to upsample the processed feature map to generate an enhanced image corresponding to the low-quality image.

[0012] Furthermore, the process of magnifying the enhanced image by setting an image magnification layer at the tail end in the initial enhancement model further includes:

[0013] The enhanced image is magnified by using bilinear interpolation through an image magnification layer to increase its size by a factor of N, resulting in the magnified enhanced image; where N is greater than 1.

[0014] Furthermore, the image size of the high-quality image included in the image sample pair is N times that of the low-quality image.

[0015] Furthermore, adjusting the weight parameters of the initial enhancement model based on the magnified enhanced image and the high-quality image in the image sample pair further includes:

[0016] Edge detection is performed on low-quality images to obtain edge detection results;

[0017] Based on the edge detection results, a loss function based on edge weights is constructed;

[0018] Based on the loss function, the loss between the magnified enhanced image and the high-quality image in the image sample pair is calculated, and the output value of the loss function is obtained;

[0019] Based on the output value, backpropagation is performed to adjust the weight parameters of the initial augmentation model.

[0020] Furthermore, based on the edge detection results, the loss function constructed based on edge weights further includes:

[0021] A mask image is generated based on the edge detection results. The mask image records the edge weights at each pixel position.

[0022] Based on the edge weights of each pixel location recorded in the mask image, a loss function based on edge weights is constructed.

[0023] Furthermore, the iteration termination condition may include: the number of iterations reaching an iteration threshold; and / or, the output value of the loss function being less than the loss threshold.

[0024] Furthermore, the target enhancement model is used to enhance each frame image in the video to be processed, resulting in enhanced frame images corresponding to each frame image, which further includes:

[0025] A frame-parallel approach is adopted, and the target enhancement model is used to enhance each frame image in the video to be processed, so as to obtain the enhanced frame image corresponding to each frame image.

[0026] Furthermore, using a frame-parallel approach, the target enhancement model is employed to enhance each frame image in the video to be processed, resulting in enhanced frame images corresponding to each frame image, which further include:

[0027] Deploy the target augmentation model across multiple image processors;

[0028] Each frame image in the video to be processed is distributed to multiple image processors. The multiple image processors run the deployed target enhancement model to enhance the received frame images and obtain the enhanced frame images corresponding to the frame images.

[0029] Furthermore, after using the target enhancement model to enhance each frame image in the video to be processed, and obtaining the enhanced frame images corresponding to each frame image, the method also includes:

[0030] The enhanced frame images corresponding to each frame image are arranged according to their order in the video to be processed, thus forming the enhanced video.

[0031] Furthermore, a low-quality image is an image whose image quality parameters are below a preset threshold, and a high-quality image is an image whose image quality parameters are above a preset threshold.

[0032] According to another aspect of the embodiments of this application, a real-time enhancement processing apparatus for video frame images is provided, comprising:

[0033] The acquisition module is adapted to acquire an image sample set containing image sample pairs, where each image sample pair includes a low-quality image and a high-quality image corresponding to the low-quality image;

[0034] The training module is suitable for inputting low-quality images from image sample pairs into the initial enhancement model for training, generating corresponding enhanced images, and amplifying the enhanced images by setting an image magnification layer at the tail of the initial enhancement model; adjusting the weight parameters of the initial enhancement model based on the magnified enhanced images and high-quality images from the image sample pairs until the iteration termination condition is met, obtaining the trained initial enhancement model, and removing the image magnification layer from the trained initial enhancement model to obtain the target enhancement model;

[0035] The application module is suitable for using the target enhancement model to enhance each frame image in the video to be processed, and to obtain the enhanced frame image corresponding to each frame image.

[0036] According to another aspect of the embodiments of this application, a computing device is provided, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0037] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the real-time enhancement processing method for the video frame image described above.

[0038] According to another aspect of the embodiments of this application, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, the executable instruction causing a processor to perform operations corresponding to the real-time enhancement processing method for video frame images described above.

[0039] According to the real-time video frame image enhancement processing method and apparatus provided in the embodiments of this application, during the model training stage, an image magnification layer that does not require parameter training is added to the tail of the initial enhancement model. The image magnification layer magnifies the enhanced image corresponding to the low-quality image generated by the model, which increases the model training requirements and the difficulty of simulating real images. After removing the image magnification layer from the trained initial enhancement model, it becomes the target enhancement model. The target enhancement model not only has a better image enhancement effect but also a higher processing speed. In the feature map conversion process, pixel inverse recombination and pixel recombination operations are used, which can significantly reduce the computational load of the model and speed up the processing speed while ensuring the image enhancement effect. Furthermore, the edge features of the image are incorporated into the loss function to construct an edge weight-based loss function, which effectively guides the model to prioritize learning high-frequency signals that are more sensitive to the human eye, thereby obtaining a better image enhancement effect and better protecting the texture details of the image. In addition, the target enhancement model is deployed in multiple image processors and a frame parallel approach is adopted. The target enhancement model is used to enhance each frame image in the video to be processed, which further improves the processing speed, ensures robust real-time operation, realizes real-time video enhancement processing, and is well applicable to scenarios such as live streaming with high real-time requirements.

[0040] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of the embodiments of this application are described below. Attached Figure Description

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0042] Figure 1 A flowchart illustrating a real-time enhancement processing method for video frame images according to an embodiment of this application is shown.

[0043] Figure 2a A flowchart illustrating a real-time enhancement processing method for video frame images according to another embodiment of this application is shown;

[0044] Figure 2b A schematic diagram of the structure of the initial enhancement model in a real-time enhancement processing method for video frame images according to another embodiment of this application is shown;

[0045] Figure 2c A schematic diagram of the network base block structure in the initial enhancement model is shown;

[0046] Figure 3 A structural block diagram of a real-time enhancement processing apparatus for video frame images according to an embodiment of this application is shown;

[0047] Figure 4 A schematic diagram of the structure of a computing device according to an embodiment of this application is shown. Detailed Implementation

[0048] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0049] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0050] Live streaming is a method of transmitting and playing video, audio, or text content in real time using network technologies such as the internet. Live streaming requires the coordination of multiple technical stages, including acquisition, encoding, transmission, distribution, and playback. Through the collaboration of these technologies, real-time transmission and playback of video, audio, or text content are achieved, allowing users to receive live streaming content through a player.

[0051] Image enhancement: This is a computer algorithm used to improve the quality of images or videos. It uses a series of image processing techniques, such as noise reduction, sharpening, and contrast enhancement, to improve the clarity, color vibrancy, and detail of images, thereby making images or videos more realistic and lifelike.

[0052] Real-time processing refers to the ability to rapidly encode, decode, enhance, compress, and transmit video, delivering high-definition video content to viewers in a very short time to achieve a smooth live streaming experience. Real-time processing requires efficient data processing capabilities, low latency, and high stability, enabling it to quickly respond to viewers' viewing requests under different network environments and process and transmit video content in real time.

[0053] Figure 1 A schematic flowchart of a real-time enhancement processing method for video frame images according to an embodiment of this application is shown, as follows: Figure 1 As shown, the method includes the following steps:

[0054] Step S101: Obtain an image sample set containing image sample pairs.

[0055] To facilitate model training, a sample set was pre-collected, forming an image sample set. This set contains numerous image sample pairs; specifically, each pair includes a low-quality image and its corresponding high-quality image. Within each pair, the low-quality and high-quality images represent comparable image content, but their image quality differs.

[0056] High-quality images can be high-resolution images collected from the internet, databases, etc., such as frames from high-definition videos. Low-quality images corresponding to high-quality images can be generated by processing the high-quality images through noise addition, upsampling, etc.

[0057] Step S102: Input the low-quality image from the image sample pair into the initial enhancement model for training to generate the corresponding enhanced image, and then enlarge the enhanced image by setting an image magnification layer at the tail of the initial enhancement model.

[0058] After obtaining the image sample set, the initial augmentation model can be trained using image sample pairs from the set. The initial augmentation model is a pre-built deep learning model, which may include convolutional layers, network base blocks, etc. The model training process is completed through multiple iterations. Specifically, image sample pairs are extracted from the image sample set, and the low-quality image from the extracted image sample pair is input into the initial augmentation model for training, generating the augmented image corresponding to that low-quality image.

[0059] To further enhance the image enhancement performance of the final trained model and improve the training requirements, an image magnification layer is added at the end of the initial enhancement model. This image magnification layer can be an upsampling layer that does not require parameter training and can easily magnify the image using interpolation. During model training, after generating an enhanced image corresponding to a low-quality image, this enhanced image is input into the image magnification layer. The image magnification layer then enlarges the enhanced image to obtain a larger image size.

[0060] Step S103: Based on the magnified enhanced image and the high-quality image in the image sample pair, adjust the weight parameters of the initial enhancement model until the iteration termination condition is met, and obtain the trained initial enhancement model.

[0061] The loss between the magnified enhanced image corresponding to the low-quality image and the high-quality image corresponding to the low-quality image in the image sample pair can be calculated based on the loss function. The output value of the loss function is then obtained. Backpropagation is performed based on the output value, and the weight parameters of the initial augmentation model are adjusted using the result. This process of inputting a low-quality image, generating the corresponding enhanced image, magnifying the enhanced image, and adjusting the weight parameters of the initial augmentation model is repeated iteratively. The iteration stops when the termination condition is met, resulting in the trained initial augmentation model.

[0062] Step S104: Remove the image magnification layer from the trained initial enhancement model to obtain the target enhancement model.

[0063] After training the initial enhancement model, the image magnification layer is removed from it. The model without the magnification layer is then used as the target enhancement model for practical image enhancement applications. Since the image magnification layer is an upsampling layer that requires no parameter training and only performs interpolation to magnify the image, removing it from the initial enhancement model does not affect the preceding network, i.e., it does not affect the normal operation of the model. Experimental results show that training with the initial enhancement model with an image magnification layer at the end increases the difficulty of simulating realistic images and effectively raises the training requirements. Compared to training with a model without an image magnification layer at the end, the model presented in this application shows a significant improvement in image enhancement performance.

[0064] Step S105: Use the target enhancement model to enhance each frame image in the video to be processed, and obtain the enhanced frame image corresponding to each frame image.

[0065] The video to be processed is obtained, which can be a live video, etc. In step S105, each frame image of the video to be processed can be input into the target enhancement model for enhancement processing, thereby obtaining the enhanced frame image corresponding to each frame image. Furthermore, the target enhancement model obtained in this embodiment not only has a good image enhancement effect, but also has a high processing speed, which can be well applied to scenarios with high real-time requirements, and efficiently realizes real-time enhancement processing of video images.

[0066] According to the real-time enhancement processing method for video frame images provided in the embodiments of this application, during the model training stage, an image magnification layer that does not require parameter training is added to the tail of the initial enhancement model. The image magnification layer magnifies the enhanced image corresponding to the low-quality image generated by the model, which increases the model training requirements and the difficulty of simulating real images. After removing the image magnification layer from the trained initial enhancement model, it is used as the target enhancement model for actual image enhancement application scenarios. The obtained target enhancement model not only has a good image enhancement effect, but also has a high processing speed, which can be well applied to scenarios with high real-time requirements, and efficiently realizes the real-time enhancement processing of video images.

[0067] Figure 2a A flowchart illustrating a real-time enhancement processing method for video frame images according to another embodiment of this application is shown, as follows: Figure 2a As shown, the method includes the following steps:

[0068] Step S201: Obtain an image sample set containing image sample pairs.

[0069] The image sample set includes multiple image sample pairs, each pair consisting of a low-quality image and its corresponding high-quality image. Low-quality images are those with image quality parameters below a preset threshold, while high-quality images are those with image quality parameters above the preset threshold. Image quality parameters may include sharpness, resolution, and other parameters that reflect image quality.

[0070] In step S202, the low-quality image from the image sample pair is input into the initial enhancement model. In the initial enhancement model, feature extraction is performed on the low-quality image to obtain a feature map. The feature map is downsampled using a pixel inverse recombination operation. The downsampled feature map is then processed through convolutional layers and network base blocks to obtain a processed feature map.

[0071] To both accelerate the model's processing speed and ensure the final image enhancement effect, the number of weight parameters in the initial enhancement model can be limited, and a lossless feature map reduction strategy can be introduced at the beginning of the model. This allows the number of parameters to be maintained at a certain baseline while significantly reducing the computational load of the model. In addition, the feature map reduction strategy introduces richer spatial information to a certain extent, enabling the model to learn higher-level semantics.

[0072] Specifically, in the initial enhancement model, feature maps are extracted from the input low-quality image. Then, a pixel unshuffle operation is used to downsample the feature maps to reduce their size and computational cost. Considering that excessive downsizing of the feature maps might introduce image artifacts, this embodiment, after multiple experiments, found that downsampling the feature maps to half their original size reduces computational cost and speeds up processing while maintaining image enhancement effects. The downsampled feature maps are then processed using convolutional layers and network base blocks to obtain the processed feature maps. Furthermore, this embodiment also incorporates numerous residual structures in the initial enhancement model to ensure gradient propagation during training.

[0073] Step S203: Upsample the processed feature map using a pixel recombination operation to generate an enhanced image corresponding to the low-quality image.

[0074] Since the feature map was downsampled using a pixel unshuffle operation in step S202, a pixel shuffle operation is needed in the later stages of network processing to upsample the processed feature map, thereby reconstructing the image and generating an enhanced image corresponding to the low-quality image. This enhanced image and the low-quality image have the same image size. During the feature map transformation, the dual operations of pixel unshuffle and pixel shuffle have extremely low computational cost, and the process is lossless and reversible.

[0075] Step S204: The enhanced image is magnified by an image magnification layer set at the tail of the initial enhancement model.

[0076] Specifically, this image magnification layer can be an upsampling layer that does not require parameter training. The image magnification layer uses bilinear interpolation to enlarge the enhanced image by a factor of N, resulting in a magnified enhanced image; where N is greater than 1. For example, when N equals 2, the dimensions of the enhanced image are magnified to twice their original size, resulting in the magnified enhanced image.

[0077] Figure 2b A schematic diagram of the structure of the initial enhancement model in a real-time enhancement processing method for video frame images according to another embodiment of this application is shown, as follows: Figure 2b As shown, the low-quality image in the image sample pair is input into the initial enhancement model 210. The initial enhancement model 210 extracts features from the low-quality image to obtain a feature map. The feature map is downsampled using a pixel inverse recombination operation. Then, the downsampled feature map is processed through a series of convolutional layers and network base blocks to obtain a processed feature map. Next, the processed feature map is upsampled using a pixel recombination operation to generate an enhanced image corresponding to the low-quality image. Finally, the enhanced image is magnified through an image magnification layer at the end of the model to obtain a magnified enhanced image.

[0078] The structural diagram of the network basic block is as follows: Figure 2c As shown, the Basic Block comprises multiple Dense Blocks. Its main idea is a convolutional neural network with tightly connected layers. Any two layers in this network are directly connected; that is, the input to each layer is the union of the outputs of all preceding layers, and the features learned by this layer are directly passed to all subsequent layers as input. This structure offers advantages such as mitigating gradient vanishing, enhancing feature propagation, and reducing the number of parameters.

[0079] Step S205: Based on the magnified enhanced image and the high-quality image in the image sample pair, adjust the weight parameters of the initial enhancement model until the iteration termination condition is met, and obtain the trained initial enhancement model.

[0080] Because this application adds an image magnification layer at the end of the initial enhancement model, the image size of the image output by the initial enhancement model is inconsistent with the image size of its input image. Specifically, the image size of the magnified enhanced image output by the initial enhancement model is N times the image size of the low-quality image input by the initial enhancement model. In order to facilitate the calculation of the loss between the magnified enhanced image and the high-quality image in the image sample pair, the image sizes of the high-quality image and the low-quality image can be set during the sample preparation process so that the image size of the high-quality image included in the image sample pair is N times that of the low-quality image.

[0081] To achieve better image enhancement, this application proposes an edge-weight-based loss function. Existing edge detection operators can be used to detect edges in low-quality images, identifying which pixel locations are edges. Based on these results, an edge-weight-based loss function is constructed. Specifically, a mask image is generated based on the edge detection results. This mask image records the edge weights for each pixel location, reflecting the degree of edge prominence, such as the probability that the corresponding pixel location is an edge within the image content. The edge-weight-based loss function is then constructed based on these edge weights. In other words, the mask image becomes the weights in the loss function used to protect high-frequency texture features of the image. This edge-weight-based loss function prioritizes the protection of textured parts of the image, rather than indiscriminately learning enhancement for every pixel location. This loss function effectively guides the model to prioritize learning high-frequency signals that are more sensitive to the human eye, given limited weight parameters, thus achieving better image enhancement and better preserving image texture details.

[0082] In step S205, based on the edge-weighted loss function, the loss between the magnified enhanced image and the high-quality image in the image sample pair is calculated to obtain the output value of the loss function. Then, based on the output value, backpropagation is performed to adjust the weight parameters of the initial enhancement model. The above steps of inputting a low-quality image, generating the corresponding enhanced image, magnifying the enhanced image, and adjusting the weight parameters of the initial enhancement model are executed iteratively. After the iteration termination condition is met, the iteration process stops, thereby obtaining the trained initial enhancement model.

[0083] The iteration termination condition may include: the number of iterations reaching an iteration threshold; and / or, the output value of the loss function being less than the loss threshold. Therefore, whether the iteration termination condition is met can be determined by whether the number of iterations reaches the iteration threshold, or by whether the output value of the loss function is less than the loss threshold. Those skilled in the art can also set other iteration termination conditions according to actual needs, which are not limited here.

[0084] Step S206: Remove the image magnification layer from the trained initial enhancement model to obtain the target enhancement model.

[0085] Step S207: Using a frame-parallel approach, the target enhancement model is used to enhance each frame image in the video to be processed, resulting in enhanced frame images corresponding to each frame image.

[0086] To better meet the real-time processing needs of scenarios such as live streaming, a frame-parallel approach is adopted. The target enhancement model is deployed across multiple image processors (GPUs). Each frame image from the video to be processed is distributed to multiple GPUs, and each GPU runs its deployed target enhancement model to enhance the received frame images, resulting in enhanced frame images. This application assigns different frame images of the video to different GPUs for processing. By processing multiple frame images in parallel using multiple GPUs, the overall image processing speed of the video can be effectively improved. Multiple experiments have verified that a real-time processing speed of 60fps (frames per second) can be achieved, realizing real-time enhancement processing of live video.

[0087] Optionally, the method further includes: arranging the enhanced frame images corresponding to each frame image according to the order of each frame image in the video to be processed, forming an enhanced video, so that the user can play the enhanced video through a viewer terminal, etc.

[0088] The real-time video frame image enhancement method provided in this application provides a lightweight model architecture. During the model training phase, an initial enhancement model with an image magnification layer added at the end of the model is used for training, which increases the difficulty of simulating real images and effectively raises the training requirements. Compared with training a model without an image magnification layer added at the end of the model, the image enhancement effect of the final model is effectively improved. In the feature map conversion process, pixel inverse recombination and pixel recombination operations are used, which can significantly reduce the computational load of the model and speed up the processing speed while ensuring the image enhancement effect. Furthermore, the edge features of the image are incorporated into the loss function to construct an edge-weight-based loss function, which effectively guides the model to prioritize learning high-frequency signals that are more sensitive to the human eye, thereby obtaining better image enhancement effects and better protecting the texture details of the image. In addition, the target enhancement model is deployed in multiple image processors and a frame-parallel approach is adopted. The target enhancement model is used to enhance each frame image in the video to be processed, which further improves the processing speed, ensures robust real-time operation, and realizes real-time video enhancement processing. It can be well applied to scenarios such as live streaming with high real-time requirements.

[0089] Figure 3 A structural block diagram of a real-time enhancement processing apparatus for video frame images according to an embodiment of this application is shown, as follows: Figure 3 As shown, the device includes: an acquisition module 310, a training module 320, and an application module 330.

[0090] The acquisition module 310 is adapted to: acquire an image sample set containing image sample pairs, each image sample pair including a low-quality image and a high-quality image corresponding to the low-quality image.

[0091] The training module 320 is adapted to: input the low-quality image from the image sample pair into the initial enhancement model for training, generate the corresponding enhanced image, and enlarge the enhanced image by setting an image magnification layer at the tail of the initial enhancement model; adjust the weight parameters of the initial enhancement model according to the magnified enhanced image and the high-quality image in the image sample pair until the iteration termination condition is met, obtain the trained initial enhancement model, and remove the image magnification layer from the trained initial enhancement model to obtain the target enhancement model.

[0092] Application module 330 is suitable for: using the target enhancement model to enhance each frame image in the video to be processed, and obtaining the enhanced frame image corresponding to each frame image.

[0093] Optionally, the training module 320 is further adapted to: extract features from the low-quality image to obtain a feature map in the initial enhancement model; downsample the feature map by performing a pixel inverse recombination operation; and process the downsampled feature map through convolutional layers and network base blocks to obtain a processed feature map; and upsample the processed feature map by performing a pixel recombination operation to generate an enhanced image corresponding to the low-quality image.

[0094] Optionally, the training module 320 is further adapted to: enlarge the enhanced image by using bilinear interpolation through the image magnification layer to enlarge the image size of the enhanced image by N times, thereby obtaining the magnified enhanced image; where N is greater than 1.

[0095] Optionally, the image size of the high-quality image included in the image sample pair is N times that of the low-quality image.

[0096] Optionally, the training module 320 is further adapted to: perform edge detection on the low-quality image to obtain the edge detection result; construct a loss function based on edge weights based on the edge detection result; calculate the loss between the magnified enhanced image and the high-quality image in the image sample pair based on the loss function to obtain the output value of the loss function; and perform backpropagation based on the output value to adjust the weight parameters of the initial enhancement model.

[0097] Optionally, the training module 320 is further adapted to: generate a mask image based on the edge detection results, wherein the mask image records the edge weights at each pixel position; and construct a loss function based on the edge weights at each pixel position recorded in the mask image.

[0098] Optionally, the iteration termination condition may include: the number of iterations reaches an iteration threshold; and / or, the output value of the loss function is less than the loss threshold.

[0099] Optionally, the application module 330 is further adapted to: use a frame-parallel approach to enhance each frame image in the video to be processed using the target enhancement model, thereby obtaining the enhanced frame image corresponding to each frame image.

[0100] Optionally, the application module 330 is further adapted to: deploy the target enhancement model in multiple image processors; distribute each frame image in the video to be processed to multiple image processors, and have the multiple image processors run the deployed target enhancement model to enhance the received frame images to obtain the enhanced frame images corresponding to the frame images.

[0101] Optionally, the application module 330 is further adapted to: arrange the enhanced frame images corresponding to each frame image according to the order of each frame image in the video to be processed, so as to form an enhanced video.

[0102] Optionally, a low-quality image is an image whose image quality parameters are below a preset threshold, and a high-quality image is an image whose image quality parameters are above a preset threshold.

[0103] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments, and will not be repeated here.

[0104] The real-time video frame image enhancement processing apparatus provided in the embodiments of this application offers a lightweight model architecture. During the model training phase, an initial enhancement model with an image magnification layer added at the end of the model is used for training, increasing the difficulty of simulating real images and effectively raising the training requirements. Compared with training a model without an image magnification layer added at the end of the model, this effectively improves the image enhancement effect of the final model. During feature map conversion, pixel inverse recombination and pixel recombination operations are employed, significantly reducing the computational load of the model and accelerating processing speed while ensuring image enhancement effects. Furthermore, edge features of the image are incorporated into the loss function to construct an edge-weight-based loss function, effectively guiding the model to prioritize learning high-frequency signals that are more sensitive to the human eye, resulting in better image enhancement effects and better preservation of image texture details. Additionally, the target enhancement model is deployed across multiple image processors in a frame-parallel manner, using the target enhancement model to enhance each frame image in the video to be processed, further improving processing speed and ensuring robust real-time operation. This achieves real-time video enhancement processing and is well-suited for scenarios with high real-time requirements, such as live streaming.

[0105] This application also provides a non-volatile computer storage medium storing at least one executable instruction that can execute the real-time enhancement processing method for video frame images in any of the above method embodiments.

[0106] Figure 4The diagram shows a structural schematic of a computing device according to one embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the computing device.

[0107] like Figure 4 As shown, the computing device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0108] in:

[0109] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0110] Communication interface 404 is used to communicate with other network elements such as clients or other servers.

[0111] The processor 402 is used to execute program 410, specifically to execute the relevant steps in the above-described embodiment of the real-time enhancement processing method for video frame images.

[0112] Specifically, program 410 may include program code that includes computer operation instructions.

[0113] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0114] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0115] Specifically, program 410 can be used to cause processor 402 to execute the real-time enhancement processing method for video frame images in any of the above method embodiments. The specific implementation of each step in program 410 can be found in the corresponding descriptions of the steps and units in the above real-time enhancement processing embodiments for video frame images, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0116] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the contents of the embodiments of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best implementation of the embodiments of this application.

[0117] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0118] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of the present application, various features of the present application embodiments are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed embodiments of the present application require more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the present application.

[0119] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0120] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are meant to be within the scope of the embodiments of this application and form different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0121] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of this application. The embodiments of this application can also be implemented as device or apparatus programs (e.g., computer programs and computer program products) for performing part or all of the methods described herein. Such programs implementing the embodiments of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0122] It should be noted that the above embodiments are illustrative of the embodiments of this application and not limiting of the embodiments of this application, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of this application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A real-time enhancement processing method for video frame images, comprising: Obtain an image sample set containing image sample pairs, where each image sample pair includes a low-quality image and a high-quality image corresponding to the low-quality image; The low-quality image from the image sample pair is input into the initial enhancement model for training to generate the corresponding enhanced image, and the enhanced image is magnified by the image magnification layer set at the tail of the initial enhancement model; Based on the magnified enhanced image and the high-quality image in the image sample pair, the weight parameters of the initial enhancement model are adjusted until the iteration termination condition is met, resulting in a trained initial enhancement model. The image magnification layer is then removed from the trained initial enhancement model to obtain the target enhancement model. The image magnification layer is an upsampling layer that does not require parameter training and uses interpolation to magnify the enhanced image. Removing the image magnification layer from the trained initial enhancement model does not affect the normal operation of the model. The target enhancement model is used to enhance each frame image in the video to be processed, thereby obtaining the enhanced frame image corresponding to each frame image.

2. The method according to claim 1, wherein, The step of inputting the low-quality image from the image sample pair into the initial enhancement model for training to generate the corresponding enhanced image further includes: In the initial enhancement model, feature maps are obtained by extracting features from the low-quality image, the feature maps are downsampled by pixel inverse recombination, and the downsampled feature maps are processed by convolutional layers and network base blocks to obtain processed feature maps. The processed feature map is upsampled using a pixel recombination operation to generate an enhanced image corresponding to the low-quality image.

3. The method according to claim 1, wherein, The step of amplifying the enhanced image by using an image amplification layer located at the tail of the initial enhancement model further includes: The enhanced image is magnified by using bilinear interpolation in the image magnification layer to increase its size by N times, resulting in a magnified enhanced image; where N is greater than 1.

4. The method according to claim 3, wherein, The image size of the high-quality image included in the image sample pair is N times that of the low-quality image.

5. The method according to claim 1, wherein, The step of adjusting the weight parameters of the initial enhancement model based on the magnified enhanced image and the high-quality image in the image sample pair further includes: Edge detection is performed on the low-quality image to obtain the edge detection results; Based on the edge detection results, a loss function based on edge weights is constructed; Based on the loss function, the loss between the magnified enhanced image and the high-quality image in the image sample pair is calculated to obtain the output value of the loss function; Based on the output value, backpropagation is performed to adjust the weight parameters of the initial augmentation model.

6. The method according to claim 5, wherein, The step of constructing a loss function based on edge weights according to the edge detection results further includes: A mask image is generated based on the edge detection results, and the mask image records the edge weights at each pixel position; Based on the edge weights of each pixel position recorded in the mask image, a loss function based on edge weights is constructed.

7. The method according to any one of claims 1-6, wherein, The iteration termination condition may include: the number of iterations reaches an iteration threshold; and / or, the output value of the loss function is less than the loss threshold.

8. The method according to any one of claims 1-6, wherein, The step of using the target enhancement model to enhance each frame image in the video to be processed, and obtaining the enhanced frame images corresponding to each frame image, further includes: Using a frame-parallel approach, the target enhancement model is used to enhance each frame image in the video to be processed, resulting in enhanced frame images corresponding to each frame image.

9. The method according to claim 8, wherein, The step of using a frame-parallel approach to enhance each frame image in the video to be processed using the target enhancement model, to obtain the enhanced frame images corresponding to each frame image, further includes: The target enhancement model is deployed across multiple image processors; Each frame image in the video to be processed is distributed to multiple image processors, and the multiple image processors run the deployed target enhancement model to enhance the received frame images to obtain the enhanced frame images corresponding to the frame images.

10. The method according to any one of claims 1-6, wherein, After enhancing each frame image in the video to be processed using the target enhancement model to obtain the enhanced frame images corresponding to each frame image, the method further includes: The enhanced frame images corresponding to each frame image are arranged according to their arrangement order in the video to be processed to form an enhanced video.

11. The method according to any one of claims 1-6, wherein, The low-quality image is an image whose image quality parameters are lower than a preset threshold, and the high-quality image is an image whose image quality parameters are higher than the preset threshold.

12. A real-time enhancement processing apparatus for video frame images, comprising: The acquisition module is adapted to acquire an image sample set containing image sample pairs, each image sample pair including a low-quality image and a high-quality image corresponding to the low-quality image; The training module is adapted to input the low-quality image from the image sample pair into the initial enhancement model for training, generate the corresponding enhanced image, and enlarge the enhanced image by setting an image magnification layer at the tail of the initial enhancement model; Based on the magnified enhanced image and the high-quality image in the image sample pair, the weight parameters of the initial enhancement model are adjusted until the iteration termination condition is met, resulting in a trained initial enhancement model. The image magnification layer is then removed from the trained initial enhancement model to obtain the target enhancement model. The image magnification layer is an upsampling layer that does not require parameter training and uses interpolation to magnify the enhanced image. Removing the image magnification layer from the trained initial enhancement model does not affect the normal operation of the model. The application module is adapted to use the target enhancement model to enhance each frame image in the video to be processed, so as to obtain the enhanced frame image corresponding to each frame image.

13. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the real-time enhancement processing method for video frame images as described in any one of claims 1-11.

14. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the real-time enhancement processing method for video frame images as described in any one of claims 1-11.

15. A computer program product comprising at least one executable instruction that causes a processor to perform an operation corresponding to the real-time enhancement processing method for video frame images as described in any one of claims 1-11.

Images