Video enhancement method and device

Abstract

The embodiment of the invention provides a video enhancement method and related equipment / products, and belongs to the technical field of video processing. The video enhancement method comprises the following steps: acquiring an original video frame and a service state flag, wherein the service state flag is used for representing a target service scene; according to the target service scene, determining a target enhanced network from two pre-trained enhanced networks, and converting the original video frame into the enhanced video frame through the target enhanced network; wherein the enhanced networks and the combination thereof respectively correspond to different service scenes, and the target enhanced network comprises at least one enhanced network. According to the technical scheme of the embodiment of the invention, the video enhancement algorithm is divided into two enhancement networks with independent functions, and the enhancement networks can be flexibly combined or separated according to the service state flag, so that the service requirements of different time periods can be efficiently met, resource waste caused by redundant training is relieved, and the user experience is improved. Therefore, the video enhancement effect is effectively improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of video processing, and in particular, to a video enhancement method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Art

[0002] Video image quality is a key indicator for measuring the visual effect of a video, directly affecting the viewing experience and information transmission. In live broadcast scenarios, due to factors such as shooting equipment and transmission bandwidth, the image quality and resolution of most videos are not ideal. However, the existing video enhancement methods have fixed processes, cannot meet the requirements of different time periods, and require a large amount of computing resources.

[0003] It should be noted that the above content is not necessarily prior art and is not used to limit the patent protection scope of the present application. Summary of the Invention

[0004] Embodiments of the present application provide a video enhancement method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the above-mentioned technical problems.

[0005] One aspect of the embodiments of the present application provides a video enhancement method, the method comprising: Obtaining an original video frame and a service status flag, the service status flag being used to represent a target service scenario; Determining a target enhancement network from two pre-trained enhancement networks according to the target service scenario, and converting the original video frame into the enhanced video frame through the target enhancement network; Wherein, the enhancement network and its combination respectively correspond to different service scenarios, and the target enhancement network includes at least one of the enhancement networks.

[0006] Optionally, the service scenario includes a peak service period and a stable service period, the service status flag includes a first service status flag and a second service status flag, the first service status flag corresponds to the peak service period, the second service status flag corresponds to the stable service period, and the two enhancement networks include a same-resolution enhancement network and a super-resolution enhancement network; Correspondingly, determining a target enhancement network from two pre-trained enhancement networks according to the target service scenario includes: When the service status flag is the first service status flag, determining the same-resolution enhancement network and the super-resolution enhancement network as the target enhancement network; When the service status flag is the second service status flag, determining the same-resolution enhancement network as the target enhancement network.

[0007] Optionally, converting the original video frame into the enhanced video frame through the target enhancement network includes: When the target enhancement network is the same-resolution enhancement network and the super-resolution enhancement network: denoising and texture reconstruction are performed on the original video frame through the same-resolution enhancement network to obtain an intermediate video frame; super-resolution processing is performed on the intermediate video frame through the super-resolution enhancement network to obtain the enhanced video frame; When the target enhancement network is the same-resolution enhancement network, denoising and texture reconstruction are performed on the original video frame through the same-resolution enhancement network to obtain the enhanced video frame.

[0008] Optionally, the service scenario further includes a super-resolution service, and the service status flag further includes a third service status flag corresponding to the super-resolution service; correspondingly, determining the target enhancement network from two pre-trained enhancement networks according to the target service scenario includes: When the service status flag is the third service status flag, the super-resolution enhancement network is determined as the target enhancement network.

[0009] Optionally, converting the original video frame into the enhanced video frame through the target enhancement network includes: When the target enhancement network is the super-resolution enhancement network, super-resolution processing is performed on the original video frame through the super-resolution enhancement network to obtain the enhanced video frame.

[0010] Optionally, the same-resolution enhancement network is obtained through the following operations: Obtain a first training set, where the first training set includes a plurality of first training pairs, and each first training pair includes a first sample frame and a first reference frame. The first sample frame is obtained by degrading the first reference frame by a first degree, and the degradation processing includes one or more of adding blur, Gaussian noise, and coding noise; Train a fully convolutional neural network based on the first training set to obtain a basic network; Obtain a second training set, where the second training set includes a plurality of second training pairs, and each second training pair includes a second sample frame and a second reference frame. The second sample frame is obtained by degrading the second reference frame by a second degree, and the second degree is greater than the first degree; Train the basic network based on the second training set and the generative adversarial network mechanism to obtain the same-resolution enhancement network.

[0011] Optionally, the super-resolution enhancement network is obtained through the following operations: Obtain a third training set, where the third training set includes a plurality of third training pairs, and each third training pair includes a third sample frame and a third reference frame. The third sample frame is obtained by reducing the resolution of the third reference frame and performing degradation processing to a third degree, and the third degree is less than the first degree; Train the residual local feature network based on the third training set to obtain the super-resolution enhancement network.

[0012] Another aspect of the embodiments of the present application provides a video enhancement device, and the device includes: A first acquisition module, configured to acquire an original video frame and a service status flag, where the service status flag is used to represent a target service scenario; A second acquisition module, configured to determine a target enhancement network from two pre-trained enhancement networks according to the target service scenario, and convert the original video frame into the enhanced video frame through the target enhancement network; Wherein, the enhancement network and its combination respectively correspond to different service scenarios, and the target enhancement network includes at least one of the enhancement networks.

[0013] Another aspect of the embodiments of the present application provides a computer device, including: At least one processor; and A memory communicatively connected to the at least one processor; Wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0014] Another aspect of the embodiments of the present application provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed by a processor, the method as described above is implemented.

[0015] Another aspect of the embodiments of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method as described above is implemented.

[0016] The embodiments of the present application adopting the above technical solutions may include the following advantages: Obtain the original video frame and the service status flag, where the service status flag is used to represent the target service scenario. Two enhancement networks are pre-trained, and each enhancement network and its combination correspond to different service scenarios respectively. According to the target service scenario, the target enhancement network can be determined from the two enhancement networks. Among them, the target enhancement network includes at least one enhancement network. The original video frame can be converted into an enhanced video frame through the target enhancement network. It can be seen that in the embodiments of the present application, by dividing the video enhancement algorithm into two functionally independent enhancement networks, these enhancement networks can be flexibly combined or separated according to the service status flag, which can efficiently meet the service requirements in different time periods, alleviate the waste of resources caused by redundant training, and effectively improve the video enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings exemplarily show the embodiments and form a part of the specification, and are used together with the written description of the specification to explain the exemplary embodiments. The shown embodiments are only for illustrative purposes and do not limit the scope of the claims. In all the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 Schematically shows the flowchart of the video enhancement method according to Embodiment 1 of the present application; Figure 2 Schematically shows Figure 1 the sub-step flowchart of step S102 in Figure 3 Schematically shows Figure 2 the sub-step flowchart of step S200 in Figure 4 Schematically shows the flowchart for obtaining the same-resolution enhancement network according to Embodiment 1 of the present application; Figure 5 Schematically shows the flowchart for obtaining the super-resolution enhancement network according to Embodiment 1 of the present application; Figure 6 is an application example diagram of the video enhancement method according to Embodiment 1 of the present application; Figure 7 Schematically shows the block diagram of the video enhancement device according to Embodiment 2 of the present application; and Figure 8 Schematically shows the hardware architecture diagram of the computer device according to Embodiment 3 of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] In order to make the objectives, technical solutions, and advantages of this application clearer and more understandable, the following further elaborates on this application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts belong to the scope of protection of this application.

[0020] It should be noted that in the embodiments of this application, the descriptions involving "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. Additionally, the technical solutions between various embodiments can be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] In the description of this application, it should be understood that the numerical labels before the steps do not identify the sequence of execution of the steps, but are only used to conveniently describe this application and distinguish each step. Therefore, it cannot be understood as a limitation to this application.

[0022] First, the following provides explanations of the terms involved in this application: Video enhancement: A technology that processes video content (video frames) through algorithms or models to improve its visual quality, including but not limited to removing noise, enhancing details, improving colors, etc., making the video clearer and more attractive.

[0023] Super resolution: A technology that converts low-resolution images or videos into high-resolution images or videos through algorithms, with the goal of generating a picture with richer details and higher clarity.

[0024] Generative Adversarial Network (GAN): A deep learning model composed of two neural networks (generator - discriminator). Among them, the generator is used to generate realistic data, and the discriminator is used to distinguish between generated data and real data, competing with each other to improve the quality of the generated data.

[0025] SDK (Software Development Kit): Software Development Kit.

[0026] Secondly, to facilitate the understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the following explains the related technologies: Video image quality is a key indicator for measuring the visual effect of videos, directly affecting the viewing experience and information transmission. In live broadcast scenarios, due to factors such as shooting equipment and transmission bandwidth, the image quality and resolution of most videos are not ideal.

[0027] However, the applicant has learned that the related video enhancement technologies still have the following defects: (1) It is necessary to design and train independent models for different business scenarios respectively, and each model is responsible for different business functions. For example, one model is used for image quality enhancement at the same resolution, and another model is used for super-resolution processing. Due to the differences in the architecture design, training strategy, and deployment process of these models, the system complexity increases significantly. Especially in the case of limited resources, this complexity is not conducive to the management and maintenance of the models.

[0028] (2) The process is fixed and cannot dynamically adjust the video enhancement strategy according to actual needs. For example: in some business scenarios, redundant training may lead to waste of computing power, while in some business scenarios, the inherent video enhancement strategy may not be able to efficiently meet the real-time requirements, especially when processing high-frame-rate video content (such as 60fps).

[0029] (3) Due to the design differences of independent models, when switching models in the same business scenario, the overall image quality style performance of the video may be inconsistent, thus affecting the viewing experience.

[0030] To this end, the embodiments of this application provide a technical solution for video enhancement. In this technical solution: (1) System flexibility is improved: Through modular network design, the video enhancement algorithm is split into two independent modules, namely same-resolution enhancement and super-resolution enhancement. Each module can be independently trained and optimized, and the trained modules can be flexibly combined or independently enabled according to business requirements in actual applications. For example, during peak business periods, both modules can be enabled simultaneously to achieve a picture quality improvement from 1080p to 4K to provide the best picture quality, while during stable or low-demand business periods, only the same-resolution enhancement module needs to be enabled to meet the requirements, thereby reducing computing power and bandwidth consumption. (2) Resource efficiency is optimized: It adapts to limited computing resources and maximizes the utilization efficiency of computing power while meeting the requirements of high-frame-rate real-time processing. Especially in low-demand scenarios, only using the same-resolution enhancement module can effectively reduce resource waste, while in high-demand scenarios, combining the two modules can ensure the best picture quality. (3) Good consistency in picture quality performance: The main picture quality restoration function is assigned to the same-resolution enhancement module. The picture quality style in the embodiments of this application in different business scenarios mainly depends on this module, thereby achieving a high level of compatibility and consistency. Even when switching different video enhancement strategies, it can ensure that the picture quality performance of the video output is stable and excellent, effectively improving the viewing experience. (4) Real-time processing ability is enhanced: Considering the characteristics of poor source picture quality in scenarios such as game live broadcasts, the modular design of the embodiments of this application focuses on the lightweight of the network to ensure meeting the requirements of real-time processing at high frame rates (such as 60fps), and adapting to various live broadcast scenarios and computing power conditions. See the following for details.

[0031] The technical solution of this application will be introduced through multiple embodiments below. It should be noted that these embodiments can be implemented in many different forms and should not be construed as being limited only to the embodiments described herein.

[0032] Embodiment 1 Figure 1 The flowchart of the video enhancement method according to Embodiment 1 of this application is schematically shown.

[0033] As Figure 1 shown, the video enhancement method may include steps S100 to S102, where: Step S100, obtain an original video frame and a service status flag, where the service status flag is used to represent the target service scenario.

[0034] Step S102, according to the target service scenario, determine a target enhancement network from two pre-trained enhancement networks, and convert the original video frame into the enhanced video frame through the target enhancement network; where the enhancement network and its combination respectively correspond to different service scenarios, and the target enhancement network includes at least one of the enhancement networks.

[0035] The video enhancement method provided in this embodiment obtains an original video frame and a service status flag, where the service status flag is used to represent a target service scenario. Two enhancement networks are pre-trained, and the enhancement network and its combination respectively correspond to different service scenarios. According to the target service scenario, a target enhancement network can be determined from the two enhancement networks. Among them, the target enhancement network includes at least one enhancement network. The original video frame can be converted into an enhanced video frame through the target enhancement network. It can be seen that in the embodiment of this application, by dividing the video enhancement algorithm into two independent enhancement networks, these enhancement networks can be flexibly combined or separated according to the service status flag, which can efficiently meet the service requirements in different time periods, alleviate the waste of various resources caused by redundant training, and effectively improve the video enhancement effect.

[0036] The following will combine Figure 1 to elaborate in detail on each step in steps S100 to S102 and other optional steps.

[0037] It should be noted that the video enhancement method provided in the embodiment of this application is applicable to videos in various service scenarios. The service scenarios can be game live broadcast, video on demand, video conferencing, monitoring, etc. Hereinafter, taking the game live broadcast as an example, an exemplary introduction to the video enhancement method will be given.

[0038] Step S100 , obtain an original video frame and a service status flag, where the service status flag is used to represent a target service scenario.

[0039] Exemplarily, a video stream of a game live broadcast can be obtained and decoded into continuous single-frame images, thereby obtaining the original video frame. At the same time, a service status flag can be obtained, and the service status flag can be used for the service scenario. The service status can include the service peak period and the service stable period (also called the trough period). Taking the game live broadcast as an example, the service peak period can be popular events, anniversary live broadcast activities, new game press conferences, etc. The service stable period or trough period can be regular live broadcasts, non-holiday live broadcasts, etc. In practical applications, different service status flags can be used for differentiation. For example, the service status flag can include a first service status flag and a second service status flag, where the first service status flag can correspond to the service peak period, and the second service status flag can correspond to the service stable period or trough period.

[0040] Step S102 , according to the target service scenario, determine a target enhancement network from the two pre-trained enhancement networks, and convert the original video frame into the enhanced video frame through the target enhancement network; where the enhancement network and its combination respectively correspond to different service scenarios, and the target enhancement network includes at least one of the enhancement networks.

[0041] Exemplarily, two independent enhancement networks (modules) can be pre-trained. Each enhancement network can be responsible for specific business functions, such as denoising, enhancement, super-resolution, or style transfer, etc., so as to adapt to different business scenarios. Of course, the combination of two or more enhancement networks can also adapt to different business scenarios. These enhancement networks can be fused at the model level, and specifically, they can be designed following the principle of high cohesion and low coupling to ensure the lowest dependence between modules, while the implementation within each module is highly optimized.

[0042] According to the business status flag, the video enhancement requirements (target business scenario) at the current moment can be clarified. For example, during the business peak period such as a popular event, the platform wants to launch high-frame-rate and high-quality game videos, but the original quality of the game live stream is poor. Therefore, the enhancement network responsible for image quality restoration and resolution improvement can be determined as the target enhancement network, and the original video frames are processed by the target enhancement network to obtain enhanced video frames. During the stable period or business trough period of regular live broadcasts and other services, the requirements for game videos are relatively low. To save computing power, only the same-resolution enhancement network can be enabled to process the original video frames to obtain enhanced video frames.

[0043] In this embodiment, by modular network design, the video enhancement algorithm is divided into two independent enhancement networks. The enhancement networks can be flexibly combined or used independently according to the actual business scenario requirements, applicable to high-frame-rate (60fps) real-time processing, and also supporting limited computing power resources, thereby enabling efficient video optimization.

[0044] Next, the video enhancement method will be further introduced exemplarily in combination with multiple embodiments.

[0045] In an alternative embodiment, the two enhancement networks can include a same-resolution enhancement network and a super-resolution enhancement network. Among them, the same-resolution enhancement network can be used for image quality restoration and detail generation of same-resolution videos (such as 1080p to 1080p, 4K to 4K), and can effectively remove noise and restore details, improving the viewing effect of the video. The super-resolution enhancement network can be used to increase the video resolution, for example: 1080p to 4K, 4K to 8K, etc. In this embodiment, the same-resolution enhancement network and the super-resolution enhancement network can be fused at the model level. For example: integrating the two enhancement networks into an overall model (video enhancement model) for video processing can reduce the inference time and improve the efficiency. Correspondingly, as Figure 2 shown, step S102 may include: Step S200, when the business status flag is the first business status flag, determining the same-resolution enhancement network and the super-resolution enhancement network as the target enhancement network.

[0046] Step S202, when the service status flag is the second service status flag, determine the same-resolution enhancement network as the target enhancement network.

[0047] Exemplarily, if the obtained service status flag is the first service status flag, it means that the current is in the business peak period. Correspondingly, a combination of the same-resolution enhancement network and the super-resolution enhancement network can be called to process the original video frame to obtain an enhanced video frame. If the obtained service status flag is the second service status flag, it means that the current is in the business stable period or the trough period. In order to save expenses such as bandwidth, only the same-resolution enhancement network can be called to process the original video frame to obtain an enhanced video frame.

[0048] In this embodiment, a flexible and adjustable video enhancement strategy is provided, and the enhancement module can be dynamically selected and enabled according to actual needs. In the business peak period, both the same-resolution enhancement network and the super-resolution enhancement network can be enabled to ensure the best video image quality. In the business stable period or the trough period, only the same-resolution enhancement network can be enabled to effectively reduce resource waste.

[0049] In an alternative embodiment, as Figure 3 shown, step S100 may include: Step S300, when the target enhancement network is the same-resolution enhancement network and the super-resolution enhancement network: perform denoising processing and texture reconstruction on the original video frame through the same-resolution enhancement network to obtain an intermediate video frame; perform super-resolution processing on the intermediate video frame through the super-resolution enhancement network to obtain the enhanced video frame.

[0050] Step S302, when the target enhancement network is the same-resolution enhancement network, perform denoising processing and texture reconstruction on the original video frame through the same-resolution enhancement network to obtain the enhanced video frame.

[0051] Exemplarily, when the target enhancement network is the same-resolution enhancement network and the super-resolution enhancement network: The original video frame can be denoised and texture-reconstructed by the same-resolution enhancement network to obtain an intermediate video frame. The intermediate video frame can be super-resolved by the super-resolution enhancement network to improve the resolution and obtain an enhanced video frame. These two enhancement networks can be model-fused at the model level, integrating the two enhancement networks into an overall model for processing, and a single input can obtain the final output, effectively reducing the inference time and improving the efficiency. In some embodiments, the same-resolution enhancement network and the super-resolution enhancement network can also work in parallel to process the original video frame in real time. Under the requirement of high frame rate (such as 60fps), the processing delay can be ensured to be within a reasonable range. When the target enhancement network is the same-resolution enhancement network, the same-resolution enhancement network can be called only to denoise and texture-reconstruct the original video frame to obtain an enhanced video frame that meets the basic requirements.

[0052] In this embodiment, enabling both the same-resolution enhancement network and the super-resolution enhancement network simultaneously can meet the video enhancement requirements of high frame rate and high image quality, improving the viewing experience; only calling the same-resolution enhancement network can effectively reduce the computing power and bandwidth consumption on the premise of ensuring a certain image quality.

[0053] In an alternative embodiment, the service scenario may further include a super-resolution scenario. In the super-resolution scenario, the image quality of the original frame has reached a relatively high level, and the main requirement is to improve the detail resolution (super-resolution), without the need for same-resolution enhancement. Correspondingly, the service status flag may include a third service status flag for indicating the super-resolution scenario. Step S102 may further include: Step S400, when the service status flag is the third service status flag, determining the super-resolution enhancement network as the target enhancement network.

[0054] Exemplarily, when receiving an original video frame, the image quality of the original video frame can be determined by methods such as calculating the signal-to-noise ratio and deep learning. If the image quality of the original video frame is higher than a preset threshold, it indicates that the image quality of the original video frame is good, the service scenario is a super-resolution scenario, and a third service status identifier is generated. To further reduce the computing power and bandwidth consumption, only the super-resolution enhancement network can be called according to the third service status identifier to improve the video resolution and thus provide a better viewing experience.

[0055] In this embodiment, dynamically selecting a suitable enhancement network according to the image quality of the original video frame saves computing power and effectively optimizes the video quality and viewing experience.

[0056] In an alternative embodiment, step S102 may further include: when the target enhancement network is the super-resolution enhancement network, performing super-resolution processing on the original video frame through the super-resolution enhancement network to obtain the enhanced video frame.

[0057] In this embodiment, performing super-resolution processing on the original video frame through the super-resolution enhancement network can further enhance the level of details in the picture, restore or generate minute details that were not reflected in the original resolution, effectively improve the resolution, and thus enhance the viewing experience.

[0058] In an alternative embodiment, as Figure 4 shown, the same-resolution enhancement network can be trained through the following steps: Step S400, obtain a first training set, the first training set includes a plurality of first training pairs, each first training pair includes a first sample frame and a first reference frame, the first sample frame is obtained by degrading the first reference frame by a first degree, and the degradation processing includes adding one or more of blur, Gaussian noise, and coding noise.

[0059] Step S402, train a fully convolutional neural network based on the first training set to obtain a basic network.

[0060] Step S404, obtain a second training set, the second training set includes a plurality of second training pairs, each second training pair includes a second sample frame and a second reference frame, the second sample frame is obtained by degrading the second reference frame by a second degree, and the second degree is greater than the first degree.

[0061] Step S406, train the basic network based on the second training set and the generative adversarial network mechanism to obtain the same-resolution enhancement network.

[0062] Exemplarily, the same-resolution enhancement network can optimize its basic image quality enhancement ability and texture reconstruction ability through a two-stage training strategy. In the first stage, basic restoration and denoising training are mainly carried out. The goal of this stage is to enable the network to effectively repair common degradations in images, such as blur, Gaussian noise, coding noise, etc. A first training set is pre-obtained. The first training set can include multiple first training pairs for supervised learning. Each first training pair can include a first sample frame and a first reference frame. Among them, the first sample frame can be obtained by subjecting the first reference frame to a first degree of degradation processing. Degradation processing is a technique for reducing the quality of an image or video, which is achieved by introducing noise, reducing color information, and reducing details. For example: adding one or more mild forms of degradation such as blur, Gaussian noise, and coding noise to the first reference frame to obtain the first sample frame. Based on the first training set and the UNet architecture, training can be carried out to obtain a basic network. The basic network learns to repair low-frequency noise in the first training set, can restore the clarity of the image, and ensures that the image has a certain repair ability under different degradation conditions. Among them, UNet is a classic fully convolutional neural network (Convolutional Neural Network, CNN), which has a symmetric encoder and decoder structure and can be applied to image restoration and enhancement tasks. In this embodiment, the original UNet architecture can also be lightweight adjusted, such as reducing the number of channels, etc., to further reduce the computational complexity of the network to adapt to hardware with limited resources, while maintaining high-quality image enhancement effects. In the second stage, texture repair and detail enhancement are mainly carried out. In this stage, the degree of degradation in the pre-obtained second training set is further deepened. Specifically, the second training set includes multiple second training pairs, and each second training pair includes a second sample frame and a second reference frame. Among them, the second sample frame can be obtained by subjecting the second reference frame to a second degree of degradation processing. The second degree is greater than the first degree, indicating that the degree of degradation in the second stage is greater than that in the first stage. Based on the second training set, the basic network will further learn how to repair complex texture details and introduce a generative adversarial network (GAN) mechanism to enhance the reconstruction ability of high-frequency details, resulting in the same-resolution enhancement network. The adversarial training mechanism of GAN enables the network to not only remove complex noise but also accurately reconstruct detailed textures through the game between the generator and the discriminator, especially performing well in edges and local details, effectively ensuring the naturalness and fidelity of image restoration.

[0063] In this embodiment, a same-resolution enhancement network is obtained through a two-stage training strategy, which undertakes the main functions of image quality restoration and reconstruction in the entire video enhancement system, and its model structure is more complex than that of the super-resolution enhancement network. At the same time, the super-resolution enhancement network does not introduce a powerful image quality restoration function, which can prevent over-enhancement phenomena such as over-sharp or over-smooth enhanced video frames when the two are combined. The embodiments of the present application clearly divide the responsibilities of the two enhancement networks to ensure that the same-resolution enhancement network focuses on image quality restoration, while the super-resolution enhancement network mainly processes resolution improvement, so as to avoid image quality distortion caused by the superposition of the two. This design can provide consistent and excellent image quality performance in different application scenarios, and significantly improve the usability and scalability of the system.

[0064] In an alternative embodiment, as Figure 5 shown, the super-resolution enhancement network is obtained through the following steps: Step S500, obtain a third training set, the third training set includes a plurality of third training pairs, each third training pair includes a third sample frame and a third reference frame, the third sample frame is obtained by reducing the resolution of the third reference frame and degrading it by a third degree, and the third degree is less than the first degree.

[0065] Step S502, train the residual local feature network based on the third training set to obtain the super-resolution enhancement network.

[0066] Exemplarily, a third training set is obtained in advance, and the third training set includes a plurality of third training pairs. Each third training pair includes a third sample frame and a third reference frame. Among them, the third sample frame can be obtained by reducing the resolution of the third reference frame and degrading it by a third degree (slight), and the third degree is less than the first degree. Train the cropped and optimized lightweight RLFN (Residual Local Feature Network) architecture based on the third training set. During the training process, the network needs to achieve two goals at the same time. One is to effectively improve the image resolution and restore clear and sharp edge details in the low-resolution image. The other is to suppress common problems caused by resolution improvement, such as the jagged effect and over-sharpening. To achieve this goal, the network is guided to learn slight denoising and texture enhancement during the training process to ensure that the overall naturalness of the image is not affected after the resolution is improved.

[0067] In this embodiment, a lightweight super-resolution enhancement network can be obtained by selecting an appropriate training set and architecture design for training, and efficient resolution improvement can be achieved. The super-resolution enhancement network focuses on resolution improvement and detail optimization, which can avoid functional overlap with the same-resolution enhancement network. This functional separation ensures the optimization effect of the system.

[0068] In some embodiments, Pixel Shuffle and Pixel unshuffle can also be introduced at the head of the RLFN network, which can reduce the size of the feature map and significantly reduce the amount of computation, thus ensuring that the super-resolution task can still be efficiently executed on hardware devices with limited resources.

[0069] The modular network design of the embodiments of the present application also helps with the expansion and upgrade of the system. When the system needs to be updated, only the enhancement network needs to be adjusted to synchronously optimize the same-resolution enhancement network and the super-resolution enhancement network while keeping the overall system architecture unchanged. This design significantly reduces the system maintenance cost and improves the flexibility of the system in the face of future requirements.

[0070] To make the present application easier to understand, the following provides Figure 6 an exemplary application.

[0071] S1: Decode the live video stream into single-frame images to obtain the original video frame Frame (the color space can be YUV), and input it into bvt (resource multimedia algorithm sdk library). bvt also receives the service status flag.

[0072] S2: bvt determines to start the same-resolution enhancement network and / or the super-resolution enhancement network according to the image quality of the original video frame and the service status flag, and processes the original video frame in real time to obtain the enhanced video frame.

[0073] In this exemplary application: Through a modular-designed video enhancement algorithm, different image quality improvement requirements in the same service scenario can be met, efficient video image quality optimization can be achieved, and the computing power and bandwidth consumption can be reduced.

[0074] Embodiment 2 Figure 7 Schematically shows a block diagram of a video enhancement device according to Embodiment 2 of the present application. The device can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. As Figure 7 shown, the device 1000 may include: a first acquisition module 1100 and a second acquisition module 1200, where: The first acquisition module 1100 is configured to acquire an original video frame and a service status flag, and the service status flag is used to represent a target service scenario; A second acquisition module 1200, configured to determine a target enhancement network from two pre-trained enhancement networks according to the target service scenario, and convert the original video frame into the enhanced video frame through the target enhancement network; Wherein, the enhancement network and its combination respectively correspond to different service scenarios, and the target enhancement network includes at least one of the enhancement networks.

[0075] As an optional embodiment, the service scenario includes a business peak period and a business stable period, the service status flag includes a first service status flag and a second service status flag, the first service status flag corresponds to the business peak period, the second service status identifier corresponds to the business stable period, and the two enhancement networks include a same-resolution enhancement network and a super-resolution enhancement network; Correspondingly, determining a target enhancement network from two pre-trained enhancement networks according to the target service scenario includes: When the service status flag is the first service status flag, determining the same-resolution enhancement network and the super-resolution enhancement network as the target enhancement network; When the service status flag is the second service status flag, determining the same-resolution enhancement network as the target enhancement network.

[0076] As an optional embodiment, converting the original video frame into the enhanced video frame through the target enhancement network includes: When the target enhancement network is the same-resolution enhancement network and the super-resolution enhancement network: performing denoising processing and texture reconstruction on the original video frame through the same-resolution enhancement network to obtain an intermediate video frame; performing super-resolution processing on the intermediate video frame through the super-resolution enhancement network to obtain the enhanced video frame; When the target enhancement network is the same-resolution enhancement network, performing denoising processing and texture reconstruction on the original video frame through the same-resolution enhancement network to obtain the enhanced video frame.

[0077] As an optional embodiment, the service scenario further includes a super-resolution service, the service status flag further includes a third service status flag, and the third service status flag corresponds to the super-resolution service; correspondingly, determining a target enhancement network from two pre-trained enhancement networks according to the target service scenario includes: When the service status flag is the third service status flag, determining the super-resolution enhancement network as the target enhancement network.

[0078] As an alternative embodiment, converting the original video frame into the enhanced video frame through the target enhancement network includes: When the target enhancement network is the super-resolution enhancement network, performing super-resolution processing on the original video frame through the super-resolution enhancement network to obtain the enhanced video frame. As an alternative embodiment, the same-resolution enhancement network is obtained through the following operations: Obtain a first training set, where the first training set includes a plurality of first training pairs, and each first training pair includes a first sample frame and a first reference frame. The first sample frame is obtained by degrading the first reference frame by a first degree, and the degradation process includes adding one or more of blur, Gaussian noise, and coding noise; Train a fully convolutional neural network based on the first training set to obtain a basic network; Obtain a second training set, where the second training set includes a plurality of second training pairs, and each second training pair includes a second sample frame and a second reference frame. The second sample frame is obtained by degrading the second reference frame by a second degree, and the second degree is greater than the first degree; Train the basic network based on the second training set and the generative adversarial network mechanism to obtain the same-resolution enhancement network.

[0079] As an alternative embodiment, the super-resolution enhancement network is obtained through the following operations: Obtain a third training set, where the third training set includes a plurality of third training pairs, and each third training pair includes a third sample frame and a third reference frame. The third sample frame is obtained by reducing the resolution of the third reference frame and degrading it by a third degree, and the third degree is less than the first degree; Train a residual local feature network based on the third training set to obtain the super-resolution enhancement network.

[0080] Embodiment III Figure 8 Schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing the video enhancement method according to Embodiment III of the present application. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers). As Figure 8As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can be communicatively linked to each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium. The readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disc, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device 10000. Of course, the memory 10010 can also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed on the computer device 10000, such as the program code of the video enhancement method. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0081] In some embodiments, the processor 10020 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0082] The network interface 10030 may include a wireless network interface or a wired network interface, which is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an enterprise intranet (Intranet), the Internet, Global System of Mobile communication (GSM for short), Wideband Code Division Multiple Access (WCDMA for short), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0083] It should be noted that Figure 8 Only the computer device with components 10010 - 10030 is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0084] In this embodiment, the video enhancement method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as the processor 10020) to complete the embodiments of the present application.

[0085] Embodiment 4 The embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the video enhancement method in the embodiments are implemented.

[0086] In this embodiment, the computer-readable storage medium includes flash memory, hard disks, multimedia cards, card-type memories (such as SD or DX memories, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memories, magnetic disks, optical discs, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., equipped on the computer device. Of course, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is generally used to store the operating system and various application software installed on the computer device, such as the program code of the video enhancement method in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various data that have been output or will be output.

[0087] Embodiment 5 The embodiment of the present application further provides a computer program product, including a computer program, which when executed by a processor implements the method in the above embodiment.

[0088] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the embodiments of the present application can be implemented by a general computer device. They can be concentrated on a single computer device or distributed on a network composed of multiple computer devices. Optionally, they can be implemented by program codes executable by the computer device. Thus, they can be stored in a storage device and executed by the computer device. And in some cases, the steps shown or described can be executed in a different order from here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to implement. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0089] It should be noted that the above are only the preferred embodiments of the present application, and do not limit the patent protection scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present application.

Claims

1. A video enhancement method, characterized in that, The method includes: Obtaining an original video frame and a service status flag, where the service status flag is used to represent a target service scenario; Determining a target enhancement network from two pre-trained enhancement networks according to the target service scenario, and converting the original video frame into the enhanced video frame through the target enhancement network; Wherein, the enhancement network and its combination respectively correspond to different service scenarios, and the target enhancement network includes at least one of the enhancement networks.

2. The method according to claim 1, wherein The service scenarios include a peak service period and a stable service period. The service status flags include a first service status flag and a second service status flag. The first service status flag corresponds to the peak service period, and the second service status flag corresponds to the stable service period. The two enhancement networks include a same-resolution enhancement network and a super-resolution enhancement network; Correspondingly, determining a target enhancement network from two pre-trained enhancement networks according to the target service scenario includes: When the service status flag is the first service status flag, determining the same-resolution enhancement network and the super-resolution enhancement network as the target enhancement network; When the service status flag is the second service status flag, determining the same-resolution enhancement network as the target enhancement network.

3. The method according to claim 2, characterized in that, Converting the original video frame into the enhanced video frame through the target enhancement network includes: When the target enhancement network is the same-resolution enhancement network and the super-resolution enhancement network: performing denoising processing and texture reconstruction on the original video frame through the same-resolution enhancement network to obtain an intermediate video frame; performing super-resolution processing on the intermediate video frame through the super-resolution enhancement network to obtain the enhanced video frame; When the target enhancement network is the same-resolution enhancement network, performing denoising processing and texture reconstruction on the original video frame through the same-resolution enhancement network to obtain the enhanced video frame.

4. The method according to claim 2, wherein The service scenario further includes a super-resolution service, and the service status flag further includes a third service status flag, and the third service status flag corresponds to the super-resolution service; Correspondingly, determining a target enhancement network from two pre-trained enhancement networks according to the target service scenario includes: When the service status flag is the third service status flag, determining the super-resolution enhancement network as the target enhancement network.

5. The method according to claim 4, characterized in that, Converting the original video frame into the enhanced video frame through the target enhancement network includes: When the target enhancement network is the super-resolution enhancement network, performing super-resolution processing on the original video frame through the super-resolution enhancement network to obtain the enhanced video frame.

6. The method according to claim 2, characterized in that, The same-resolution enhancement network is obtained through the following operations: Obtaining a first training set, where the first training set includes a plurality of first training pairs, and each first training pair includes a first sample frame and a first reference frame. The first sample frame is obtained by degrading the first reference frame by a first degree, and the degradation processing includes adding one or more of blur, Gaussian noise, and coding noise; Train a fully convolutional neural network based on the first training set to obtain a basic network; Obtain a second training set, where the second training set includes a plurality of second training pairs, each second training pair includes a second sample frame and a second reference frame, and the second sample frame is obtained by degrading the second reference frame by a second degree, and the second degree is greater than the first degree; Train the basic network based on the second training set and a generative adversarial network mechanism to obtain the same-resolution enhancement network.

7. The method according to claim 6, wherein The super-resolution enhancement network is obtained through the following operations: Obtain a third training set, where the third training set includes a plurality of third training pairs, each third training pair includes a third sample frame and a third reference frame, and the third sample frame is obtained by reducing the resolution of the third reference frame and degrading it by a third degree, and the third degree is less than the first degree; Train a residual local feature network based on the third training set to obtain the super-resolution enhancement network.

8. A video enhancement device, characterized in that, The device includes: A first acquisition module, configured to acquire an original video frame and a service status flag, where the service status flag is used to represent a target service scenario; A second acquisition module, configured to determine a target enhancement network from two pre-trained enhancement networks according to the target service scenario, and convert the original video frame into the enhanced video frame through the target enhancement network; Wherein, the enhancement network and its combination respectively correspond to different service scenarios, and the target enhancement network includes at least one of the enhancement networks.

9. A computer device, characterized in that, Includes: At least one processor; And A memory communicatively connected to the at least one processor; wherein: The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.

Images