Method and device for controlling image quality enhancement

By monitoring the changes in video resolution and dynamically adjusting the image quality enhancement parameters and methods, the lag caused by the large amount of video quality enhancement calculations at high resolution is solved, and the smoothness and image quality enhancement effect is improved.

CN118945425AActive Publication Date: 2024-11-12HUNAN HAPPLY SUNSHINE INTERACTIVE ENTERTAINMENT MEDIA CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202410976939.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-12
Estimated Expiration
2044-07-19

AI Technical Summary

Technical Problem

The existing image processing technology has a large amount of calculation when the video quality is enhanced, resulting in delays in video at high resolution, unable to make full use of terminal performance, and poor image quality enhancement effect.

Method used

By monitoring the output resolution changes of the target image relative to the previous frame image, dynamically adjust the image quality enhancement parameters, and select different image sub-regions to use different image quality enhancement methods (such as super-segment models and USMs) to ensure image quality enhancement consumption Do not exceed the target threshold to avoid lag.

Benefits of technology

It effectively improves the smoothness of image output, ensures the usability of video quality enhancement function in high resolution, avoids lag problems caused by excessive time-consuming, and optimizes the performance utilization of terminal devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945425A_ABST
    Figure CN118945425A_ABST
Patent Text Reader

Abstract

The invention discloses an image quality enhancement control method and device. The method comprises the following steps: monitoring whether the output resolution of a target image relative to a current image is changed or not; the current image is a previous frame image of the target image; in response to the change of the output resolution, obtaining an image quality enhancement parameter of the target image; according to the image quality enhancement parameter of the target image, image quality enhancement is performed on the target image, and the image quality enhancement parameter of the target image enables time consumed by image quality enhancement of the target image to be smaller than or equal to a target threshold value; wherein the image quality enhancement parameter comprises an image quality enhancement mode used by each image sub-region contained in the target image, the image quality enhancement mode comprises a first enhancement mode or a second enhancement mode, and the image quality enhancement time consumption of the first enhancement mode is greater than the image quality enhancement time consumption of the second enhancement mode; the image quality enhancement effect of the first enhancement mode is better than that of the second enhancement mode.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a control method and device for image quality enhancement. Background Art

[0002] When the player plays a video, the player can enhance the image quality when the video image is decoded, so that you can experience the viewing effect of ultra-high-definition video with a low-definition video stream.

[0003] Currently, machine learning models are commonly used to achieve image quality enhancement. However, the computational complexity of this image quality enhancement solution is relatively large. If the resolution of the video itself is high, video freezes may occur due to the larger computational complexity of image quality enhancement. Summary of the invention

[0004] In view of the above problems, the present application provides a control method and device for image quality enhancement to avoid image output jamming, thereby improving the smoothness of image output. The specific solution is as follows:

[0005] A first aspect of the present application provides a control method for image quality enhancement, the method comprising:

[0006] Monitoring whether the output resolution of a target image relative to a current image changes; the current image is a previous frame image of the target image;

[0007] In response to the output resolution changing, obtaining image quality enhancement parameters of the target image;

[0008] According to the image quality enhancement parameter of the target image, the image quality enhancement parameter of the target image makes the time consumption of image quality enhancement of the target image less than or equal to the target threshold;

[0009] Among them, the image quality enhancement parameters include the image quality enhancement method used for each image sub-region contained in the target image, the image quality enhancement method includes a first enhancement method or a second enhancement method, the image quality enhancement time of the first enhancement method is greater than the image quality enhancement time of the second enhancement method, and the image quality enhancement effect of the first enhancement method is better than the image quality enhancement effect of the second enhancement method.

[0010] In a possible implementation, obtaining the image quality enhancement parameter of the target image includes:

[0011] Determining whether image quality enhancement is performed on the current image;

[0012] In the case where image quality enhancement is performed on the current image, obtaining image quality enhancement parameters of the target image in a first acquisition manner;

[0013] In a case where image quality enhancement is not performed on the current image, obtaining image quality enhancement parameters of the target image in a second acquisition manner;

[0014] The first acquisition method and the second acquisition method are different.

[0015] In a possible implementation, obtaining the image quality enhancement parameter of the target image in a first acquisition manner includes:

[0016] According to the size relationship between the output resolution of the target image and the output resolution of the current image, the image quality enhancement parameters of the current image are adjusted to obtain the image quality enhancement parameters of the target image.

[0017] In a possible implementation, adjusting the image quality enhancement parameter of the current image according to the size relationship between the output resolution of the target image and the output resolution of the current image to obtain the image quality enhancement parameter of the target image includes:

[0018] Determining whether the output resolution of the target image is greater than the output resolution of the current image;

[0019] If the output resolution of the target image is greater than the output resolution of the current image, at least one first sub-region of the image quality enhancement parameter of the current image is adjusted from the first enhancement mode to the second enhancement mode in sequence until a candidate enhancement parameter that meets the preferred condition is obtained;

[0020] If the output resolution of the target image is smaller than the output resolution of the current image, at least one second sub-region of the image quality enhancement parameter of the current image is adjusted from the second enhancement mode to the first enhancement mode in sequence until a candidate enhancement parameter that meets the preferred condition is obtained;

[0021] The candidate enhancement parameters that meet the preferred conditions are determined as the image quality enhancement parameters of the target image.

[0022] In a possible implementation, the area of ​​the first sub-region adjusted last time is smaller than the area of ​​the first sub-region adjusted next time;

[0023] The area of ​​the second sub-region adjusted last time is larger than the area of ​​the second sub-region adjusted later.

[0024] In a possible implementation, obtaining the image quality enhancement parameter of the target image in the second acquisition mode includes:

[0025] When the output resolution of the target image is less than the output resolution of the current image, obtaining an enhancement judgment result, wherein the enhancement judgment result indicates whether to perform image quality enhancement on the target image;

[0026] If the enhancement judgment result indicates that image quality enhancement is performed on the target image, adjusting the initial enhancement parameters to obtain image quality enhancement parameters of the target image;

[0027] Wherein, each of the image sub-regions in the initial enhancement parameters uses the second enhancement method.

[0028] In a possible implementation, adjusting the initial enhancement parameters to obtain the image quality enhancement parameters of the target image includes:

[0029] sequentially adjusting at least one third sub-region in the initial enhancement parameters from the second enhancement mode to the first enhancement mode until a candidate enhancement parameter that meets the preferred condition is obtained;

[0030] The candidate enhancement parameters that meet the preferred conditions are determined as the image quality enhancement parameters of the target image.

[0031] In a possible implementation, the area of ​​the third sub-region adjusted last time is smaller than the area of ​​the third sub-region adjusted later.

[0032] In one possible implementation, the preferred condition includes: the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image less than or equal to the target threshold, and the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image the largest among all the selected enhancement parameters.

[0033] A second aspect of the present application provides a control device for image quality enhancement, comprising:

[0034] A resolution monitoring unit, used to monitor whether the output resolution of a target image changes relative to a current image; the current image is a previous frame image of the target image;

[0035] A parameter obtaining unit, configured to obtain a quality enhancement parameter of the target image in response to a change in the output resolution;

[0036] an enhancement execution unit, configured to perform image quality enhancement on the target image according to an image quality enhancement parameter of the target image, wherein the image quality enhancement parameter of the target image makes the time consumed for performing image quality enhancement on the target image less than or equal to a target threshold;

[0037] Among them, the image quality enhancement parameters include the image quality enhancement method used for each image sub-region contained in the target image, the image quality enhancement method includes a first enhancement method or a second enhancement method, the image quality enhancement time of the first enhancement method is greater than the image quality enhancement time of the second enhancement method, and the image quality enhancement effect of the first enhancement method is better than the image quality enhancement effect of the second enhancement method.

[0038] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the control method for image quality enhancement according to the first aspect or any implementation method of the first aspect.

[0039] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0040] The memory is used to store computer programs;

[0041] The processor is used to execute the computer program so that the electronic device can implement the control method for image quality enhancement according to the first aspect or any implementation manner of the first aspect.

[0042] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can realize a control method for enhancing the image quality of the above-mentioned first aspect or any implementation of the first aspect.

[0043] By means of the above technical solution, in a control method and device for image quality enhancement provided by the present application, when it is detected that the output resolution of the target image relative to the previous frame image changes, the quality enhancement parameters of the target image are obtained, and the quality enhancement parameters include the quality enhancement methods applicable to each image sub-region contained in the target image, such as the first enhancement method and the second enhancement method with different quality enhancement time and quality enhancement effect, and the quality enhancement parameters make the time consumed for quality enhancement of the target image less than or equal to the target threshold. Therefore, in the present application, in response to the change of the output resolution, the quality enhancement parameters that can make the time consumed for quality enhancement of the target image not exceed the target threshold are obtained, so that as long as the output resolution changes, the quality enhancement parameters that consume time not exceeding the target threshold are used to enhance the image quality, thereby avoiding the situation where the image output is stuck due to the time consumed for quality enhancement exceeding the target threshold, thereby improving the smoothness of image output. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flowchart of a control method for image quality enhancement provided in an embodiment of the present application;

[0046] Figure 2 A partial flow chart of a control method for image quality enhancement provided in an embodiment of the present application;

[0047] Figure 3 This is an example diagram of the image quality enhancement control of the player in the embodiment of the present application;

[0048] Figure 4 Another partial flow chart of a method for controlling image quality enhancement provided by an embodiment of the present application;

[0049] Figure 5 Another partial flow chart of a method for controlling image quality enhancement provided by an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of a control device for image quality enhancement provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0052] Figure 8 This is an overall flow chart for realizing image quality enhancement in the scenario where the mobile phone player plays videos applicable to this application;

[0053] Fig. 9 This is a flowchart for performing image quality enhancement in a scenario where a mobile phone player plays a video applicable to this application;

[0054] Fig.10 This application is applicable to the flowchart of updating the estimated value of picture quality enhancement feedback in the scenario of playing videos on a mobile player;

[0055] Fig.11 This is a flowchart of the total time consumption for estimating the quality enhancement of the current frame in the scenario where the mobile player plays the video.

[0056] Fig.12 This application is applicable to the flowchart of readjusting the optimal image quality enhancement strategy logic in the scenario where a mobile phone player plays videos. DETAILED DESCRIPTION

[0057] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0058] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0059] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0060] The present application can be applied to the field of image processing. The technical solution of the present application is introduced below by taking the scenario in which a terminal outputs a video through a player as an example.

[0061] First, when the player is playing a video, if the local network of the user terminal is poor or the user chooses to play the video in standard definition or ultra-fast definition to save traffic or the video source is not clear, the terminal can enhance the image quality through the player when the video picture is decoded, so that the user can experience the viewing effect of ultra-high-definition video with a low-definition video stream.

[0062] However, the super-resolution model (machine learning model) used for image quality enhancement has a relatively large amount of computation. When image quality is enhanced in feature areas (portrait explicit features, subtitles, station logos (TV station logos), etc.), the image quality enhancement effect is not optimal due to large differences in terminal performance. Moreover, the computational complexity of the super-resolution model will increase significantly with the area as the resolution increases, making it difficult to fully utilize terminal performance at multiple resolutions. Alternatively, the image quality enhancement function at high resolution has to be turned off, or the image quality enhancement at high resolution will cause the video output to freeze.

[0063] In order to solve the above problems, the embodiment of the present application provides a control method for enhancing image quality. The control method for enhancing image quality of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0064] Reference Figure 1, is a flow chart of an implementation of a control method for image quality enhancement provided in an embodiment of the present application, which can be applied to electronic devices capable of image processing, such as mobile phones and tablet computers with players. The technical solution in this embodiment is mainly used to avoid the situation where the image output is stuck due to the time consumed for image quality enhancement exceeding the target threshold, thereby improving the smoothness of image output.

[0065] Specifically, the method in this embodiment may include the following steps:

[0066] Step 101: monitor whether the output resolution of the target image relative to the current image has changed. If the output resolution has not changed, execute step 102; if the output resolution has changed, execute step 103.

[0067] The current image is the previous frame of the target image. For example, taking the video player in the mobile phone outputting a variety video as an example, the current image is the image being output in the variety video, and the target image is the image to be rendered and output in the next frame of the current image in the variety video.

[0068] Step 102: Using the image quality enhancement parameters of the current image as the image quality enhancement parameters of the target image.

[0069] That is to say, if the image quality of each frame is enhanced during the video output process, if the output resolution remains unchanged, then the image quality enhancement parameters used for the previous frame will continue to be used to enhance the subsequent images. If the output resolution changes, then the image quality enhancement parameters for the subsequent images need to be re-determined.

[0070] Step 103: Obtain image quality enhancement parameters of the target image.

[0071] The image quality enhancement parameter of the target image makes the time consumed for image quality enhancement of the target image less than or equal to the target threshold.

[0072] It should be noted that if the image quality enhancement parameter whose time consumption for image quality enhancement is less than or equal to the target threshold is not obtained, then the image quality enhancement function for the target image can be turned off, for example, the image quality enhancement function of the video player can be turned off.

[0073] Step 104: enhancing the image quality of the target image according to the image quality enhancement parameters of the target image.

[0074] The target image includes multiple image sub-regions and remaining sub-regions, each image sub-region is a sub-region that needs to be enhanced, and the remaining sub-regions are sub-regions that do not need to be enhanced. Based on this, the image quality enhancement parameters include: the image quality enhancement method used by each image sub-region included in the target image. Different image quality enhancement parameters have different image quality enhancement methods used by one or more image sub-regions.

[0075] It should be noted that the image quality enhancement method includes the first enhancement method or the second enhancement method. The image quality enhancement time of the first enhancement method is greater than the image quality enhancement time of the second enhancement method, and the image quality enhancement effect of the first enhancement method is better than the image quality enhancement effect of the second enhancement method.

[0076] For example, the first enhancement method is to use a super-resolution model to enhance the image quality, and the second enhancement method is to use USM (Unsharp Masking) to enhance the image quality. Although the USM method is less time-consuming, the image quality enhancement effect is not as good as the super-resolution model method. Taking the video player in a mobile phone outputting a variety show video as an example, the video screen in the variety show video contains a portrait explicit feature sub-area, a subtitle sub-area, a station logo sub-area, and a remaining sub-area. Each sub-area uses a corresponding image quality enhancement method for image quality enhancement.

[0077] It should be noted that the time consumed for image quality enhancement of the target image can also be referred to as the image quality enhancement time of the target image, and the time consumed for image quality enhancement of the target image at least includes: the total time consumed for image quality enhancement of each image sub-region in the target image, or the time consumed for image quality enhancement of the target image can be: the maximum value of the time consumed for image quality enhancement of each image sub-region in the target image. The time consumed for image quality enhancement of an image sub-region can be obtained by multiplying the unit statistical value of the time consumed of the image sub-region by the area of ​​the image sub-region. The unit statistical value of the time consumed of an image sub-region refers to: the statistical time consumed for image quality enhancement of the image sub-region within a unit area. The unit statistical value of the time consumed of an image sub-region can be: the initial unit statistical value, or the unit statistical value of the time consumed of an image sub-region can be: the average value of the historical unit time consumed for image quality enhancement of an image region of the same area type as the image sub-region within a unit area, and the historical unit time consumed refers to the historical time consumed for image quality enhancement of an image region of the same area type as the image sub-region within a unit area in historical image quality enhancement.

[0078] Among them, different image quality enhancement methods have different time-consuming unit statistics for image quality enhancement of image sub-regions. For example, the time-consuming unit statistics for image quality enhancement of a certain image sub-region using the first enhancement method is greater than the time-consuming unit statistics for image quality enhancement of the same image sub-region using the second enhancement method, but the image quality enhancement effect of image quality enhancement of the certain image sub-region using the first enhancement method is better than the image quality enhancement effect of image quality enhancement of the same image sub-region using the second enhancement method.

[0079] According to the above technical solution, in a control method for image quality enhancement provided by an embodiment of the present application, when it is monitored that the output resolution of the target image relative to the previous frame image changes, the quality enhancement parameter of the target image is obtained, and the quality enhancement parameter includes the quality enhancement mode applicable to each image sub-region contained in the target image, such as the first enhancement mode and the second enhancement mode with different quality enhancement time and quality enhancement effect, and the quality enhancement parameter makes the time consumed for quality enhancement of the target image less than or equal to the target threshold. Therefore, in this embodiment, in response to the change of the output resolution, the quality enhancement parameter that can make the time consumed for quality enhancement of the target image not exceed the target threshold is obtained, so that as long as the output resolution changes, the quality enhancement parameter that consumes less than the target threshold is used to enhance the image quality, thereby avoiding the situation where the image output is stuck due to the time consumed for quality enhancement exceeding the target threshold, thereby improving the smoothness of image output.

[0080] In one implementation, when obtaining the image quality enhancement parameters of the target image in step 103, it can be implemented in the following manner: Figure 2 As shown in:

[0081] Step 201: Determine whether the current image performs image quality enhancement. If the current image performs image quality enhancement, execute step 202; if the current image does not perform image quality enhancement, execute step 203.

[0082] Specifically, in this embodiment, determining whether the current image performs image quality enhancement may include determining whether a player outputting the current image has an image quality enhancement function enabled.

[0083] For example, taking a mobile phone player as an example, the player has image quality enhancement controls, such as Figure 3 As shown in , when the image quality enhancement control is adjusted to the "on" state, image quality enhancement is performed on the current image, and when the image quality enhancement control is adjusted to the "off" state, image quality enhancement is not performed on the current image.

[0084] Step 202: Obtain image quality enhancement parameters of a target image in a first acquisition manner.

[0085] Step 203: Obtain image quality enhancement parameters of the target image in a second acquisition manner.

[0086] The first acquisition method is different from the second acquisition method.

[0087] For example, the first acquisition method is a method of obtaining the image quality enhancement parameters of the target image according to the image quality enhancement parameters of the current image, and the first acquisition method is different from the second acquisition method.

[0088] In one implementation, the first acquisition method in step 202 is: according to the size relationship between the output resolution of the target image and the output resolution of the current image, the image quality enhancement parameters of the current image are adjusted to obtain the image quality enhancement parameters of the target image.

[0089] Wherein, when the output resolution of the target image and the output resolution of the current image have different size relationships, the image quality enhancement parameters of the current image are adjusted in different ways to obtain the image quality enhancement parameters of the target image.

[0090] Specifically, step 202 can be implemented in the following manner: Figure 4 As shown in:

[0091] Step 401: Determine whether the output resolution of the target image is greater than the output resolution of the current image. If the output resolution of the target image is greater than the output resolution of the current image, execute step 402; if the output resolution of the target image is less than the output resolution of the current image, execute step 403.

[0092] Step 402: sequentially adjusting at least one first sub-region of the image quality enhancement parameters of the current image from the first enhancement mode to the second enhancement mode, until a candidate enhancement parameter satisfying the preferred condition is obtained.

[0093] Among them, when the output resolution becomes larger, if the target image quality enhancement is continued with the image quality enhancement parameters of the current image, the time consumed for image quality enhancement of the target image may exceed the target threshold. Therefore, the first sub-area in the image quality enhancement parameters of the current image can be adjusted from the first enhancement method to the second enhancement method as a candidate enhancement parameter to reduce the time consumed for image quality enhancement.

[0094] Specifically, in step 402, the image quality enhancement parameters of the current image can be used to first calculate the image quality enhancement time of the target image. If the image quality enhancement time of the target image is less than or equal to the target threshold, the image quality enhancement parameters of the current image are used as the image quality enhancement parameters of the target image. If the image quality enhancement time of the target image is greater than the target threshold, then at least one first sub-region in the image quality enhancement parameters of the current image is adjusted from the first enhancement mode to the second enhancement mode in sequence until the candidate enhancement parameters that meet the preferred conditions are obtained. If the preferred conditions are still not met when all image sub-regions in the image quality enhancement parameters of the current image are set to the second enhancement mode, then there are no candidate enhancement parameters that meet the preferred conditions.

[0095] It should be noted that the first sub-region is adjusted from the first enhancement mode to the second enhancement mode, which can reduce the time consumption of image quality enhancement by sacrificing the image quality enhancement effect.

[0096] Among them, the preferred conditions may include: the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image less than or equal to the target threshold, and the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image the largest among all the candidate enhancement parameters.

[0097] Furthermore, the preferred condition may also include: the image quality enhancement parameter makes the area of ​​the sub-region in the target image that is enhanced in the first enhancement manner the largest among all the enhancement parameters to be selected.

[0098] Based on this, in this embodiment, based on the image quality enhancement parameters of the current image, the image quality enhancement parameters with the best image quality enhancement effect, the longest image quality enhancement time but less than or equal to the target threshold are obtained as the candidate enhancement parameters that meet the preferred conditions.

[0099] In one implementation, in step 402, the area of ​​the first sub-region adjusted last time is smaller than the area of ​​the first sub-region adjusted next time.

[0100] That is to say, in step 402, for the image quality enhancement parameters of the current image, first, starting from the image sub-region with the smallest area using the first enhancement method, its image quality enhancement method is adjusted to the second enhancement method with lower time consumption. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image less than or equal to the target threshold, then the adjusted image quality enhancement parameters are determined as the candidate enhancement parameters that meet the preferred conditions. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image greater than the target threshold, then only the image quality enhancement method of the image sub-region with the second smallest area using the first enhancement method is adjusted to the second enhancement method with lower time consumption. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image less than or equal to the target threshold, then the adjusted image quality enhancement parameters are determined as the candidate enhancement parameters that meet the preferred conditions. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image greater than the target threshold, then only the image quality enhancement method of the image sub-region with a larger area using the first enhancement method is adjusted to the second enhancement method with lower time consumption. By analogy, if only the image quality enhancement method of the image sub-region with the largest area using the first enhancement method is adjusted If the second enhancement method with lower time consumption still makes the image quality enhancement time of the target image greater than the target threshold, then starting from the image sub-region with the smallest area using the first enhancement method, the image quality enhancement methods of the two image sub-regions with the smallest area and the second smallest area using the first enhancement method are adjusted to the second enhancement method with lower time consumption. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image less than or equal to the target threshold, then the adjusted image quality enhancement parameters are determined as the candidate enhancement parameters that meet the preferred conditions. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement methods of the two image sub-regions with the smallest area and the larger area using the first enhancement method are adjusted to the second enhancement method with lower time consumption, and so on, until the adjusted image quality enhancement parameters make the image quality enhancement time of the target image less than or equal to the target threshold, at this time the adjusted image quality enhancement parameters are determined as the candidate enhancement parameters that meet the preferred conditions, or, until the image quality enhancement methods of all image sub-regions are adjusted to the second enhancement method, the image quality enhancement time of the target image cannot be less than or equal to the target threshold, then there are no candidate enhancement parameters that meet the preferred conditions.

[0101] Among them, the time consumption of image quality enhancement of the target image can be obtained in the following way: according to the image quality enhancement method used for each image sub-region in the current image, for each image sub-region in the target image, obtain the product of the area of ​​the image sub-region and the time consumption unit statistical value corresponding to the image quality enhancement method used for the image sub-region of the same region type in the current image, and then obtain the sum of the products to obtain the time consumption of image quality enhancement of the target image.

[0102] Step 403: sequentially adjusting at least one second sub-region of the image quality enhancement parameters of the current image from the second enhancement mode to the first enhancement mode, until a candidate enhancement parameter satisfying the preferred condition is obtained.

[0103] Among them, when the output resolution becomes smaller, if the target image quality enhancement is continued with the image quality enhancement parameters of the current image, the time consumed for image quality enhancement of the target image will be less than or equal to the target threshold, but the image quality enhancement effect may be poor. Therefore, the second sub-area in the image quality enhancement parameters of the current image can be adjusted from the second enhancement mode to the first enhancement mode as a candidate enhancement parameter to optimize the image quality enhancement effect.

[0104] Specifically, in step 403, the image quality enhancement parameters of the current image can be used to first calculate the image quality enhancement time of the target image. If the image quality enhancement time of the target image is less than or equal to the target threshold, then at least one second sub-region in the image quality enhancement parameters of the current image is adjusted from the second enhancement mode to the first enhancement mode in sequence until the selected enhancement parameters that meet the preferred conditions are obtained. Among them, if the preferred conditions are still met when all image sub-regions in the image quality enhancement parameters of the current image are in the first enhancement mode, then the image quality enhancement parameters in which the image quality enhancement modes of all image sub-regions are in the first enhancement mode are used as the selected enhancement parameters.

[0105] It should be noted that the second sub-region is adjusted from the second enhancement mode to the first enhancement mode, and the image quality enhancement effect can be optimized by sacrificing image quality and increasing time consumption.

[0106] Among them, the preferred conditions may include: the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image less than or equal to the target threshold, and the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image the largest among all the candidate enhancement parameters.

[0107] Furthermore, the preferred condition may also include: the image quality enhancement parameter makes the area of ​​the sub-region in the target image that is enhanced in the first enhancement manner the largest among all the enhancement parameters to be selected.

[0108] Based on this, in this embodiment, based on the image quality enhancement parameters of the current image, the image quality enhancement parameters with the best image quality enhancement effect, the longest image quality enhancement time but less than or equal to the target threshold are obtained as the candidate enhancement parameters that meet the preferred conditions.

[0109] In one implementation, in step 403, the area of ​​the second sub-region adjusted last time is larger than the area of ​​the second sub-region adjusted last time.

[0110] That is to say, in step 403, for the image quality enhancement parameters of the current image, first starting from the image sub-region with the largest area using the second enhancement method, its image quality enhancement method is adjusted to the first enhancement method with better effect. If the image quality enhancement parameters after the adjustment make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement parameters before the adjustment are determined as the candidate enhancement parameters that meet the preferred conditions. If the image quality enhancement time of the target image after the adjustment is still less than or equal to the target threshold, then the image quality enhancement method of the image sub-region with the second largest area using the second enhancement method is continued to be adjusted to the first enhancement method with better effect. If the image quality enhancement parameters after the adjustment make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement parameters before the adjustment are determined as the candidate enhancement parameters that meet the preferred conditions. For the candidate enhancement parameters that meet the preferred conditions, if the image quality enhancement parameters after adjustment make the image quality enhancement time of the target image still less than or equal to the target threshold, then continue to adjust the image quality enhancement method of the smaller image sub-region using the second enhancement method to the first enhancement method with better effect, and so on, until the adjusted image quality enhancement parameters make the image quality enhancement time of the target image greater than the target threshold, at this time the image quality enhancement parameters before adjustment are determined as the candidate enhancement parameters that meet the preferred conditions, or, until the image quality enhancement methods of all image sub-regions are adjusted to the first enhancement method and none of them can make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement parameters whose image quality enhancement methods of all image sub-regions are adjusted to the first enhancement method are determined as the candidate enhancement parameters that meet the preferred conditions.

[0111] Step 404: taking the candidate enhancement parameters that meet the optimal conditions as the image quality enhancement parameters of the target image.

[0112] It should be noted that if there is no candidate enhancement parameter that meets the preferred condition, the image quality enhancement function for the target image can be turned off.

[0113] In one implementation, the second acquisition method in step 203 may be: obtaining the image quality enhancement parameters of the target image according to the initial enhancement parameters.

[0114] Specifically, in step 203, it can be first determined whether the output resolution of the target image is greater than the output resolution of the current image, wherein:

[0115] In the case where the output resolution of the target image is greater than the output resolution of the current image, the target image is not enhanced. That is to say, in the case where the current image is not enhanced, if the output resolution of the target image is greater than the output resolution of the current image, it means that when outputting a target image with a higher output resolution, if image enhancement is performed, more device resources will be consumed, and image freeze will certainly occur. Therefore, the target image is not enhanced at this time.

[0116] In the case where the output resolution of the target image is less than the output resolution of the current image, an enhancement judgment result is first obtained, and the enhancement judgment result indicates whether to perform image quality enhancement on the target image. For example, the initial enhancement parameter can be used to calculate the time consumption of image quality enhancement of the target image. If the initial enhancement parameter makes the time consumption of image quality enhancement of the target image greater than the target threshold, then an enhancement judgment result indicating that image quality enhancement is not performed on the target image is obtained. If the initial enhancement parameter makes the time consumption of image quality enhancement of the target image less than or equal to the target threshold, then an enhancement judgment result indicating that image quality enhancement is performed on the target image can be obtained.

[0117] Based on this, if the enhancement judgment result indicates that image quality enhancement is performed on the target image, the initial enhancement parameters may be adjusted to obtain image quality enhancement parameters of the target image.

[0118] Wherein, each image sub-region in the initial enhancement parameters uses the second enhancement mode.

[0119] Specifically, in step 203, when the initial enhancement parameters are adjusted to obtain the image quality enhancement parameters of the target image, it can be achieved in the following manner: Figure 5 As shown in:

[0120] Step 501: sequentially adjust at least one third sub-region in the initial enhancement parameters from the second enhancement mode to the first enhancement mode, until a candidate enhancement parameter that meets the preferred condition is obtained.

[0121] Among them, if the target image is enhanced with the initial enhancement parameters, the time consumed for image quality enhancement of the target image will definitely be less than or equal to the target threshold, but the image quality enhancement effect is poor. Therefore, the third sub-area in the initial enhancement parameters can be adjusted from the second enhancement method to the first enhancement method as a candidate enhancement parameter to optimize the image quality enhancement effect.

[0122] Specifically, in step 501, the initial enhancement parameters may be used to first calculate the time consumption of the image quality enhancement of the target image. If the time consumption of the image quality enhancement of the target image is less than or equal to the target threshold, then at least one third sub-region in the initial enhancement parameters is adjusted from the second enhancement mode to the first enhancement mode in sequence until the selected enhancement parameters that meet the preferred conditions are obtained. If the preferred conditions are still met when all image sub-regions in the initial enhancement parameters are in the first enhancement mode, then the image quality enhancement parameters in which the image quality enhancement modes of all image sub-regions are in the first enhancement mode are used as the selected enhancement parameters.

[0123] It should be noted that the third sub-region is adjusted from the second enhancement mode to the first enhancement mode, and the image quality enhancement effect can be optimized by sacrificing the time consumption of image quality enhancement.

[0124] Among them, the preferred conditions may include: the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image less than or equal to the target threshold, and the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image the largest among all the candidate enhancement parameters.

[0125] Furthermore, the preferred condition may also include: the image quality enhancement parameter makes the area of ​​the sub-region in the target image that is enhanced in the first enhancement manner the largest among all the enhancement parameters to be selected.

[0126] Based on this, in this embodiment, based on the initial enhancement parameters, the image quality enhancement parameters with the best image quality enhancement effect and the longest image quality enhancement time but less than or equal to the target threshold are obtained as the candidate enhancement parameters that meet the preferred conditions.

[0127] In one implementation, in step 501, the area of ​​the third sub-region adjusted last time is smaller than the area of ​​the third sub-region adjusted next time.

[0128] Specifically, in step 501, for the initial enhancement parameters, first start from the image sub-region with the smallest area using the second enhancement method, and adjust its image quality enhancement method to the first enhancement method with better effect. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement parameters before adjustment are determined as the candidate enhancement parameters that meet the preferred conditions. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image still less than or equal to the target threshold, then continue to adjust the image quality enhancement method of the image sub-region with the second smallest area using the second enhancement method to the first enhancement method with better effect. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement parameters before adjustment are determined as the candidate enhancement parameters that meet the preferred conditions. The candidate enhancement parameters of the preferred condition, if the image quality enhancement parameters after adjustment make the image quality enhancement time of the target image still less than or equal to the target threshold, then continue to adjust the image quality enhancement method of the larger image sub-region using the second enhancement method to the first enhancement method with better effect, and so on, until the image quality enhancement parameters after adjustment make the image quality enhancement time of the target image greater than the target threshold, at this time the image quality enhancement parameters before adjustment are determined as the candidate enhancement parameters that meet the preferred condition, or, until the image quality enhancement methods of all image sub-regions are adjusted to the first enhancement method and none of them can make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement parameters of all image sub-regions adjusted to the first enhancement method are determined as the candidate enhancement parameters that meet the preferred condition.

[0129] In another implementation, in step 501, the area of ​​the third sub-region adjusted last time is greater than the area of ​​the third sub-region adjusted next time.

[0130] Specifically, in step 501, for the initial enhancement parameters, first start from the image sub-region with the largest area using the second enhancement method, and adjust its image quality enhancement method to the first enhancement method with better effect. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement parameters before adjustment are determined as the candidate enhancement parameters that meet the preferred conditions. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image still less than or equal to the target threshold, then continue to adjust the image quality enhancement method of the image sub-region with the second largest area using the second enhancement method to the first enhancement method with better effect. If the adjusted image quality enhancement parameters make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement parameters before adjustment are determined as the candidate enhancement parameters that meet the preferred conditions. The candidate enhancement parameters of the preferred condition, if the image quality enhancement parameters after adjustment make the image quality enhancement time of the target image still less than or equal to the target threshold, then continue to adjust the image quality enhancement method of the image sub-region with a smaller area using the second enhancement method to the first enhancement method with better effect, and so on, until the image quality enhancement parameters after adjustment make the image quality enhancement time of the target image greater than the target threshold, at this time the image quality enhancement parameters before adjustment are determined as the candidate enhancement parameters that meet the preferred condition, or, until the image quality enhancement methods of all image sub-regions are adjusted to the first enhancement method and none of them can make the image quality enhancement time of the target image greater than the target threshold, then the image quality enhancement parameters of all image sub-regions adjusted to the first enhancement method are determined as the candidate enhancement parameters that meet the preferred condition.

[0131] Step 502: Determine the candidate enhancement parameters that meet the optimal conditions as the image quality enhancement parameters of the target image.

[0132] A control method for image quality enhancement provided in an embodiment of the present application is introduced above, and a device for executing the control method for image quality enhancement will be introduced below.

[0133] See also Figure 6 , which is a schematic diagram of the structure of a control device for image quality enhancement provided in an embodiment of the present application. The device can be configured in an electronic device capable of image processing, such as a mobile phone with a player, a tablet computer, etc. The technical solution in this embodiment is mainly used to avoid the situation where the image output is stuck due to the time consumed for image quality enhancement exceeding the target threshold, thereby improving the smoothness of image output.

[0134] Specifically, the device in this embodiment may include the following units:

[0135] A resolution monitoring unit 601 is used to monitor whether the output resolution of a target image changes relative to a current image; the current image is a previous frame image of the target image;

[0136] A parameter obtaining unit 602, configured to obtain a quality enhancement parameter of the target image in response to a change in the output resolution;

[0137] An enhancement execution unit 603 is used to enhance the image quality of the target image according to the image quality enhancement parameter of the target image, wherein the image quality enhancement parameter of the target image makes the time consumption of enhancing the image quality of the target image less than or equal to the target threshold;

[0138] Among them, the image quality enhancement parameters include the image quality enhancement method used for each image sub-region contained in the target image, the image quality enhancement method includes a first enhancement method or a second enhancement method, the image quality enhancement time of the first enhancement method is greater than the image quality enhancement time of the second enhancement method, and the image quality enhancement effect of the first enhancement method is better than the image quality enhancement effect of the second enhancement method.

[0139] According to the above technical solution, in a control device for image quality enhancement provided by an embodiment of the present application, when it is detected that the output resolution of a target image has changed relative to that of a previous frame of image, a quality enhancement parameter of the target image is obtained, and the quality enhancement parameter includes a quality enhancement method applicable to each image sub-region contained in the target image, such as a first enhancement method and a second enhancement method with different quality enhancement time and quality enhancement effect, and the quality enhancement parameter makes the time consumed for quality enhancement of the target image less than or equal to the target threshold. Therefore, in this embodiment, in response to the change in output resolution, a quality enhancement parameter is obtained that can make the time consumed for quality enhancement of the target image not exceed the target threshold, so that as long as the output resolution changes, the quality enhancement parameter that consumes less than the target threshold is used to enhance the image quality, thereby avoiding the situation where the image output is stuck due to the time consumed for quality enhancement exceeding the target threshold, thereby improving the smoothness of image output.

[0140] In one implementation, when the parameter acquisition unit 602 obtains the image quality enhancement parameters of the target image, it is specifically used to: determine whether the current image performs image quality enhancement; when the current image performs image quality enhancement, obtain the image quality enhancement parameters of the target image in a first acquisition method; when the current image does not perform image quality enhancement, obtain the image quality enhancement parameters of the target image in a second acquisition method; wherein the first acquisition method and the second acquisition method are different.

[0141] In one implementation, when the parameter acquisition unit 602 obtains the image quality enhancement parameters of the target image in a first acquisition method, it is specifically used to: adjust the image quality enhancement parameters of the current image according to the size relationship between the output resolution of the target image and the output resolution of the current image to obtain the image quality enhancement parameters of the target image.

[0142] For example, the parameter acquisition unit 602 determines whether the output resolution of the target image is greater than the output resolution of the current image; if the output resolution of the target image is greater than the output resolution of the current image, at least one first sub-area of ​​the image quality enhancement parameters of the current image is adjusted from the first enhancement mode to the second enhancement mode in sequence until the selected enhancement parameters that meet the preferred conditions are obtained; if the output resolution of the target image is less than the output resolution of the current image, at least one second sub-area of ​​the image quality enhancement parameters of the current image is adjusted from the second enhancement mode to the first enhancement mode in sequence until the selected enhancement parameters that meet the preferred conditions are obtained; the selected enhancement parameters that meet the preferred conditions are determined as the image quality enhancement parameters of the target image.

[0143] In a preferred embodiment, the area of ​​the first sub-region adjusted last time is smaller than the area of ​​the first sub-region adjusted last time; the area of ​​the second sub-region adjusted last time is larger than the area of ​​the second sub-region adjusted last time.

[0144] In one implementation, when the parameter acquisition unit 602 obtains the image quality enhancement parameters of the target image in the second acquisition method, it is specifically used to: when the output resolution of the target image is less than the output resolution of the current image, obtain an enhancement judgment result, and the enhancement judgment result indicates whether to perform image quality enhancement on the target image; if the enhancement judgment result indicates that image quality enhancement is to be performed on the target image, adjust the initial enhancement parameters to obtain the image quality enhancement parameters of the target image; wherein each of the image sub-regions in the initial enhancement parameters uses the second enhancement method.

[0145] For example, the parameter acquisition unit 602 sequentially adjusts at least one third sub-area of ​​the initial enhancement parameters from the second enhancement mode to the first enhancement mode until a candidate enhancement parameter that meets the preferred condition is obtained; and the candidate enhancement parameter that meets the preferred condition is determined as the image quality enhancement parameter of the target image.

[0146] In a preferred embodiment, the area of ​​the third sub-region adjusted last time is smaller than the area of ​​the third sub-region adjusted later.

[0147] Among them, the preferred conditions include: the image quality enhancement parameters make the time consumed for image quality enhancement of the target image less than or equal to the target threshold, and the image quality enhancement parameters make the time consumed for image quality enhancement of the target image the largest among all the candidate enhancement parameters.

[0148] It should be noted that the specific implementation of each unit in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0149] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0150] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0151] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0152] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the image quality enhancement control methods provided in the embodiments of the present application.

[0153] A computer storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the image quality enhancement control methods provided in the embodiment of the present application.

[0154] Taking the scenario of playing a video on a mobile phone player as an example, the technical solution of this application is illustrated below:

[0155] In order to improve the quality of video playback, when users are watching standard or low-definition videos, the player can use the AI ​​super-resolution model for calculation and enhance the standard or low-definition videos to ultra-high-definition videos. This can not only save traffic for users, but also allow them to watch higher-quality videos. At the same time, the bandwidth cost and pressure on the server can also be reduced, achieving three goals at one stroke.

[0156] However, there are large differences in the computing performance of terminal devices such as mobile phones, and as the image resolution increases, the computing area increases greatly, resulting in a large increase in the amount of calculation, which may force the AI ​​super-resolution image quality enhancement function to be turned off due to insufficient performance.

[0157] Therefore, this application proposes a method of dynamically and real-time statistics on the time consumption of image quality enhancement in each image sub-region, and also introduces the USM traditional image enhancement algorithm (the amount of calculation is greatly reduced compared to the AI ​​super-resolution model). As the resolution changes, the optimal calculation strategy, i.e., the image quality enhancement parameters mentioned above, is adaptively adjusted to ensure that the image quality enhancement function of the AI ​​super-resolution model runs smoothly and improve the user's viewing experience.

[0158] The key technical improvements in the technical solution of this application are as follows:

[0159] (1) Add the traditional USM image enhancement algorithm to ensure that the image quality enhancement algorithm of the AI ​​super-resolution model runs smoothly on low-end performance machines. It can also solve the problem of time-consuming, power-consuming, and machine heating caused by the large amount of AI super-resolution calculations for videos with a resolution of 720p or above. If this function cannot be met, turn it off.

[0160] (2) By real-time statistics of calculation time, the optimal AI image quality enhancement calculation strategy is adaptively selected. Under the premise of ensuring the power consumption and heat generation of the machine, the machine hardware performance is fully utilized to maximize the AI ​​image quality enhancement effect.

[0161] (3) When switching between multiple resolutions, the system automatically determines whether the target resolution can meet the calculation time requirement of AI image quality enhancement, and automatically turns on or off the AI ​​image quality enhancement function.

[0162] like Figure 8As shown in , it is the overall process of this application. It includes: before decoding the video frame, first obtain the current frame (ie, the target image), then determine whether the output resolution has changed relative to the previous frame, if so, then readjust the optimal image quality enhancement strategy, and determine whether to switch image quality enhancement and determine whether to change the execution flag accordingly (the execution flag indicates whether the image quality enhancement function is turned on). If the resolution has not changed or after changing the execution flag, determine whether to perform image quality enhancement. If not, then directly render the current frame and output the current frame. If yes, then perform image quality enhancement according to the adjusted enhancement strategy. After that, render the enhanced current frame and enhance the calculation time consumption data in real time, and then update the super-resolution model time consumption statistics and USM calculation time consumption statistics according to the execution real-time time consumption, so as to facilitate the subsequent adjustment of the enhancement strategy.

[0163] It should be noted that the final region set of the current frame can be generated when the server-side video is encoded and obtained through a fusion algorithm. Taking variety shows as an example, the video frame is mainly divided into three categories of image sub-regions: portrait explicit feature region, subtitle region, and station logo region.

[0164] 1. Video frame decoding to obtain the current frame: The player decodes the video stream sent by the server to obtain continuous video image data and a set of feature interval sequences for the image, and obtains the final region set to be super-resolution calculated through a fusion algorithm.

[0165] 2. Whether the resolution has changed: If it has changed, recalculate the strategy and adjust it.

[0166] 3. Whether to perform enhanced calculation: Determine whether to perform AI super-resolution image quality enhancement calculation based on whether the calculation is executed.

[0167] 4. Perform AI super-resolution image quality enhancement calculation: Perform AI super-resolution calculation based on the most regional set and the calculation strategy corresponding to the set.

[0168] 5. Execute AI super-resolution image quality enhancement calculation feedback to update the estimated value: According to the actual time consumed by AI super-resolution image quality enhancement calculation, update the model or USM calculation statistics respectively.

[0169] 6. Enter rendering: replace the current frame with the image frame calculated by super-resolution and enter the rendering queue for image rendering.

[0170] The following nodes need special explanation:

[0171] (1) Super-resolution region description: The final region set of the current frame is generated during the video encoding on the server side and obtained through a fusion algorithm (mainly divided into three categories: portrait explicit feature region, subtitle region, and station logo region).

[0172] (2) AI super-resolution image quality enhancement calculation strategy: The calculation amount of AI super-resolution model and USM is as follows: super-resolution model corresponding to the portrait explicit feature area > super-resolution model corresponding to the subtitle area > super-resolution model corresponding to the logo area > USM. Therefore, when the resolution increases and the AI ​​model calculation amount is too large to meet the real-time requirements, the USM calculation replaces the AI ​​super-resolution model calculation to meet the real-time performance requirements. Conversely, when there is excess performance, the AI ​​super-resolution model replaces the USM calculation to ensure the image quality enhancement effect.

[0173] (3) The values ​​that need to be counted in real time are: the super-resolution model time g1 corresponding to the portrait explicit feature area, the super-resolution model time g2 corresponding to the subtitle area, the super-resolution model time g3 corresponding to the station logo area, and the USM calculation time g4 (the time unit is ms / s, that is, the time per unit area), which is the time unit statistical value mentioned above.

[0174] like Fig. 9 As shown in the figure, it is a flowchart for performing image quality enhancement. First, the current video frame is decoded, and the final three sets of regions are obtained through fusion calculation: a set of portrait explicit feature regions, a set of subtitle regions, and a set of station logo regions. According to the strategy (i.e., image quality enhancement parameters), the corresponding super-resolution model is selected to perform AI super-resolution calculation on the region or USM image enhancement calculation on the region. Then, the final image after image quality enhancement calculation can be obtained through the image fusion algorithm.

[0175] like Fig.10 As shown in , it is a flow chart of updating the estimated value for picture quality enhancement feedback. Among them:

[0176] 1) Update of the super-resolution model statistic g1 corresponding to the human-dominant characteristic area: Since the AI ​​super-resolution model corresponding to the human-dominant characteristic area is calculated or directly calculated by the USM image algorithm, when the super-resolution model is used for calculation, the super-resolution model statistic g1 corresponding to the human-dominant characteristic area is updated according to the calculation formula g1=(g1*(n-1)+tn / sn) / n, where n is the nth image quality enhancement, tn is the time taken for this image quality enhancement, and sn is the area for this image quality enhancement. When playback starts, the initial value of g1 is 0, where n is the number of super-resolution model calculations corresponding to the human-dominant characteristic area (other models or USM calculation statistics may be different). When USM calculation is used, USM calculation statistics are performed (that is, the historical average value is calculated).

[0177] 2) Update of super-resolution model statistics corresponding to the subtitle area: Similar to the update of super-resolution model statistics corresponding to the portrait explicit feature area, the update of the subtitle model statistics g2 is calculated as g2 = (g2*(n-1)+tn / sn) / n. The initial value of g2 is 0 at the beginning of playback.

[0178] 3) Update of the super-resolution model statistics corresponding to the logo area: Similar to the update of the super-resolution model statistics corresponding to the portrait explicit feature area, the update of the logo model statistics g3 is calculated as g3 = (g3*(n-1)+tn / sn) / n. The initial value of g3 is 0 at the beginning of playback.

[0179] 4) USM calculation statistics update: USM calculation statistics g4 is updated, and the calculation formula is g4 = (g4*(n-1)+tn / sn) / n. When playback starts, the initial value of g4 is 0. When there are multiple areas for USM calculation, multiple updates are sufficient.

[0180] like Fig.11 As shown in , it is a flowchart of the total time consumption for estimating the image quality enhancement of the current frame, where:

[0181] When adjusting the optimal calculation strategy, the time consumption for image quality enhancement of the current frame will be estimated. The estimated time consumption for the dominant feature area of ​​the portrait is t1, s1 is the area of ​​the dominant area of ​​the portrait in the current frame, and the estimated value of t1 is s1*g1 or s1*g4. Similarly, the estimated time consumption of subtitles t2 and the estimated time consumption of the station logo t3 can be calculated. The total time consumption t=t1+t2+t3, or the total time consumption is the maximum value of t1, t2, and t3. Among them, since the calculation of the remaining area is relatively simple, and the time consumption is very small compared to t1, t2, and t3, it can be ignored.

[0182] like Fig.12 As shown in , it is a flowchart for re-adjusting the logic of the optimal image quality enhancement strategy, where:

[0183] Step 1201: When the resolution changes, the policy adjustment logic is triggered. First, determine whether the current state is on. If it is off, jump to step 1202, otherwise jump to step 1203.

[0184] Step 1202: Determine whether the change is an increase or decrease.

[0185] Among them, if the resolution increases, jump to step 1204. If the resolution is in a decreasing state, first calculate the value t1 of the time-consuming estimation for all three major regions using USM calculation. Then, replace the AI super-resolution model in the logo area, subtitle area, and human portrait dominant feature area in sequence, and calculate the estimated time-consuming t2 (using the AI super-resolution model in the logo area), t3 (using the AI super-resolution model in the subtitle area), t4, etc. Among them, t1 < t2 < t3 < t4. Then, judge from t1, t2, t3, and t4. If t1 is greater than the threshold m (i.e., the target threshold, m is the minimum activation threshold, usually 20 ms), then the activation adjustment is not satisfied, and jump to step 1204. If t1 is less than m, judge t1, t2, t3, and t4 in sequence to find the value closest to the threshold m but less than m, and adjust according to the corresponding calculation strategy, then jump to step 1205.

[0186] Step 1203: Determine whether the resolution increases or decreases this time. If it increases, first calculate the value t0 estimated by the current calculation strategy. If t0 is greater than the threshold M, then replace the USM calculation in sequence (logo model, subtitle model, human portrait dominant model) and recalculate the new estimate t until t is less than or equal to the threshold m, then adjust to the current calculation strategy, and return to step 1205. If the recalculated value t of all regions after USM calculation is greater than the threshold m, then return to step 1204.

[0187] If the resolution decreases, if the threshold t0 is less than or equal to the threshold m, then replace the USM calculation in the current strategy in sequence (human portrait dominant model, subtitle model, logo model) to obtain a new estimated value t0. If t0 is greater than the threshold m, then update according to the previous enhancement strategy and return to step 1205. If t0 is less than the threshold m, then continue to replace the strategy until the new estimated value t0 calculated by replacing all with the AI super-resolution model is also less than the threshold m, then update the calculation strategy to all AI super-resolution model calculations and return to step 1205.

[0188] Step 1204: Set the super-resolution image quality enhancement to the off state and end the strategy adjustment.

[0189] Step 1205: Set the super-resolution image quality enhancement to the on state and end the strategy adjustment.

[0190] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0191] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0192] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0193] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A control method for image quality enhancement, characterized in that: The method comprises: Monitoring whether the output resolution of a target image relative to a current image changes; the current image is a previous frame image of the target image; In response to the output resolution changing, obtaining image quality enhancement parameters of the target image; According to the image quality enhancement parameter of the target image, the image quality enhancement parameter of the target image makes the time consumption of image quality enhancement of the target image less than or equal to the target threshold; Among them, the image quality enhancement parameters include the image quality enhancement method used for each image sub-region contained in the target image, the image quality enhancement method includes a first enhancement method or a second enhancement method, the image quality enhancement time of the first enhancement method is greater than the image quality enhancement time of the second enhancement method, and the image quality enhancement effect of the first enhancement method is better than the image quality enhancement effect of the second enhancement method.

2. The control method according to claim 1, characterized in that: Obtaining image quality enhancement parameters of the target image, including: Determining whether image quality enhancement is performed on the current image; In the case where image quality enhancement is performed on the current image, obtaining image quality enhancement parameters of the target image in a first acquisition manner; In a case where image quality enhancement is not performed on the current image, obtaining image quality enhancement parameters of the target image in a second acquisition manner; The first acquisition method and the second acquisition method are different.

3. The control method according to claim 2, characterized in that: Obtaining the image quality enhancement parameters of the target image in a first acquisition manner includes: According to the size relationship between the output resolution of the target image and the output resolution of the current image, the image quality enhancement parameters of the current image are adjusted to obtain the image quality enhancement parameters of the target image.

4. The control method according to claim 3, characterized in that: According to the size relationship between the output resolution of the target image and the output resolution of the current image, adjusting the image quality enhancement parameter of the current image to obtain the image quality enhancement parameter of the target image includes: Determining whether the output resolution of the target image is greater than the output resolution of the current image; If the output resolution of the target image is greater than the output resolution of the current image, at least one first sub-region of the image quality enhancement parameter of the current image is adjusted from the first enhancement mode to the second enhancement mode in sequence until a candidate enhancement parameter that meets the preferred condition is obtained; If the output resolution of the target image is smaller than the output resolution of the current image, at least one second sub-region of the image quality enhancement parameter of the current image is adjusted from the second enhancement mode to the first enhancement mode in sequence until a candidate enhancement parameter that meets the preferred condition is obtained; The candidate enhancement parameters that meet the preferred conditions are determined as the image quality enhancement parameters of the target image.

5. The control method according to claim 4, characterized in that: The area of ​​the first sub-region adjusted last time is smaller than the area of ​​the first sub-region adjusted next time; The area of ​​the second sub-region adjusted last time is greater than the area of ​​the second sub-region adjusted later.

6. The control method according to claim 2, characterized in that: Obtaining the image quality enhancement parameters of the target image in a second acquisition manner includes: When the output resolution of the target image is less than the output resolution of the current image, obtaining an enhancement judgment result, wherein the enhancement judgment result indicates whether to perform image quality enhancement on the target image; If the enhancement judgment result indicates that image quality enhancement is performed on the target image, adjusting the initial enhancement parameters to obtain image quality enhancement parameters of the target image; Wherein, each of the image sub-regions in the initial enhancement parameters uses the second enhancement method.

7. The control method according to claim 6, characterized in that: The initial enhancement parameters are adjusted to obtain image quality enhancement parameters of the target image, including: sequentially adjusting at least one third sub-region in the initial enhancement parameters from the second enhancement mode to the first enhancement mode until a candidate enhancement parameter that meets the preferred condition is obtained; The candidate enhancement parameters that meet the preferred conditions are determined as the image quality enhancement parameters of the target image.

8. The control method according to claim 7, characterized in that: The area of ​​the third sub-region adjusted last time is smaller than the area of ​​the third sub-region adjusted later.

9. The control method according to claim 4 or 7, characterized in that: The preferred conditions include: the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image less than or equal to the target threshold, and the image quality enhancement parameter makes the time consumed for image quality enhancement of the target image the largest among all the selected enhancement parameters.

10. A control device for enhancing image quality, characterized in that: include: A resolution monitoring unit, used to monitor whether the output resolution of the target image relative to the current image changes; The current image is a previous frame image of the target image; A parameter obtaining unit, configured to obtain a quality enhancement parameter of the target image in response to a change in the output resolution; an enhancement execution unit, configured to perform image quality enhancement on the target image according to an image quality enhancement parameter of the target image, wherein the image quality enhancement parameter of the target image makes the time consumed for performing image quality enhancement on the target image less than or equal to a target threshold; Among them, the image quality enhancement parameters include the image quality enhancement method used for each image sub-region contained in the target image, the image quality enhancement method includes a first enhancement method or a second enhancement method, the image quality enhancement time of the first enhancement method is greater than the image quality enhancement time of the second enhancement method, and the image quality enhancement effect of the first enhancement method is better than the image quality enhancement effect of the second enhancement method.

Citation Information

Patent Citations

  • Video processing method and device, electronic device, and storage medium

    CN109587560A

  • Video processing method and device, electronic equipment and storage medium

    CN109660821A

  • Image quality enhancement method and device, electronic equipment and readable storage medium

    CN111325692A

  • Video super-division processing method, video super-division processing device and storage medium

    CN112950465A

  • Image processing method and device, electronic equipment and storage medium

    CN116362972A