Video sharpening method, sharpening intensity prediction model training method and device

Abstract

The present disclosure provides a video sharpening method, a training method and a device for a sharpening strength prediction model. The present disclosure relates to the field of image processing technology, and in particular to the field of computer vision, video processing, and multimedia technology. The specific implementation scheme is: according to the first feature of the k-th frame image of the target video, using the sharpening strength prediction model, determine the first sharpening parameter of the k+1-th frame image of the target video; determine the second feature of the multiple image blocks constituting the k+1-th frame image; according to the first sharpening parameter and the second features of the multiple image blocks, determine the second sharpening parameters of the multiple image blocks; and according to the second sharpening parameters of the multiple image blocks, perform sharpening processing on the k+1-th frame image. According to the scheme of the embodiment of the present disclosure, the first sharpening parameter of a frame image is predicted by the model, and on this basis, the second sharpening parameter of each image block of the frame image is determined. This not only reduces the amount of calculation, but also increases the sharpening speed, thereby improving the overall sharpening effect of the video.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to the field of computer vision, video processing, and multimedia technology. Background Art

[0002] Sharpening is an effective video enhancement method for improving subjective video quality. Sharpening, performed during the preprocessing phase before encoding, effectively emphasizes textures and edges, significantly improving video quality. However, sharpening increases the video bitrate, and excessive sharpening has limited impact on video quality and may even negatively impact it. Determining the appropriate sharpening strength requires evaluating the subjective quality changes brought about by sharpening and balancing this with the increased bitrate associated with sharpening. This allows us to determine the optimal sharpening strength that maximizes video quality while minimizing the bitrate penalty, which serves as the video sharpening parameter. Summary of the Invention

[0003] The present disclosure provides a video sharpening method, a training method, an apparatus, a device, and a storage medium for a sharpening intensity prediction model.

[0004] According to one aspect of the present disclosure, a video sharpening method is provided, comprising:

[0005] Determine, based on the first feature of the k-th frame image of the target video, a first sharpening parameter of the k+1-th frame image of the target video using a sharpening strength prediction model; wherein k is a positive integer;

[0006] Determining second features of a plurality of image blocks constituting the k+1th frame image;

[0007] determining second sharpening parameters for the plurality of image blocks based on the first sharpening parameters and the second characteristics of the plurality of image blocks; and

[0008] The k+1th frame image is sharpened according to the second sharpening parameters of the multiple image blocks.

[0009] According to another aspect of the present disclosure, a method for training a sharpening intensity prediction model is provided, comprising:

[0010] Determining predicted values of sharpening parameters of the plurality of frame images according to the fifth features of the plurality of frame images extracted from the video sample using the first prediction model;

[0011] Determining a first model parameter group of a first prediction model according to true values and predicted values of sharpening parameters of a plurality of frame images;

[0012] determining a plurality of second model parameter groups based on a plurality of model parameters of the first model parameter group;

[0013] Constructing a plurality of second prediction models based on the first prediction model according to the plurality of second model parameter groups;

[0014] Determining multiple sets of third sharpening parameters for the multiple frame images using multiple second prediction models according to the fifth features of the multiple frame images; and

[0015] A sharpening intensity prediction model is determined from a plurality of second prediction models according to a plurality of coding efficiency indicators obtained after a plurality of frame images are sharpened according to a plurality of sets of third sharpening parameters.

[0016] According to another aspect of the present disclosure, there is provided a video sharpening apparatus, comprising:

[0017] A first prediction module is configured to determine a first sharpening parameter of the k+1th frame of the target video based on a first feature of the kth frame of the target video using a sharpening strength prediction model, wherein k is a positive integer;

[0018] A first determining module is used to determine a second feature of a plurality of image blocks constituting the k+1th frame image;

[0019] A first adjustment module is configured to determine second sharpening parameters of the plurality of image blocks according to the first sharpening parameters and second features of the plurality of image blocks; and

[0020] The first sharpening module is configured to perform sharpening processing on the (k+1)th frame image according to the second sharpening parameters of the plurality of image blocks.

[0021] According to another aspect of the present disclosure, a training apparatus for a sharpening intensity prediction model is provided, comprising:

[0022] a second prediction module, configured to determine predicted values of sharpening parameters of the plurality of frame images according to the fifth features of the plurality of frame images extracted from the video sample and using the first prediction model;

[0023] A second determining module is configured to determine a first model parameter group of a first prediction model according to true values and predicted values of sharpening parameters of a plurality of frame images;

[0024] a parameter generation module, configured to determine a plurality of second model parameter groups based on a plurality of model parameters of the first model parameter group;

[0025] A construction module, configured to construct a plurality of second prediction models based on the first prediction model according to the plurality of second model parameter groups;

[0026] a third prediction module, configured to determine, based on the fifth features of the plurality of frame images, a plurality of second prediction models, a plurality of sets of third sharpening parameters for the plurality of frame images; and

[0027] The model determination module is used to determine a sharpening intensity prediction model from multiple second prediction models based on multiple coding efficiency indicators obtained after multiple frame images are sharpened according to multiple sets of third sharpening parameters.

[0028] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0029] at least one processor; and

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to execute the method of any embodiment of the present disclosure.

[0032] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any one of the embodiments of the present disclosure.

[0033] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements the method according to any one of the embodiments of the present disclosure.

[0034] According to the solution of the embodiment of the present disclosure, the model is used to predict the first sharpening parameters of the current frame image based on the first feature of the previous frame image. On the basis of the first sharpening parameters, the sharpening parameters are fine-tuned according to the second features of each image block of the current frame. This not only reduces the amount of calculation, but also increases the sharpening speed, thereby improving the overall sharpening effect of the video.

[0035] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0037] Figure 1 is a flowchart of a video sharpening method according to the first embodiment of the present disclosure;

[0038] Figure 2 is a flowchart of a method for training a sharpening intensity prediction model according to the second embodiment of the present disclosure;

[0039] Figure 3 is a structural diagram of a video sharpening device according to a third embodiment of the present disclosure;

[0040] Figure 4 2 is a schematic structural diagram of a training device for a sharpening intensity prediction model according to a fourth embodiment of the present disclosure;

[0041] Figure 5 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0043] Currently, the most effective indicator for evaluating video quality is the Video Multimethod Assessment Fusion (VMAF) indicator. The trade-off between VMAF improvement and video bitrate increase can be measured using VMAF BD-rate ( BD-rate is expressed as a delta rate, representing bitrate savings at the same VMAF. BD-rate measures the bitrate reduction provided by a codec or codec feature while maintaining the same quality as measured by objective metrics. It essentially represents the bitrate improvement achieved while maintaining the same quality as the objective metric. Its calculation involves deriving a function from two rate-distortion curves: a reference curve and a test curve. When the two curves have the same performance, BD-rate is 0%. When the test curve demonstrates better results than the reference curve, the BD-rate value is less than zero.

[0044] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the embodiments of the present disclosure provide a video sharpening method and a training method for a sharpening intensity prediction model. By using the technical solutions of the embodiments of the present disclosure, the optimal sharpening intensity can be quickly determined.

[0045] Figure 1 FIG. 1 is a flow chart of a video sharpening method according to an embodiment of the present disclosure. Figure 1 As shown, the method comprises at least the following steps:

[0046] S110: Determine a first sharpening parameter for the k+1th frame of the target video using a sharpening strength prediction model based on the first feature of the kth frame of the target video, where k is a positive integer.

[0047] In the disclosed embodiments, the first feature can be understood as the complexity of the content of each frame of the target video, or the degree of detail prominence in the image. Specifically, it can be the complexity of the pixel values of multiple pixels in the image. The arrangement of pixels with different pixel values forms texture characteristics.

[0048] It should be noted that the first frame of the target video can be sharpened using preset sharpening parameters, and the specific sharpening parameter values used are not specifically limited here. For example, 0.5-0.9. The first frame of the target video can also be unsharpened. Starting from the second frame of the target video, for each frame, the first feature of the previous frame is input into the sharpening strength prediction model, and the sharpening strength prediction model outputs the first sharpening parameter of the current frame.

[0049] S120: Determine second features of a plurality of image blocks constituting the (k+1)th frame image.

[0050] In the disclosed embodiments, the second feature can also be understood as representing the complexity of the content in the image block. The second feature can be calculated using the same method as the first feature, or a different method. Using the same method can avoid the complexity overhead caused by additional feature calculations.

[0051] An image frame consists of multiple non-overlapping image blocks, and the size of each block can be adjusted based on a preset precision. Smaller block sizes increase the number of blocks per frame and provide finer granularity. However, if the block size is too small, there will be too few pixels, resulting in inaccurate feature extraction. As a trade-off, each block can be 64×64 pixels in length and width.

[0052] S130: Determine second sharpening parameters for the plurality of image blocks according to the first sharpening parameters and the second characteristics of the plurality of image blocks.

[0053] In the embodiment of the present disclosure, according to the second features of the plurality of image blocks of the (k+1)th frame image, fine-tuning is performed on the basis of the first sharpening parameters to determine the second sharpening parameters for each image block.

[0054] Specifically, a relatively optimal frame sharpening strength (first sharpening parameter) is obtained through step S110. On this basis, for each 64×64 image block, a three-classification model is determined based on the second feature that characterizes the texture characteristics of the image block. Based on the frame sharpening strength, the sharpening strength (second sharpening parameter) of each image block is increased, decreased, or maintained unchanged by a fixed step size. The fixed step size can be determined based on experimental results, for example, it can be set to 0.1-0.3.

[0055] S140: Performing sharpening processing on the (k+1)th frame image according to the second sharpening parameters of the plurality of image blocks.

[0056] In the embodiment of the present disclosure, each image block is sharpened according to the second sharpening parameter of each image block. After each image block is sharpened, the sharpening process of the frame image is achieved.

[0057] According to the solution of the embodiment of the present disclosure, the model is used to predict the first sharpening parameters of the current frame image based on the first feature of the previous frame image. On the basis of the first sharpening parameters, the first sharpening parameters are further fine-tuned according to the second features of each image block of the current frame to obtain the second sharpening parameters of each image block. Since each image block fine-tunes the sharpening parameters according to its own characteristics, this not only reduces the amount of calculation, but also increases the sharpening speed, thereby improving the overall sharpening effect of the video.

[0058] It should be noted that directly predicting the sharpening intensity of each image block is somewhat difficult. Image blocks contain too little information, and the correlation between each feature and the optimal sharpening intensity of the image block is low. Therefore, direct prediction of the sharpening intensity of image blocks is not considered. Furthermore, direct prediction may result in large discrepancies between the predicted sharpening intensities of adjacent blocks, which can easily create a subjective sense of disconnection. Therefore, we first predict the optimal sharpening intensity for each frame and then fine-tune the sharpening intensity of the image blocks.

[0059] In one possible implementation, the video sharpening method of the embodiment of the present disclosure includes steps S110 to S140, wherein step S140: determining second sharpening parameters for the plurality of image blocks based on the first sharpening parameters and the second features of the plurality of image blocks, further includes the following steps:

[0060] S141: Determine the first feature of the (k+1)th frame image according to the second features of the plurality of image blocks.

[0061] S142: Adjust the first sharpening parameter according to the second features of the multiple image blocks and the first features of the (k+1)th frame image to obtain second sharpening parameters of the multiple image blocks.

[0062] In the disclosed embodiment, since the first and second features can be calculated using the same method, after determining the second features of multiple image blocks in the k+1th frame image, the first feature of the k+1th frame image can be further determined based on the second features. Subsequently, based on the second features of each image block and the first features of the frame image in which the image block resides, the first sharpening parameters of the frame image are fine-tuned to obtain the second sharpening features of each image block.

[0063] According to the solution of the embodiment of the present disclosure, the first feature of the previous frame image can be used to predict the first sharpening parameter of the current frame, and the second feature of each image block of the current frame can be used to calculate the first feature of the current frame image after being used to fine-tune the first sharpening parameter. The first feature of the current frame image can be used to predict the first sharpening parameter of the next frame image. It can also be understood that calculating the second features of each image block that constitutes a frame image is an intermediate step in calculating the first feature of the frame image. Therefore, calculating the second feature of each image block does not increase the additional computational effort. Therefore, the sharpening intensity of each frame image and each image block can be quickly determined, and the sharpening effect is improved while ensuring the sharpening processing speed.

[0064] In one possible implementation, the video sharpening method of the embodiment of the present disclosure includes steps S110 to S140, and S141 to S142, wherein step S142: adjusting the first sharpening parameter according to the second features of the multiple image blocks and the first features of the k+1th frame image to obtain the second sharpening parameters of the multiple image blocks, further includes the steps of:

[0065] The third feature is determined according to the first functional relationship of the first feature of the (k+1)th frame image.

[0066] When the second feature of a first image block among the multiple image blocks is smaller than the third feature, a sharpening parameter obtained by increasing the first sharpening parameter is used as the second sharpening parameter of the first image block, where the first image block is any one of the multiple image blocks.

[0067] In the embodiment of the present disclosure, the first functional relationship may be the product of the first feature and a first preset coefficient a, and the value range of a may be (0, 1). For example, if the value of a is 0.7, then the third feature = 0.7 × the first feature.

[0068] The second feature of the first image block is smaller than the third feature. This can be understood as indicating that the content contained in the first image block is less complex, or that the first image block is a simple image block. For simple image blocks, the sharpening strength can be further increased based on the first sharpening parameter.

[0069] According to the solution of the embodiment of the present disclosure, the sharpening strength is appropriately fine-tuned according to the second feature of the image block, and the sharpening strength is increased for the image block with lower complexity, so that the sharpening effect of the entire frame image can be better.

[0070] In one possible implementation, the video sharpening method of the embodiment of the present disclosure includes steps S110 to S140, and S141 to S142, wherein step S142: adjusting the first sharpening parameter according to the second features of the multiple image blocks and the first features of the k+1th frame image to obtain the second sharpening parameters of the multiple image blocks, further includes the steps of:

[0071] The fourth feature is determined according to the second functional relationship of the first feature of the (k+1)th frame image.

[0072] When the second feature of a second image block among the multiple image blocks is greater than the fourth feature, a sharpening parameter obtained by reducing the first sharpening parameter is used as a second sharpening parameter of the second image block, where the second image block is any one of the multiple image blocks.

[0073] In the embodiment of the present disclosure, the second functional relationship may be the product of the first characteristic and a second preset coefficient b, and the value range of b may be (1, 2). For example, if the value of b is 1.4, then the fourth characteristic = 1.4 × the first characteristic.

[0074] The second feature of the second image block is greater than the fourth feature, which can be understood as the complexity of the content contained in the second image block is relatively high, or it can be understood as a complex image block. For complex image blocks, the sharpening intensity can be further reduced based on the first sharpening parameter.

[0075] According to the solution of the embodiment of the present disclosure, the sharpening intensity is appropriately fine-tuned according to the second feature of the image block, and the sharpening intensity is reduced for image blocks with higher complexity, so that the sharpening effect of the entire frame image can be better.

[0076] In one possible implementation, the video sharpening method of the embodiment of the present disclosure includes steps S110 to S140, and S141 to S142, wherein step S142: adjusting the first sharpening parameter according to the second features of the multiple image blocks and the first features of the k+1th frame image to obtain the second sharpening parameters of the multiple image blocks, further includes the steps of:

[0077] The third feature is determined according to the first functional relationship of the first feature of the (k+1)th frame image.

[0078] A fourth feature is determined based on the second functional relationship of the first feature of the (k+1)th frame image, wherein the fourth feature is greater than the third feature.

[0079] When the second feature of a third image block among the multiple image blocks is smaller than the fourth feature and larger than the third feature, the first sharpening parameter is used as the second sharpening parameter of the third image block, where the third image block is any one of the multiple image blocks.

[0080] In the embodiment of the present disclosure, the second feature of the third image block is between the third feature and the fourth feature. It can be understood that the complexity of the content contained in the third image block is moderate. Therefore, the first sharpening parameter can be directly used to sharpen the third image block.

[0081] According to the solution of the embodiment of the present disclosure, by appropriately fine-tuning the sharpening intensity according to the second feature of the image block, for an image block with moderate complexity, directly using the first sharpening parameter as the second sharpening parameter of the image block for sharpening processing can make the sharpening effect of the entire frame of the image better.

[0082] In addition to using the above function relationship and preset coefficient method, in one example, a function f(frame feature, block feature) can be constructed, and its relationship with the threshold is compared to determine the three-class model. Among them, the frame feature is the first feature of the (k + 1)-th frame image, and the block feature is the second feature of each image block. The general model construction method is as follows:

[0083] If f(frame feature, block feature) < t0, it is judged as increasing the intensity, and the sharpening intensity is increased by 0.2.

[0084] If f(frame feature, block feature) > t1, it is judged as decreasing the intensity, and the sharpening intensity is decreased by 0.2. It is ensured that t1 > t0.

[0085] In other cases, it remains unchanged.

[0086] The three categories of increasing, decreasing or remaining unchanged are divided by t0 and t1. The values of t0 and t1 can be tried with multiple different sets of values, and the true values of t0 and t1 are determined for the actual sharpening effect of the classification scheme for each set of t0 and t1 values through a quality evaluation index. Specifically, since the block level is too small and the bitrate is meaningless, only the bitrate in the video scenario is meaningful, so the VMAF BD-rate evaluation method in the frame model is not applicable. Therefore, the following quality evaluation index is adopted: score index = (variance of the source image / variance of the sharpened image) * (sharpened VMAF - source VMAF). Take the set of values with the optimal score index as the true value for realizing the category division.

[0087] By observing the feature and true value data distribution, it is found that there is a certain rule between the feature and the category. Most simple blocks need to increase the intensity, and complex blocks need to decrease the intensity. A simple rule-based three-class model can be constructed:

[0088] When the block feature < coefficient a * frame feature, select to increase the intensity by 0.2, and the value range of coefficient a is (0, 1).

[0089] When the block feature > coefficient b * frame feature, select to decrease the intensity by 0.2, and the value range of coefficient b is (1, 2).

[0090] In one possible implementation manner, the first feature of the k-th frame image in the embodiment of the present disclosure is obtained according to any of the following calculation methods:

[0091] The mean square of the difference between the pixel values of multiple pixel points of the k-th frame image and the domain Gaussian mean of the multiple pixel points.

[0092] The mean of the absolute values of the differences between the pixel values of multiple pixels in the k-th frame image and the Gaussian mean of the domain of the multiple pixels.

[0093] The variance of the difference between the pixel values of multiple pixels in the k-th frame image and the Gaussian mean of the domain of the multiple pixels.

[0094] The mean value of the variance of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

[0095] The mean value after exponential operation of the variance of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

[0096] The mean value after logarithm operation of the variance of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

[0097] The mean value of the sum of the AC high-frequency coefficients after Hadamard transformation of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

[0098] In the disclosed embodiment, the pixel value of a pixel point can be a brightness value in the YUV color space. The neighborhood of a pixel point can be an area formed by a pixel block of odd side lengths, such as 5×5 pixels or 7×7 pixels. The size of the image block affects the calculation speed and accuracy of the calculation results and can be set according to actual needs. For example, it can be 8×8 pixels, 16×16 pixels, or 32×32 pixels.

[0099] According to the solution of the embodiment of the present disclosure, the first features obtained by the above-mentioned various calculation methods have a high correlation with the true value of the optimal sharpening intensity, and can more accurately predict the sharpening intensity.

[0100] In one possible implementation, the video sharpening method of the embodiment of the present disclosure includes steps S110 to S140 and S141 to S142, wherein step S120: determining the second features of multiple image blocks constituting the k+1th frame image further includes the steps of:

[0101] The difference between the pixel value of each pixel point of the multiple image blocks constituting the k+1th frame image and the Gaussian mean of the pixel points is calculated to obtain an intermediate result.

[0102] The square mean of the intermediate results of each pixel of the image block is calculated to obtain the second feature of the image block.

[0103] It should be noted that when the first and second features use the same calculation method, the amount of calculation can be reduced. For example, when the first feature is calculated based on the square mean of the difference between the pixel value of each pixel in a frame image and the Gaussian mean of the pixel area, the second feature is also calculated based on the square mean of the difference between the pixel value of each pixel in its image block and the Gaussian mean of the pixel area. Since the frame image is composed of multiple image blocks, the first feature of the frame image can be obtained by taking the average of the second features of each image block.

[0104] In the embodiment of the present disclosure, the difference between the pixel value of each pixel and the Gaussian mean of the domain of the pixel is first calculated as an intermediate result, and then the square mean of the intermediate result of each pixel in the image block is calculated to obtain the second feature of the image block.

[0105] It should be noted that when the first and second features use the same calculation method, the amount of calculation can be reduced. For example, when the first feature is calculated based on the square mean of the difference between the pixel value of each pixel in a frame image and the Gaussian mean of the pixel area, the second feature is also calculated based on the square mean of the difference between the pixel value of each pixel in its image block and the Gaussian mean of the pixel area. Since the frame image is composed of multiple image blocks, the first feature of the frame image can be obtained by taking the average of the second features of each image block.

[0106] According to the solution of the embodiment of the present disclosure, the second feature is obtained based on the square average of the difference between the pixel value of each pixel point and the Gaussian mean of the domain of the pixel point. The intermediate result before taking the square average can be reused in feature calculation and sharpening processing, reducing the amount of calculation and improving the speed of sharpening processing.

[0107] In one possible implementation, the video sharpening method of the embodiment of the present disclosure includes steps S110 to S140 and S141 to S142, wherein step S140: sharpening the k+1th frame image according to the second sharpening parameters of the plurality of image blocks, further includes the steps of:

[0108] For each image block, the pixel value of each pixel point of the image block is added with the product of the second sharpening parameter of the image block and the intermediate result.

[0109] In the disclosed embodiment, since the calculation method for sharpening each pixel point in each image block is: sharpened pixel value = source pixel value of the pixel point + second sharpening parameter × (source pixel value of the pixel point - domain Gaussian mean), the difference between the pixel value of the pixel point when calculating the second feature and the domain Gaussian mean can be reused in the image sharpening process.

[0110] According to the solution of the embodiment of the present disclosure, when each image block is sharpened according to the second sharpening parameter of each image block, the intermediate results in the process of calculating the second feature are reused, which reduces the amount of calculation and improves the speed of sharpening processing.

[0111] In one example, in the preprocessing module of actual encoding, the frame sharpening strength is first obtained through the frame prediction model. The sharpening strength of the first frame is an initial value of 0.7, and the sharpening strength of each subsequent frame is predicted based on the features of the previous frame. Then, each 64x64 pixel image block of the current frame is traversed, and Gaussian filtering and statistical features are first calculated for each image block. The above three classification models are used to determine the sharpening strength adjustment method for the block, and the image block is sharpened after adjustment. At the same time, since the frame and block features are the same, after processing each block, the average of the block features can be used to predict the sharpening strength of the next frame, and no additional feature calculation time is required.

[0112] According to the embodiment of the present disclosure, the optimal sharpening intensity of the k+1th frame is predicted in real time based on the video content of the kth frame through the constructed sharpening intensity prediction model. In order to achieve a better improvement effect, the sharpening intensity is fine-tuned according to the image content of each image block. That is to say, the video sharpening method of the embodiment of the present disclosure is divided into two steps: (1) Frame sharpening intensity determination: Based on the frame scene characteristics, the optimal sharpening intensity of a frame is predicted through a regression model. (2) Block sharpening intensity fine-tuning: For each 64x64 block, according to the block texture characteristics, the sharpening intensity of each block is fine-tuned based on the frame sharpening intensity through a three-classification model.

[0113] Figure 2 FIG. 1 is a flow chart of a method for training a sharpening intensity prediction model according to an embodiment of the present disclosure. Figure 2 As shown, the method comprises at least the following steps:

[0114] S210: Determine predicted values of sharpening parameters of the plurality of frame images according to the fifth features of the plurality of frame images extracted from the video sample using the first prediction model.

[0115] The fifth feature is the same as the first feature mentioned above, and can be understood as the complexity of the content of each frame of the target video, or as the degree of detail highlighting in the image.

[0116] The first prediction model can be a linear regression model. Video samples are used to train the first prediction model. The video samples include a dataset formed by multiple videos of multiple scenes, and the dataset includes videos of multiple different resolutions. After determining the video samples, multiple frame images are extracted from the video samples of each scene to determine the true value of the sharpening intensity of the multiple frame images. Extracting frames can reduce computational complexity.

[0117] The true value of the sharpening intensity can be determined using a traversal search method using related techniques. For example, in the interval (0, 1.5), different sharpening intensities are tried at intervals of 0.1. The video is then encoded in Constant Rate Factor (CRF) mode with four CRF parameters: 22, 27, 32, and 37. The bitrate and VMAF are calculated, and the VMAF BD-rate is calculated relative to the unsharpened case. The sharpening intensity with the optimal VMAF BD-rate is taken as the true value corresponding to the scene video.

[0118] S220: Determine a first model parameter group of a first prediction model according to true values and predicted values of sharpening parameters of a plurality of frame images.

[0119] In one example, the first prediction model uses a polynomial as the prediction model, and the model effect is evaluated based on the mean square error (MSE) between the predicted value and the true value. Since the first-order polynomial is not as effective as the second-order polynomial, and the difference between the third-order polynomial and the second-order polynomial is not much, the second-order polynomial is used. Specifically, the second-order polynomial is: y = ax 2 +bx+c. x is the fifth feature of each frame, and y is the true value of the sharpening intensity of each frame. By fitting the relationship between the features of different scenes and the true value of the optimal sharpening intensity with a quadratic polynomial, the values of a, b, and c in the first prediction model are determined to obtain the first model parameter set.

[0120] S230: Determine multiple second model parameter groups based on multiple model parameters of the first model parameter group.

[0121] Select 1 to 3 parameters from each of the first model parameter groups at a time and fine-tune them up or down according to a preset range to obtain multiple second model parameter groups. In one example, the values of a, b, and c in the first model parameter group are (3, 6, 10), respectively. Select 1 to 3 parameters at a time and fine-tune them upward by 10%, and multiple second model parameter groups can be obtained: (3.3, 6.6, 11), (3, 6.6, 11), (3, 6, 11), (3.3, 6, 10), (3.3, 6.6, 10), (3.3, 6, 11). Similarly, when fine-tuning downward, multiple second model parameter groups can also be obtained.

[0122] S240: Constructing multiple second prediction models based on the first prediction model according to the multiple second model parameter groups.

[0123] The multiple second model parameter groups generated by fine-tuning are respectively substituted into the polynomials of the first prediction model to obtain multiple second prediction models corresponding to the multiple second model parameter groups.

[0124] S250: Determine multiple sets of third sharpening parameters for the multiple frame images using multiple second prediction models according to the fifth features of the multiple frame images.

[0125] The fifth features of multiple frame images of each scene in the video sample are input into multiple second prediction models, and each second prediction model outputs a set of third sharpening parameters, each set of third sharpening parameters including the third sharpening parameters corresponding to each frame image, thereby obtaining multiple sets of third sharpening parameters corresponding to the multiple second prediction models.

[0126] S260: Determine a sharpening intensity prediction model from a plurality of second prediction models according to a plurality of coding efficiency indicators obtained after a plurality of frame images are sharpened according to a plurality of sets of third sharpening parameters.

[0127] After sharpening the multiple frames according to each set of third sharpening parameters, coding efficiency indicators are calculated for the multiple frames obtained by processing the frames according to the different sharpening parameters. The second prediction model corresponding to the set of third sharpening parameters having the highest coding efficiency score is selected as the final sharpening intensity prediction model.

[0128] In the disclosed embodiments, it should be noted that VMAF BD-rate or a similar metric can be used as a coding efficiency metric. Taking VMAF BD-rate as an example, since the first model parameter set obtained based on the optimal MSE may not necessarily optimize VMAF BD-rate, the first model parameter set is fine-tuned to obtain multiple second model parameter sets. From the multiple second prediction models constructed from these multiple second model parameter sets, the one that optimizes VMAF BD-rate is determined as the sharpening intensity prediction model.

[0129] According to the solution of the disclosed embodiments, a linear regression sharpening intensity prediction model is trained to predict the optimal sharpening intensity for a frame based on its texture characteristics. The model's parameters are fine-tuned after being determined using samples to ensure an optimal VMAF BD-rate.

[0130] In one possible implementation, the method for training a sharpening intensity prediction model of an embodiment of the present disclosure includes steps S210 to S260, wherein step S260: determining a sharpening intensity prediction model from multiple second prediction models based on multiple coding efficiency indicators obtained after multiple frame images are sharpened according to multiple sets of third sharpening parameters, further includes the steps of:

[0131] S261: Sharpening the plurality of frame images using each of the plurality of third sharpening parameter groups.

[0132] S262: Encode the multiple frame images after the sharpening process.

[0133] S263: Determine coding efficiency indicators of the plurality of frame images after being sharpened and encoded according to each set of third sharpening parameters.

[0134] S264: Determine a sharpening intensity prediction model from multiple second prediction models based on multiple coding efficiency indicators corresponding to multiple groups of third sharpening parameters.

[0135] In the disclosed embodiment, multiple frame images are sharpened using multiple sets of sharpening intensities predicted by a second prediction model, and then encoded and the VMAF BD-rate under each set of sharpening intensities is calculated as the final subjective improvement evaluation indicator.

[0136] According to the solution of the embodiment of the present disclosure, multiple frame images of the video sample are sharpened and encoded using multiple sets of sharpening intensities. According to the VMAF BD-rate under the multiple sets of sharpening intensities, the sharpening intensity with the lowest bit rate under the same VMAF can be obtained, thereby deriving the optimal sharpening intensity prediction model.

[0137] In one possible implementation, the training method of the sharpening intensity prediction model of the embodiment of the present disclosure includes steps S210 to S260 and S261 to S264, wherein step S263: determining the coding efficiency index of multiple frame images after sharpening and encoding according to each set of third sharpening parameters, further includes the steps of:

[0138] Determining bitrate values and quality evaluation scores of a plurality of frame images after being sharpened and encoded according to each set of third sharpening parameters;

[0139] The value of the coding efficiency index is determined according to the bit rate value and the quality evaluation score; wherein the value of the coding efficiency index represents the amount of reduction in the bit rate value under the same quality evaluation score.

[0140] In the embodiment of the present disclosure, the quality evaluation score may be VMAF, and the bitrate value is also referred to as bitrate.

[0141] According to the solution of the embodiment of the present disclosure, the coding efficiency index is determined based on VMAF and bit rate values. The optimal sharpening intensity prediction model determined based on the coding efficiency index can ensure the balance between VMAF improvement and video bit rate increase during actual video preprocessing, and minimize the bit rate under the same VMAF.

[0142] In a possible implementation manner, the fifth feature of the frame image in the above embodiment of the present disclosure is obtained according to any one of the following calculation methods:

[0143] The square mean of the differences between the pixel values of multiple pixels in the frame image and the Gaussian mean of the region of the multiple pixels.

[0144] The mean of the absolute values of the differences between the pixel values of multiple pixels in the frame image and the Gaussian mean of the region of the multiple pixels.

[0145] The variance of the difference between the pixel values of multiple pixels in the frame image and the Gaussian mean of the region of the multiple pixels.

[0146] The mean value of the variance of the pixel values of each pixel point in the multiple image blocks constituting the frame image.

[0147] The mean value after exponential operation of the variance of the pixel values of each pixel point in the multiple image blocks constituting the frame image.

[0148] The mean value of the logarithm of the variance of the pixel values of each pixel point in the multiple image blocks constituting the frame image.

[0149] The pixel value of each pixel point of the multiple image blocks constituting the frame image is subjected to Hadamard transformation and the mean value of the sum of the AC high-frequency coefficients.

[0150] In the disclosed embodiment, feature selection is performed by calculating various features related to the image content on the video samples of each scene. The Spearman rank correlation coefficient (SRCC) is then used to calculate the correlation between various features and the true value of the optimal sharpening intensity. In the experiment, it was found that the features with higher correlation coefficients are:

[0151] 1. The square mean of the difference between the pixel values of all pixels in a frame and the Gaussian mean of the area of the pixel (or the absolute mean, variance, etc.).

[0152] 2. The mean of the variances of all 8x8 or 16x16 pixel blocks in a frame (or the mean after exponential operation or logarithmic operation).

[0153] 3. The mean of the sum of the AC high-frequency coefficients after Hadamard transform of all 8x8 pixel blocks in a frame.

[0154] Since the meanings of the above features are relatively similar, the final feature selected is the square mean of the difference between the pixel value and the domain Gaussian mean, because it has good prediction accuracy and VMAF BD-rate indicators in the final test and verification. In addition, since the sharpening calculation is:

[0155] Sharpened pixel value = source pixel value + sharpening intensity * (source pixel value – domain Gaussian mean)

[0156] Common sharpening methods also calculate the difference between pixel values and the Gaussian mean. It's possible to calculate features simultaneously during the sharpening process, using the features of the previous frame for prediction in each frame without requiring additional computation. The initial sharpening strength is used directly for the first frame.

[0157] According to the solution of the embodiment of the present disclosure, the fifth features obtained by the above-mentioned various calculation methods have a high correlation with the true value of the optimal sharpening intensity, and can more accurately predict the sharpening intensity.

[0158] Figure 3 This is a schematic diagram of the structure of a video sharpening device provided according to an embodiment of the disclosure. Figure 3 As shown, the device 300 at least includes:

[0159] The first prediction module 310 is configured to determine a first sharpening parameter for the k+1th frame of the target video based on a first feature of the kth frame of the target video using a sharpening strength prediction model, where k is a positive integer.

[0160] The first determining module 320 is configured to determine second features of a plurality of image blocks constituting the (k+1)th frame image.

[0161] The first adjustment module 330 is configured to determine the second sharpening parameters of the plurality of image blocks according to the first sharpening parameters and the second characteristics of the plurality of image blocks.

[0162] The first sharpening module 340 is configured to perform sharpening processing on the (k+1)th frame image according to the second sharpening parameters of the plurality of image blocks.

[0163] In a possible implementation, the first adjustment module 330 includes:

[0164] The feature determination submodule 331 is configured to determine the first feature of the (k+1)th frame image according to the second features of the plurality of image blocks.

[0165] The fine-tuning submodule 332 is configured to adjust the first sharpening parameter according to the second features of the multiple image blocks and the first features of the (k+1)th frame image to obtain the second sharpening parameters of the multiple image blocks.

[0166] In one possible implementation, the fine-tuning submodule 332 is configured to:

[0167] The third feature is determined according to the first functional relationship of the first feature of the (k+1)th frame image.

[0168] When the second feature of a first image block among the multiple image blocks is smaller than the third feature, a sharpening parameter obtained by increasing the first sharpening parameter is used as the second sharpening parameter of the first image block, where the first image block is any one of the multiple image blocks.

[0169] In one possible implementation, the fine-tuning submodule 332 is configured to:

[0170] The fourth feature is determined according to the second functional relationship of the first feature of the (k+1)th frame image.

[0171] When the second feature of a second image block among the multiple image blocks is greater than the fourth feature, a sharpening parameter obtained by reducing the first sharpening parameter is used as a second sharpening parameter of the second image block, where the second image block is any one of the multiple image blocks.

[0172] In one possible implementation, the fine-tuning submodule 332 is configured to:

[0173] The third feature is determined according to the first functional relationship of the first feature of the (k+1)th frame image.

[0174] A fourth feature is determined based on the second functional relationship of the first feature of the (k+1)th frame image, wherein the fourth feature is greater than the third feature.

[0175] When the second feature of a third image block among the multiple image blocks is smaller than the fourth feature and larger than the third feature, the first sharpening parameter is used as the second sharpening parameter of the third image block, where the third image block is any one of the multiple image blocks.

[0176] In a possible implementation, the first feature of the k-th frame image is obtained according to any of the following calculation methods:

[0177] The square mean of the difference between the pixel values of multiple pixels in the k-th frame image and the Gaussian mean of the domain of multiple pixels.

[0178] The mean of the absolute values of the differences between the pixel values of multiple pixels in the k-th frame image and the Gaussian mean of the domain of the multiple pixels.

[0179] The variance of the difference between the pixel values of multiple pixels in the k-th frame image and the Gaussian mean of the domain of the multiple pixels.

[0180] The mean value of the variance of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

[0181] The mean value after exponential operation of the variance of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

[0182] The mean value after logarithm operation of the variance of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

[0183] The mean value of the sum of the AC high-frequency coefficients after Hadamard transformation of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

[0184] In a possible implementation, the first determining module 320 is configured to:

[0185] A plurality of image blocks constituting the k+1th frame image is determined.

[0186] The intermediate result is obtained according to the difference between the pixel value of each pixel point in each image block and the Gaussian mean of the domain of each pixel point.

[0187] The second feature of each image block is obtained according to the square mean of the intermediate results.

[0188] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above-mentioned video sharpening method embodiment, which will not be repeated here.

[0189] Figure 4 Schematic diagram of the structure of the training device of the sharpening intensity prediction model provided according to an embodiment of the disclosure. Figure 4 As shown, the device 400 at least includes:

[0190] The second prediction module 410 is configured to determine predicted values of sharpening parameters of the plurality of frame images extracted from the video sample using the first prediction model according to the fifth feature of the plurality of frame images.

[0191] The second determination module 420 is configured to determine a first model parameter group of a first prediction model according to true values and predicted values of sharpening parameters of a plurality of frame images.

[0192] The parameter generation module 430 is configured to determine a plurality of second model parameter groups according to a plurality of model parameters of the first model parameter group.

[0193] The construction module 440 is configured to construct a plurality of second prediction models based on the first prediction model according to the plurality of second model parameter groups.

[0194] The third prediction module 450 is configured to determine multiple sets of third sharpening parameters for the multiple frame images using multiple second prediction models based on the fifth features of the multiple frame images.

[0195] The model determination module 460 is configured to determine a sharpening intensity prediction model from a plurality of second prediction models based on a plurality of coding efficiency indicators obtained after a plurality of frame images are sharpened according to a plurality of sets of third sharpening parameters.

[0196] In one possible implementation, the model determination module 460 includes:

[0197] The sharpening submodule 461 is configured to perform sharpening processing on a plurality of frame images using each set of third sharpening parameters in the plurality of sets of third sharpening parameters.

[0198] The encoding submodule 462 is configured to encode the multiple frame images after the sharpening process.

[0199] The index determination submodule 463 is used to determine the coding efficiency index of the multiple frame images after being sharpened and encoded according to each set of third sharpening parameters.

[0200] The comparison submodule 464 is configured to determine a sharpening intensity prediction model from the plurality of second prediction models according to the plurality of coding efficiency indicators corresponding to the plurality of sets of third sharpening parameters.

[0201] In one possible implementation, the indicator determination submodule 463 is configured to:

[0202] Determine the bit rate values and quality evaluation scores of multiple frame images after sharpening and encoding according to each set of third sharpening parameters.

[0203] The value of the coding efficiency index is determined based on the bit rate value and the quality evaluation score, wherein the value of the coding efficiency index represents the amount of reduction in the bit rate value under the same quality evaluation score.

[0204] In a possible implementation, the fifth feature of the frame image is obtained by any of the following calculation methods:

[0205] The square mean of the differences between the pixel values of multiple pixels in the frame image and the Gaussian mean of the region of the multiple pixels.

[0206] The mean of the absolute values of the differences between the pixel values of multiple pixels in the frame image and the Gaussian mean of the region of the multiple pixels.

[0207] The variance of the difference between the pixel values of multiple pixels in the frame image and the Gaussian mean of the region of the multiple pixels.

[0208] The mean value of the variance of the pixel values of each pixel point in the multiple image blocks constituting the frame image.

[0209] The mean value after exponential operation of the variance of the pixel values of each pixel point in the multiple image blocks constituting the frame image.

[0210] The mean value of the logarithm of the variance of the pixel values of each pixel point in the multiple image blocks constituting the frame image.

[0211] The pixel value of each pixel point of the multiple image blocks constituting the frame image is subjected to Hadamard transformation and the mean value of the sum of the AC high-frequency coefficients.

[0212] For the description of specific functions and examples of each module and sub-module of the device of the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above-mentioned embodiment of the training method of the sharpening intensity prediction model, which will not be repeated here.

[0213] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0214] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0215] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0216] Various components in device 500 are connected to I / O interface 505, including: an input unit 506, such as a keyboard, mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, optical disk, etc.; and a communication unit 509, such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0217] The computing unit 501 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the video sharpening method or the training method of the sharpening intensity prediction model. For example, in some embodiments, the video sharpening method or the training method of the sharpening intensity prediction model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the video sharpening method or the training method of the sharpening intensity prediction model described above can be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute a video sharpening method or a sharpening strength prediction model training method in any other appropriate manner (eg, by means of firmware).

[0218] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0219] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0220] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0221] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0222] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0223] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0224] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0225] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A video sharpening method, comprising: Determining a first sharpening parameter of the k+1th frame image of the target video using a sharpening strength prediction model according to a first feature of the kth frame image of the target video; wherein k is a positive integer; Determining second features of a plurality of image blocks constituting the k+1th frame image; determining second sharpening parameters for the plurality of image blocks according to the first sharpening parameters and second characteristics of the plurality of image blocks; and The k+1th frame image is sharpened according to the second sharpening parameters of the multiple image blocks.

2. The method according to claim 1, wherein The determining, according to the first sharpening parameter and the second features of the plurality of image blocks, second sharpening parameters of the plurality of image blocks comprises: determining a first feature of the k+1th frame image according to the second features of the plurality of image blocks; The first sharpening parameter is adjusted according to the second features of the multiple image blocks and the first features of the (k+1)th frame image to obtain second sharpening parameters of the multiple image blocks.

3. The method according to claim 2, wherein: The adjusting the first sharpening parameter according to the second features of the plurality of image blocks and the first features of the k+1th frame image to obtain the second sharpening parameters of the plurality of image blocks includes: determining a third feature according to a first functional relationship of the first feature of the k+1th frame image; When the second feature of a first image block among the multiple image blocks is smaller than the third feature, a sharpening parameter obtained by increasing the first sharpening parameter is used as the second sharpening parameter of the first image block; wherein the first image block is any one of the multiple image blocks.

4. The method according to claim 2, wherein: The adjusting the first sharpening parameter according to the second features of the plurality of image blocks and the first features of the k+1th frame image to obtain the second sharpening parameters of the plurality of image blocks includes: determining a fourth feature according to a second functional relationship of the first feature of the k+1th frame image; When the second feature of the second image block among the multiple image blocks is greater than the fourth feature, the sharpening parameter obtained by reducing the first sharpening parameter is used as the second sharpening parameter of the second image block; wherein the second image block is any image block among the multiple image blocks.

5. The method according to claim 2, wherein: The adjusting the first sharpening parameter according to the second features of the plurality of image blocks and the first features of the k+1th frame image to obtain the second sharpening parameters of the plurality of image blocks includes: determining a third feature according to a first functional relationship of the first feature of the k+1th frame image; determining a fourth feature based on a second functional relationship of the first feature of the k+1th frame image; wherein the fourth feature is greater than the third feature; When the second feature of a third image block among the multiple image blocks is smaller than the fourth feature and larger than the third feature, the first sharpening parameter is used as the second sharpening parameter of the third image block; wherein the third image block is any one of the multiple image blocks.

6. The method according to any one of claims 1 to 5, wherein The first feature of the k-th frame image is obtained by any of the following calculation methods: The square mean of the differences between the pixel values of the plurality of pixel points of the k-th frame image and the Gaussian mean of the region of the plurality of pixel points; The mean of the absolute values of the differences between the pixel values of the plurality of pixel points of the k-th frame image and the Gaussian mean of the domain of the plurality of pixel points; The variance of the difference between the pixel values of the plurality of pixel points of the k-th frame image and the Gaussian mean of the region of the plurality of pixel points; a mean value of the variance of pixel values of each pixel point of a plurality of image blocks constituting the k-th frame image; the mean value after exponential operation of the variance of the pixel values of each pixel point of the plurality of image blocks constituting the k-th frame of image; a mean value after logarithmic operation of variances of pixel values of each pixel point of a plurality of image blocks constituting the k-th frame of image; The mean value of the sum of the AC high-frequency coefficients after Hadamard transformation of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

7. The method according to any one of claims 1 to 5, wherein determining the second features of the plurality of image blocks constituting the k+1th frame image comprises: Determining a plurality of image blocks constituting the k+1th frame image; Obtaining an intermediate result based on the difference between the pixel value of each pixel point of each image block and the local Gaussian mean of the pixel points; The second feature of each image block is obtained according to the square mean of the intermediate results.

8. The method according to claim 1, wherein The sharpening strength prediction model is trained by the following steps, including: Determining predicted values of sharpening parameters of the plurality of frame images extracted from the video sample using the first prediction model according to the fifth features of the plurality of frame images; determining a first model parameter group of the first prediction model according to true values of the sharpening parameters of the plurality of frame images and the predicted values; determining a plurality of second model parameter groups according to a plurality of model parameters of the first model parameter group; constructing a plurality of second prediction models based on the first prediction model according to the plurality of second model parameter groups; Determining, according to the fifth features of the plurality of frame images, a plurality of sets of third sharpening parameters for the plurality of frame images using the plurality of second prediction models; and According to a plurality of coding efficiency indicators obtained after the plurality of frame images are sharpened according to the plurality of sets of third sharpening parameters, a sharpening intensity prediction model is determined from the plurality of second prediction models.

9. The method according to claim 8, wherein The step of determining a sharpening intensity prediction model from the plurality of second prediction models based on the plurality of coding efficiency indicators obtained after the plurality of frame images are sharpened according to the plurality of sets of third sharpening parameters comprises: performing sharpening processing on the plurality of frame images using each set of third sharpening parameters in the plurality of sets of third sharpening parameters; encoding the plurality of frame images after the sharpening process; Determining a coding efficiency index of the plurality of frame images after being sharpened and encoded according to each set of the third sharpening parameters; According to the multiple coding efficiency indicators corresponding to the multiple groups of third sharpening parameters, a sharpening intensity prediction model is determined from the multiple second prediction models.

10. The method according to claim 9, wherein: The determining of the coding efficiency indexes of the plurality of frame images after the sharpening process and encoding according to each set of the third sharpening parameters includes: Determining bitrate values and quality evaluation scores of the plurality of frame images after being sharpened and encoded according to each set of third sharpening parameters; The value of the coding efficiency indicator is determined according to the bit rate value and the quality evaluation score; wherein the value of the coding efficiency indicator represents the amount of reduction in the bit rate value under the same quality evaluation score.

11. The method according to any one of claims 8 to 10, wherein The fifth feature of the frame image is obtained by any one of the following calculation methods: The square mean of the differences between the pixel values of the plurality of pixel points of the frame image and the Gaussian mean of the region of the plurality of pixel points; The mean of the absolute values of the differences between the pixel values of the plurality of pixel points of the frame image and the Gaussian mean of the region of the plurality of pixel points; The variance of the difference between the pixel values of the plurality of pixel points of the frame image and the Gaussian mean of the region of the plurality of pixel points; a mean value of variance of pixel values of each pixel point of a plurality of image blocks constituting the frame image; a mean value after exponential operation of variances of pixel values of each pixel point of a plurality of image blocks constituting the frame image; a mean value after logarithm operation of variances of pixel values of each pixel point of a plurality of image blocks constituting the frame image; The mean value of the sum of the AC high-frequency coefficients after Hadamard transformation of the pixel values of each pixel point of the multiple image blocks constituting the frame image.

12. A video sharpening device, comprising: A first prediction module is configured to determine a first sharpening parameter of the k+1th frame image of the target video according to a first feature of the kth frame image of the target video using a sharpening strength prediction model, wherein k is a positive integer; A first determining module, configured to determine a second feature of a plurality of image blocks constituting the k+1th frame image; a first adjustment module, configured to determine second sharpening parameters for the plurality of image blocks according to the first sharpening parameters and second features of the plurality of image blocks; and The first sharpening module is configured to perform sharpening processing on the (k+1)th frame image according to the second sharpening parameters of the plurality of image blocks.

13. The device according to claim 12, wherein The first adjustment module includes: a feature determination submodule, configured to determine a first feature of the k+1th frame image based on the second features of the plurality of image blocks; The fine-tuning submodule is configured to adjust the first sharpening parameter according to the second features of the plurality of image blocks and the first features of the k+1th frame image to obtain the second sharpening parameters of the plurality of image blocks.

14. The device according to claim 13, wherein The fine-tuning submodule is used to: determining a third feature according to a first functional relationship of the first feature of the k+1th frame image; When the second feature of a first image block among the multiple image blocks is smaller than the third feature, a sharpening parameter obtained by increasing the first sharpening parameter is used as the second sharpening parameter of the first image block; wherein the first image block is any one of the multiple image blocks.

15. The device according to claim 13, wherein The fine-tuning submodule is used to: determining a fourth feature according to a second functional relationship of the first feature of the k+1th frame image; When the second feature of the second image block among the multiple image blocks is greater than the fourth feature, the sharpening parameter obtained by reducing the first sharpening parameter is used as the second sharpening parameter of the second image block; wherein the second image block is any image block among the multiple image blocks.

16. The device according to claim 13, wherein The fine-tuning submodule is used to: determining a third feature according to a first functional relationship of the first feature of the k+1th frame image; determining a fourth feature based on a second functional relationship of the first feature of the k+1th frame image; wherein the fourth feature is greater than the third feature; When the second feature of a third image block among the multiple image blocks is smaller than the fourth feature and larger than the third feature, the first sharpening parameter is used as the second sharpening parameter of the third image block; wherein the third image block is any one of the multiple image blocks.

17. The device according to any one of claims 12 to 16, wherein The first feature of the k-th frame image is obtained by any of the following calculation methods: The square mean of the differences between the pixel values of the plurality of pixel points of the k-th frame image and the Gaussian mean of the region of the plurality of pixel points; The mean of the absolute values of the differences between the pixel values of the plurality of pixel points of the k-th frame image and the Gaussian mean of the domain of the plurality of pixel points; The variance of the difference between the pixel values of the plurality of pixel points of the k-th frame image and the Gaussian mean of the region of the plurality of pixel points; a mean value of the variance of pixel values of each pixel point of a plurality of image blocks constituting the k-th frame image; the mean value after exponential operation of the variance of the pixel values of each pixel point of the plurality of image blocks constituting the k-th frame of image; a mean value after logarithmic operation of variances of pixel values of each pixel point of a plurality of image blocks constituting the k-th frame of image; The mean value of the sum of the AC high-frequency coefficients after Hadamard transformation of the pixel values of each pixel point of the multiple image blocks constituting the k-th frame image.

18. The apparatus according to any one of claims 12 to 16, wherein the first determining module is configured to: Determining a plurality of image blocks constituting the k+1th frame image; Obtaining an intermediate result based on the difference between the pixel value of each pixel point of each image block and the local Gaussian mean of the pixel points; The second feature of each image block is obtained according to the square mean of the intermediate results.

19. The device according to claim 12, wherein The sharpening strength prediction model is obtained by training using a sharpening strength prediction model training device, and the sharpening strength prediction model training device includes: a second prediction module, configured to determine predicted values of sharpening parameters of the plurality of frame images extracted from the video sample using the first prediction model based on the fifth feature of the plurality of frame images; a second determining module, configured to determine a first model parameter group of the first prediction model according to true values of the sharpening parameters of the plurality of frame images and the predicted values; a parameter generating module, configured to determine a plurality of second model parameter groups based on a plurality of model parameters of the first model parameter group; A construction module, configured to construct a plurality of second prediction models based on the first prediction model according to the plurality of second model parameter groups; a third prediction module, configured to determine, according to the fifth features of the plurality of frame images, a plurality of sets of third sharpening parameters for the plurality of frame images using the plurality of second prediction models; and The model determination module is used to determine a sharpening intensity prediction model from the multiple second prediction models based on the multiple coding efficiency indicators obtained after the multiple frame images are sharpened according to the multiple groups of third sharpening parameters.

20. The device according to claim 19, wherein The model determination module includes: a sharpening submodule, configured to perform sharpening processing on the plurality of frame images using each set of third sharpening parameters in the plurality of sets of third sharpening parameters; An encoding submodule, configured to encode the plurality of frame images after the sharpening process; an index determination submodule, configured to determine a coding efficiency index of the plurality of frame images after being sharpened and encoded according to each set of third sharpening parameters; The comparison submodule is used to determine a sharpening intensity prediction model from the multiple second prediction models based on the multiple coding efficiency indicators corresponding to the multiple groups of third sharpening parameters.

21. The device according to claim 20, wherein The indicator determination submodule is used for: Determining bitrate values and quality evaluation scores of the plurality of frame images after being sharpened and encoded according to each set of third sharpening parameters; The value of the coding efficiency indicator is determined according to the bit rate value and the quality evaluation score; wherein the value of the coding efficiency indicator represents the amount of reduction in the bit rate value under the same quality evaluation score.

22. The device according to any one of claims 19 to 21, wherein The fifth feature of the frame image is obtained by any one of the following calculation methods: The square mean of the differences between the pixel values of the plurality of pixel points of the frame image and the Gaussian mean of the region of the plurality of pixel points; The mean of the absolute values of the differences between the pixel values of the plurality of pixel points of the frame image and the Gaussian mean of the region of the plurality of pixel points; The variance of the difference between the pixel values of the plurality of pixel points of the frame image and the Gaussian mean of the region of the plurality of pixel points; a mean value of variance of pixel values of each pixel point of a plurality of image blocks constituting the frame image; a mean value after exponential operation of variances of pixel values of each pixel point of a plurality of image blocks constituting the frame image; a mean value after logarithm operation of variances of pixel values of each pixel point of a plurality of image blocks constituting the frame image; The mean value of the sum of the AC high-frequency coefficients after Hadamard transformation of the pixel values of each pixel point of the multiple image blocks constituting the frame image.

23. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images