Video sharpness assessment method, apparatus and device, storage medium, and product

Abstract

Embodiments of the present application provide a video sharpness assessment method, apparatus and device, a storage medium, and a product. The method comprises: converting a video image frame to be assessed into a gray-scale map, and performing discrete wavelet transform on the gray-scale map to obtain a high-frequency sub-band map; then performing saliency calculation on the gray-scale map to obtain a saliency map, scaling the saliency map, and converting the scaled saliency map into a weight matrix with the same size as the high-frequency sub-band map; and finally, on the basis of the weight matrix and the high-frequency sub-band map, calculating a total high-frequency energy value corresponding to the video image frame, so as to represent a sharpness assessment result of the video image frame. According to the scheme, saliency detection is introduced into video sharpness assessment, so that the video sharpness assessment focuses more on the part of an image that attracts human eyes' attention and is more in line with the human eye perception, and the accuracy of sharpness estimation is improved, thereby better feeding back the definition of videos.

Description

Video sharpness evaluation method, device, equipment, storage medium and product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 15, 2023, with application number 202310713304.1, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of video technology, and in particular to a video sharpness assessment method, apparatus, device, storage medium, and product. Background Art

[0003] Sharpness, as an indicator that reflects the sharpness of image edges and details, is a crucial component of clarity assessment. As a key indicator of video quality, video sharpness assessment is crucial for multiple aspects of video encoding, transmission, and storage. For live streaming, in particular, sharpness assessment can directly determine the clarity of the video on the viewer's end. This allows for enhanced clarity of videos with insufficient clarity to be enhanced through certain image quality enhancement techniques, improving the user experience. Sharpness estimation can also evaluate the effectiveness of image quality enhancement technology, guiding technical staff in adjusting their algorithms. Furthermore, sharpness estimation can score the massive amount of videos available to viewers of live streaming, identifying low-definition videos and providing recommendations to assist with the recommendation side's low-definition suppression strategy. Therefore, video sharpness assessment is crucial for the profitability of live streaming or short videos.

[0004] In the related art, sharpness assessment methods can be roughly divided into three categories. One is the spatial domain method, which uses mathematical parameters to represent the width of the transition zone of the image edge. The idea is that the more drastic the normal grayscale change of the edge, the narrower the edge width, the higher the image sharpness assessment value, and the clearer the image; one is the frequency domain method, which judges sharpness based on the peak value of the transformation coefficients such as discrete cosine transform and discrete wavelet transform; and one is a fusion of edge feature-based and transformation-based methods, such as using the statistical information of the image gradient histogram and the high-frequency detail information based on the wavelet transform for sharpness assessment. Hybrid methods based on the spatial domain and frequency domain have been shown to generally perform better than methods based only on edges or only on transformations, but often at the cost of increased computational complexity. The aforementioned sharpness assessment methods only consider the global information of the image, and the accuracy of sharpness estimation is low.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a video sharpness assessment method, apparatus, device, storage medium, and product, which solve the problem that existing video sharpness assessment ignores the human eye's perception of local image features. By introducing saliency detection into video sharpness assessment, video sharpness assessment focuses more on the parts of the image that attract the human eye's attention, is more in line with human eye perception, improves the accuracy of sharpness estimation, and thus better reflects the clarity of the video.

[0007] In a first aspect, an embodiment of the present application provides a method for evaluating video sharpness, the method comprising:

[0008] Converting the video image frame to be evaluated into a grayscale image, and performing discrete wavelet transform on the grayscale image to obtain a high-frequency subband image;

[0009] Calculating the saliency of the grayscale image to obtain a saliency map, and scaling the saliency map to convert it into a weight matrix of the same size as the high-frequency subband map;

[0010] Based on the weight matrix and the high-frequency sub-band map, a total high-frequency energy value corresponding to the video image frame is calculated to represent a sharpness evaluation result of the video image frame.

[0011] In a second aspect, an embodiment of the present application further provides a video sharpness assessment device, comprising:

[0012] a high-frequency sub-band determination module configured to convert the video image frame to be evaluated into a grayscale image, and perform discrete wavelet transform on the grayscale image to obtain a high-frequency sub-band image;

[0013] a saliency determination module configured to perform saliency calculation on the grayscale image to obtain a saliency map, and scale the saliency map to convert it into a weight matrix of the same size as the high-frequency subband map;

[0014] The high-frequency energy calculation module is configured to calculate the total high-frequency energy value corresponding to the video image frame based on the weight matrix and the high-frequency sub-band map to represent the sharpness evaluation result of the video image frame.

[0015] In a third aspect, an embodiment of the present application further provides a video sharpness assessment device, the device comprising:

[0016] one or more processors;

[0017] a storage device configured to store one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the video sharpness assessment method described in the embodiment of the present application.

[0019] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are configured to execute the video sharpness assessment method described in the embodiment of the present application.

[0020] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device executes the video sharpness assessment method described in the embodiment of the present application.

[0021] In an embodiment of the present application, the video image frame to be evaluated is converted into a grayscale image, and a discrete wavelet transform is performed on the grayscale image to obtain a high-frequency subband image. A saliency calculation is then performed on the grayscale image to obtain a saliency map, which is then scaled and converted into a weight matrix of the same size as the high-frequency subband map. Finally, based on the weight matrix and the high-frequency subband map, the total high-frequency energy corresponding to the video image frame is calculated to represent the sharpness assessment result of the video image frame. By introducing saliency detection into video sharpness assessment, video sharpness assessment focuses more on the parts of the image that attract the human eye, which is more consistent with human perception, improves the accuracy of sharpness estimation, and thus provides better feedback on video clarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a flow chart of a method for evaluating video sharpness according to an embodiment of the present application;

[0023] FIG2 is a schematic diagram of an exemplary three-level Haar wavelet transform;

[0024] FIG3 is a flow chart of a method for calculating the total value of high-frequency energy provided by an embodiment of the present application;

[0025] FIG4 is a structural block diagram of a video sharpness evaluation device provided in an embodiment of the present application;

[0026] FIG5 is a schematic structural diagram of a video sharpness evaluation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the embodiments of the present application, and are not intended to limit the embodiments of the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the embodiments of the present application, rather than all structures.

[0028] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0029] The video sharpness assessment method provided in the embodiment of the present application is used to assess the clarity of videos or images, and can be applied to multiple links such as video encoding, transmission, and storage. Specific application scenarios may include: video conferencing, video calls, indoor live broadcasts, outdoor live broadcasts, and short videos. Taking indoor live broadcasts as an example, it can be used to judge the real-time clarity of live videos viewed by viewers, and can also be used to assess the actual effect of live images after image quality enhancement processing. The several application scenarios listed above are only exemplary and explanatory. In actual applications, the video sharpness assessment method can also be used in video sharpness assessment in other scenarios, and the embodiment of the present application does not limit this. The present application aims to provide a video sharpness assessment method, which introduces saliency detection into video sharpness assessment, so that the video sharpness assessment pays more attention to the part of the image that attracts the attention of the human eye, is more in line with the perception of the human eye, improves the accuracy of sharpness estimation, and thus better feedbacks the clarity of the video.

[0030] In addition, the video sharpness evaluation method provided in the embodiment of the present application is intended to use the total high-frequency energy value of the video image frame as an evaluation index of video sharpness. By setting a test set, the accuracy of the evaluation index corresponding to the embodiment of the present application can be verified. For example, objective indicators and quality correlation evaluation indicators in image quality evaluation, such as SROCC (Spearman rankorder correlation coefficient) and PLCC (Pearson linear correlation coefficient), can be used to evaluate the monotonicity and linear correlation between the total high-frequency energy value and the subjective MOS (Mean Opinion Score) score. The calculation results are shown in the following table:

[0031] Therefore, the video sharpness evaluation method provided in the embodiment of the present application uses the total high-frequency energy value as the evaluation index of video sharpness, which is highly consistent with the subjective MOS score. This evaluation index has good accuracy and reliability.

[0032] Furthermore, in order to verify the stability of the evaluation index in the video time domain, the live video can be used as test data, and the video sharpness evaluation method provided by the embodiment of the present application is used to calculate the total high-frequency energy value of each image frame in the live video. By integrating and statistically analyzing the calculation results, the following conclusions can be obtained: when the video picture changes slightly, the difference in the evaluation index value between the corresponding calculated frames is small, and it can be inferred that the evaluation index has good stability in the time domain; when the video anchor is in a state of intense exercise, the video anchor will appear and leave the picture from time to time, and the difference in the evaluation index value between the corresponding calculated frames is large, and it can be inferred that the evaluation index is consistent with the volatility between the video frames; when the video background is in a changing state, the image of the video anchor in the picture remains basically unchanged, and the change in the evaluation index value between the corresponding calculated frames is small. It can be inferred that the video sharpness evaluation method provided by the embodiment of the present application introduces saliency detection into the video sharpness evaluation, which can better focus on the part of the image that attracts the attention of the human eye, is more in line with human eye perception, and meets design expectations.

[0033] In the video sharpness assessment method provided in the embodiment of the present application, the execution entity of each step can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as a mobile phone, PC (Personal Computer), tablet computer and other terminal devices, or a server and other devices, which is not limited in the embodiment of the present application.

[0034] FIG1 is a flow chart of a video sharpness evaluation method provided in an embodiment of the present application, which specifically includes the following steps:

[0035] Step S101: convert the video image frame to be evaluated into a grayscale image, and perform discrete wavelet transform on the grayscale image to obtain a high-frequency subband image.

[0036] In one embodiment, since sharpness mainly describes the clarity of the edges and details of an image, and is manifested as high-frequency information of the image, it is necessary to transform the video image signal into the frequency domain. In this embodiment of the present application, discrete wavelet transform is used to perform frequency domain transform on the video image frame. The video image frame to be evaluated is usually a color image. Since each pixel in the color image corresponds to three color channels of R, G, and B, directly performing discrete wavelet transform on the color image requires processing the three color channels separately and selecting a suitable color space for processing. This has high computational cost and low processing efficiency. Therefore, in this embodiment of the present application, the video image frame to be evaluated is first converted into a grayscale image. When performing discrete wavelet transform, each pixel value can be directly processed. The specific discrete wavelet transform can be Haar wavelet, Daubechies wavelet, etc., which is not limited by the present invention. In addition, since sharpness feedback is the high-frequency information of the image, the high-frequency sub-band image corresponding to the grayscale image can be obtained by discrete wavelet transform. The high-frequency sub-band image reflects the fluctuation value of the local signal and stores the detailed information of the image.

[0037] In one embodiment, to reference multi-scale high-frequency information and improve the accuracy of sharpness estimation, a multi-level discrete wavelet transform can be performed on the grayscale image to obtain high-frequency subband images at different scales. Optionally, performing a discrete wavelet transform on the grayscale image to obtain the high-frequency subband images includes: performing a multi-level discrete wavelet transform on the grayscale image to obtain multiple layers of high-frequency subband images, each layer of the high-frequency subband images including a horizontal high-frequency subband image, a vertical high-frequency subband image, and a diagonal high-frequency subband image. Among them, taking the three-level Haar wavelet transform as an example, Figure 2 is a schematic diagram of an exemplary three-level Haar wavelet transform. As shown in Figure 2, performing the first-level Haar wavelet transform on the grayscale image can obtain the high-frequency subband image of the first layer: horizontal high-frequency subband image 1011, vertical high-frequency subband image 1012 and diagonal high-frequency subband image 1013, as well as the low-frequency subband image. Based on the low-frequency subband image of the first layer, performing the second-level Haar wavelet transform can obtain the high-frequency subband image of the second layer: horizontal high-frequency subband image 1021, vertical high-frequency subband image 1022 and diagonal high-frequency subband image 1023, as well as the low-frequency subband image. Based on the low-frequency subband image of the second layer, performing the third-level Haar wavelet transform can obtain the high-frequency subband image of the third layer: horizontal high-frequency subband image 1031, vertical high-frequency subband image 1032 and diagonal high-frequency subband image 1033, as well as the low-frequency subband image. This yields three layers of multi-scale high-frequency subband maps, each of which can include horizontal, vertical, and diagonal high-frequency subband maps. The horizontal high-frequency subband map is a wavelet coefficient map generated by convolving the image with a low-pass wavelet filter in the row direction and then with a high-pass wavelet filter in the column direction, representing the image's horizontal singularity. The vertical high-frequency subband map is a wavelet coefficient map generated by convolving the image with a high-pass wavelet filter in the row direction and then with a low-pass wavelet filter in the column direction, representing the image's vertical singularity. The diagonal high-frequency subband map is a wavelet coefficient map generated by convolving the image with a high-pass wavelet filter in both the row and column directions, representing the image's diagonal edge characteristics. Therefore, the multi-level discrete wavelet transform can decompose a grayscale image into high-frequency information at different scales, making the high-frequency information more prominent and retaining more detail, which helps improve the accuracy of sharpness estimation.

[0038] Step S102: Calculate the saliency of the grayscale image to obtain a saliency map, and scale the saliency map to convert it into a weight matrix of the same size as the high-frequency subband map.

[0039] In one embodiment, saliency can be the degree to which the human eye attracts attention in visual perception, reflecting the visual appeal and visual quality of an image. A saliency map obtained by calculating the saliency of a grayscale image can quickly locate the portion of the grayscale image that attracts the human eye's attention. Saliency calculation can be performed using methods such as frequency domain analysis, region segmentation, or global contrast, which are not limited in this invention. The saliency map is converted into a weight matrix of the same size as the high-frequency subband map by scaling. This weight matrix is ​​used to perform a fusion calculation of the high-frequency subband map and the saliency map. The scaling method can be nearest neighbor interpolation, bilinear interpolation, or bilinear interpolation, which are not limited in this invention.

[0040] In one embodiment, since the video sharpness assessment method provided in the embodiment of the present application uses the frequency domain information of the video image frame, it is easy to ignore the spatial domain information of the video image frame. Therefore, a method based on global contrast can be used to calculate the saliency of the grayscale image. The specific implementation process includes:

[0041] Traverse each pixel of the grayscale image, and use the sum of the color distance values ​​between the current pixel and other pixels as the saliency value to calculate the saliency map. The calculation formula is as follows:

[0042] Among them, I k is the pixel currently traversed, I i represents other pixels in the grayscale image, and ||*|| represents the Euclidean distance. It can be understood that the degree of brightness difference between different regions in a grayscale image reflects the spatial information of the image. Therefore, by calculating the brightness difference between all pixels in the grayscale image, namely the global contrast, we can obtain the spatial information between different regions in the image. Based on the saliency of the reference spatial information, we can better align with the human eye's attention habits and improve the accuracy of sharpness estimation.

[0043] In one embodiment, since the grayscale image may have the same pixel value, and for the same pixel value, the calculated Euclidean distance result is the same, in order to improve the calculation efficiency, the specific implementation process of calculating the saliency of the grayscale image optionally includes:

[0044] Obtain the color value corresponding to the pixel in the grayscale image, count the frequency of occurrence of each color value; calculate the saliency value corresponding to each color value based on the frequency of occurrence of each color value and the color distance between each color value and other color values; assign the saliency value corresponding to each color value to the corresponding pixel to obtain the saliency map. The calculation formula is as follows:

[0045] Among them, I k is the pixel currently traversed, Cl is the color value corresponding to the currently traversed pixel, C j is the color value corresponding to other pixels in the grayscale image, f j is the color value C j The corresponding frequency of occurrence, ||*|| represents the Euclidean distance. It can be understood that by calculating the saliency value using color values ​​and their corresponding frequency of occurrence, there is no need to traverse every pixel in the grayscale image for calculation, which reduces computational complexity, improves the efficiency of saliency calculation, and accelerates the processing of the video image frame to be evaluated.

[0046] In one embodiment, when a grayscale image is subjected to a multi-level discrete wavelet transform, the saliency map is scaled and converted into a weight matrix of the same size as the high-frequency subband map, including:

[0047] The saliency maps are scaled and converted into weight matrices of the same size as the high-frequency subband maps in each layer.

[0048] It is understood that performing a multi-level discrete wavelet transform on a grayscale image can yield high-frequency information at multiple scales. Accordingly, a corresponding weight matrix can be assigned to the high-frequency information at each scale. Therefore, the saliency map needs to be converted into a weight matrix of the same size as the high-frequency subband maps in each layer. Thus, the present embodiment provides multi-scale saliency information corresponding to the multi-scale high-frequency information, and through multi-scale fusion, sharpness estimation is more accurate.

[0049] Step S103 : Based on the weight matrix and the high frequency subband map, a total high frequency energy value corresponding to the video image frame is calculated to represent a sharpness evaluation result of the video image frame.

[0050] The total high-frequency energy value can reflect the energy ratio of high-frequency signals in a video image frame, indicating the energy distribution of details in the video image frame. It is understood that the higher the total high-frequency energy value of a video image frame, the clearer the image details and the richer the texture information. Therefore, the total high-frequency energy value can be used as a sharpness assessment result for the video image frame.

[0051] FIG3 is a flow chart of a method for calculating the total high-frequency energy value provided by an embodiment of the present application. As shown in FIG3 , when a multi-level discrete wavelet transform is performed on a grayscale image, the specific steps of calculating the total high-frequency energy value include:

[0052] Step S1031: Based on each high-frequency sub-band graph in each layer and a weight matrix of the same size as the high-frequency sub-band graph, calculate the high-frequency energy value corresponding to each high-frequency sub-band graph in each layer;

[0053] Among them, the multi-level discrete wavelet transform corresponds to multiple layers of high-frequency sub-band graphs. The number of high-frequency sub-band graphs in each layer can be multiple, and the high-frequency energy value of the high-frequency sub-band graph can be calculated based on the weight matrix corresponding to each high-frequency sub-band graph and its own high-frequency information.

[0054] For example, the calculation formula for the high-frequency energy value corresponding to each high-frequency sub-band graph in each layer is as follows:

[0055] in, Represents the high-frequency energy value of each high-frequency sub-band graph in the nth layer, Represents each high-frequency sub-band graph in the nth layer, Represents the pixel value of the pixel located in the i-th row and j-th column in the currently calculated high-frequency subband image, Represents the element value of the i-th row and j-th column of the weight matrix corresponding to the currently calculated high-frequency subband graph, N n Represents the total number of pixels in the currently calculated high-frequency subband image. It's understandable that each pixel value in each high-frequency subband image is weighted by the corresponding element value in the weight matrix. For the saliency map, the saliency values ​​of high-frequency components are more prominent, corresponding to larger weights in the weight matrix. Therefore, the high-frequency energy values ​​calculated by combining the high-frequency subband image with the weight matrix focus more on the parts of the image that attract the human eye, which is more consistent with human perception.

[0056] Step S1032: Perform weighted calculation on the high-frequency energy values ​​corresponding to the high-frequency sub-band graphs in each layer to obtain the high-frequency energy values ​​corresponding to the high-frequency sub-band graphs in each layer.

[0057] Each layer of high-frequency subband maps can include horizontal, vertical, and diagonal high-frequency subband maps. The horizontal high-frequency subband map reflects the high-frequency portion of the grayscale image in the horizontal direction, and its high-frequency energy is generally proportional to the number and intensity of horizontal edges in the grayscale image. The vertical high-frequency subband map reflects the high-frequency portion of the grayscale image in the vertical direction, and its high-frequency energy is generally proportional to the number and intensity of vertical edges in the grayscale image. The diagonal high-frequency subband map reflects the high-frequency portion of the grayscale image in the diagonal direction, and its high-frequency energy is generally proportional to the number and intensity of diagonal edges in the grayscale image. Therefore, based on the edge distribution information of the video image frame to be evaluated in actual application scenarios, different weights can be assigned to each high-frequency subband map, thereby obtaining the high-frequency energy value of each high-frequency subband map layer.

[0058] For example, the calculation formula for the high-frequency energy value corresponding to each layer of the high-frequency sub-band graph is as follows:

[0059] Among them, E n Represents the high-frequency energy value of the high-frequency sub-band graph of the nth layer, Represents the high-frequency energy value of the n-th level high-frequency sub-band graph, Represents the high-frequency energy value of the vertical high-frequency sub-band graph of the nth layer, represents the high-frequency energy value of the nth layer's diagonal high-frequency subband map, and α represents the energy weighting coefficient. It is understood that the value of α ranges from 0 to 1. For images of people or objects in the video image frame being evaluated, when they are distributed horizontally or vertically within the frame, the number and intensity of horizontal and vertical edges in the video image frame will be more prominent than diagonal edges. Therefore, α can be set between 0 and 0.5, giving greater reference to the high-frequency energy values ​​of the horizontal and vertical high-frequency subband maps. Furthermore, for images of people or objects in the video image frame being evaluated, when they are distributed more diagonally within the frame, the number and intensity of diagonal edges in the video image frame will be more prominent than horizontal and vertical edges. Therefore, α can be set between 0.5 and 1, giving greater reference to the high-frequency energy values ​​of the diagonal high-frequency subband maps. Therefore, based on the characteristics of the video content in actual application scenarios, including the distribution of people and objects, the weights of different high-frequency subband maps can be better allocated to align with the human eye's attention habits and improve the accuracy of high-frequency energy calculation.

[0060] Step S1033: Perform weighted calculation on the high-frequency energy values ​​corresponding to each layer of the high-frequency sub-band graph to obtain a total high-frequency energy value.

[0061] High-frequency energy at different scales represents different levels of detail in the video image frame. In one embodiment, using a three-level discrete wavelet transform (DWT) as an example, the high-frequency energy values ​​of the first-level high-frequency subband map represent the coarsest detail information in the video image frame, namely, the high-frequency portion at a large scale, which contains the basic outline information of the video image frame. The high-frequency energy values ​​of the second-level high-frequency subband map are obtained by performing another DWT on the detail information at the scale of the first-level high-frequency subband map, and therefore represent more detailed detail information. The high-frequency energy values ​​of the third-level high-frequency subband map are obtained by performing another DWT on the detail information at the scale of the second-level high-frequency subband map, and therefore represent the most detailed detail information, namely, the high-frequency portion at the finest scale, which contains information such as subtle texture and noise in the image. Therefore, different weights can be set for high-frequency energy values ​​at different scales. The higher the level, the more detailed the high-frequency portion of interest, and the relatively lower its impact. Therefore, a layer-by-layer decreasing weight setting method can be used to better align with the areas of human eye attention and improve the accuracy of sharpness assessment.

[0062] For example, the calculation formula for the total high-frequency energy is as follows:

[0063] Among them, Index represents the total value of high-frequency energy, En = represents the high-frequency energy value of the n-th layer high-frequency sub-band image, and N represents the grayscale image using the N-level discrete wavelet transform. In one embodiment, taking the three-level discrete wavelet transform as an example, the calculation formula of the total high-frequency energy value is as follows:

[0064] It can be understood that, from the formula, the weight corresponding to the high-frequency energy value of the first-layer high-frequency sub-band graph is 4, the weight corresponding to the high-frequency energy value of the second-layer high-frequency sub-band graph is 2, and the weight corresponding to the high-frequency energy value of the first-layer high-frequency sub-band graph is 1. On the basis of the high-frequency energy value corresponding to the large-scale detail part as the main component of the total high-frequency energy value, the high-frequency energy value corresponding to the finer-scale detail part is added to reasonably ensure the effect of the high-frequency energy value of each layer of the high-frequency sub-band graph, so that the sharpness estimation is more accurate.

[0065] In this approach, the video frame to be evaluated is converted into a grayscale image and then subjected to a discrete wavelet transform to obtain a high-frequency subband image. A saliency calculation is then performed on the grayscale image to obtain a saliency map, which is then scaled and converted into a weight matrix of the same size as the high-frequency subband image. Finally, based on the weight matrix and the high-frequency subband image, the total high-frequency energy corresponding to the video frame is calculated to represent the sharpness assessment result of the video frame. By incorporating saliency detection into video sharpness assessment, video sharpness assessment focuses more on the image components that attract the human eye, better aligning with human perception and improving the accuracy of sharpness estimation, thereby providing better feedback on video clarity.

[0066] FIG4 is a block diagram of a video sharpness assessment device provided in an embodiment of the present application. The device is configured to execute the video sharpness assessment method provided in the above embodiment and has the corresponding functional modules and beneficial effects of the execution method. As shown in FIG4 , the device specifically includes:

[0067] The high frequency sub-band determination module 201 is configured to convert the video image frame to be evaluated into a grayscale image, and perform discrete wavelet transform on the grayscale image to obtain a high frequency sub-band image;

[0068] A saliency determination module 202 is configured to perform saliency calculation on the grayscale image to obtain a saliency map, and scale the saliency map to convert it into a weight matrix of the same size as the high-frequency subband map;

[0069] The high frequency energy calculation module 203 is configured to calculate the total high frequency energy value corresponding to the video image frame based on the weight matrix and the high frequency subband map to represent the sharpness evaluation result of the video image frame.

[0070] The above scheme demonstrates that the video image frame to be evaluated is converted into a grayscale image, which is then subjected to a discrete wavelet transform to obtain a high-frequency subband image. A saliency calculation is then performed on the grayscale image to obtain a saliency map, which is then scaled and converted into a weight matrix of the same size as the high-frequency subband image. Finally, based on the weight matrix and the high-frequency subband image, the total high-frequency energy corresponding to the video image frame is calculated to represent the sharpness assessment result of the video image frame. By incorporating saliency detection into video sharpness assessment, the video sharpness assessment focuses more on the image components that attract the human eye, which is more consistent with human perception, improving the accuracy of sharpness estimation and providing better feedback on video clarity.

[0071] In a possible embodiment, the high frequency sub-band determination module 201 is specifically configured as follows:

[0072] The grayscale image is subjected to multi-level discrete wavelet transform to obtain multiple layers of high-frequency sub-band images. Each layer of high-frequency sub-band images includes a horizontal high-frequency sub-band image, a vertical high-frequency sub-band image and a diagonal high-frequency sub-band image.

[0073] Accordingly, the saliency determination module 202 is specifically configured as follows:

[0074] The saliency maps are scaled and converted into weight matrices of the same size as the high-frequency subband maps in each layer.

[0075] In a possible embodiment, the high-frequency energy calculation module 203 is specifically configured as follows:

[0076] Based on each high-frequency sub-band graph in each layer and a weight matrix of the same size, the high-frequency energy value corresponding to each high-frequency sub-band graph in each layer is calculated;

[0077] The high-frequency energy values ​​corresponding to each high-frequency sub-band graph in each layer are weighted and calculated to obtain the high-frequency energy values ​​corresponding to the high-frequency sub-band graph in each layer;

[0078] The high-frequency energy values ​​corresponding to each layer of the high-frequency sub-band graph are weighted and calculated to obtain the total high-frequency energy value.

[0079] In a possible embodiment, the calculation formula for the high-frequency energy value corresponding to each high-frequency sub-band graph in each layer is as follows:

[0080] in, Represents the high-frequency energy value of each high-frequency sub-band graph in the nth layer, Represents each high-frequency sub-band graph in the nth layer, Represents the pixel value of the pixel located in the i-th row and j-th column in the currently calculated high-frequency subband image, Represents the element value of the i-th row and j-th column of the weight matrix corresponding to the currently calculated high-frequency subband graph, Nn Indicates the total number of pixels in the currently calculated high-frequency subband map.

[0081] In a possible embodiment, the calculation formula of the high-frequency energy value corresponding to each layer of the high-frequency sub-band graph is as follows:

[0082] Among them, E n Represents the high-frequency energy value of the high-frequency sub-band graph of the nth layer, Represents the high-frequency energy value of the n-th level high-frequency sub-band graph, Represents the high-frequency energy value of the vertical high-frequency sub-band graph of the nth layer, It represents the high-frequency energy value of the diagonal high-frequency subband graph of the nth layer, and α represents the energy weight coefficient.

[0083] In a possible embodiment, the calculation formula of the total high-frequency energy value is as follows:

[0084] Among them, Index represents the total value of high-frequency energy, E n It represents the high-frequency energy value of the high-frequency sub-band image of the nth layer, and N means that the grayscale image adopts N-level discrete wavelet transform.

[0085] In a possible embodiment, the saliency determination module 202 is specifically configured as follows:

[0086] Each pixel of the grayscale image is traversed, and the sum of the color distance values ​​between the current pixel and other pixels is used as the saliency value to calculate the saliency map.

[0087] In a possible embodiment, the saliency determination module 202 is specifically configured as follows:

[0088] Get the color value corresponding to the pixel in the grayscale image and count the frequency of occurrence of each color value;

[0089] The saliency value corresponding to each color value is calculated based on the frequency of occurrence of each color value and the color distance between each color value and other color values;

[0090] The saliency value corresponding to each color value is assigned to the corresponding pixel to obtain the saliency map.

[0091] Figure 5 is a schematic diagram of the structure of a video sharpness assessment device provided in an embodiment of the present application. As shown in Figure 5, the device includes a processor 301, a memory 302, an input device 303, and an output device 304. The device may contain one or more processors 301, with one processor 301 being used as an example in Figure 5. The processor 301, memory 302, input device 303, and output device 304 may be connected via a bus or other means, with bus connection being used as an example in Figure 5. Memory 302, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the video sharpness assessment method in the embodiment of the present application. Processor 301 executes the software programs, instructions, and modules stored in memory 302 to execute various functional applications and data processing of the device, thereby implementing the aforementioned video sharpness assessment method. Input device 303 can be configured to receive input digital or character information and generate key signal input related to user settings and function control of the device. Output device 304 may include a display device such as a display screen.

[0092] An embodiment of the present application further provides a non-volatile storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are configured to perform a video sharpness assessment method described in the above embodiment, which includes:

[0093] Converting the video image frame to be evaluated into a grayscale image, and performing discrete wavelet transform on the grayscale image to obtain a high-frequency subband image;

[0094] Calculating the saliency of the grayscale image to obtain a saliency map, and scaling the saliency map to convert it into a weight matrix of the same size as the high-frequency subband map;

[0095] Based on the weight matrix and the high-frequency sub-band map, a total high-frequency energy value corresponding to the video image frame is calculated to represent a sharpness evaluation result of the video image frame.

[0096] It is worth noting that in the embodiment of the above-mentioned video sharpness assessment device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the scope of protection of the embodiments of the present application.

[0097] In some possible implementations, various aspects of the methods provided herein may also be implemented in the form of a program product, which includes program code. When the program product is executed on a computer device, the program code is configured to cause the computer device to perform the steps of the methods according to the various exemplary embodiments of the present application described above. For example, the computer device may perform the video sharpness assessment method described in the embodiments of the present application. The program product may be implemented using any combination of one or more readable media.

Claims

1. Video sharpness evaluation method, wherein: include: Converting the video image frame to be evaluated into a grayscale image, and performing discrete wavelet transform on the grayscale image to obtain a high-frequency subband image; Calculating the saliency of the grayscale image to obtain a saliency map, and scaling the saliency map to convert it into a weight matrix with the same size as the high-frequency subband map; Based on the weight matrix and the high-frequency sub-band graph, a total value of high-frequency energy corresponding to the video image frame is calculated to represent a sharpness evaluation result of the video image frame.

2. The video sharpness evaluation method according to claim 1, wherein: The step of performing discrete wavelet transform on the grayscale image to obtain a high-frequency subband image includes: Performing a multi-level discrete wavelet transform on the grayscale image to obtain multiple layers of high-frequency sub-band images, each layer of the high-frequency sub-band image includes a horizontal high-frequency sub-band image, a vertical high-frequency sub-band image, and a diagonal high-frequency sub-band image; Accordingly, scaling the saliency map to convert it into a weight matrix of the same size as the high-frequency subband map includes: The saliency map is scaled and converted into a weight matrix having the same size as each high frequency subband map in each layer.

3. The video sharpness assessment method according to any one of claims 1 or 2, wherein: The calculating, based on the weight matrix and the high frequency sub-band graph, a total high frequency energy value corresponding to the video image frame comprises: Based on each high-frequency sub-band graph in each layer and a weight matrix of the same size, the high-frequency energy value corresponding to each high-frequency sub-band graph in each layer is calculated; The high-frequency energy values ​​corresponding to each high-frequency sub-band graph in each layer are weighted and calculated to obtain the high-frequency energy values ​​corresponding to the high-frequency sub-band graph in each layer; The high-frequency energy values ​​corresponding to each layer of the high-frequency sub-band graph are weightedly calculated to obtain the total high-frequency energy value.

4. The video sharpness evaluation method according to claim 3, wherein: The calculation formula of the high-frequency energy value corresponding to each high-frequency sub-band graph in each layer is as follows: in, Represents the high-frequency energy value of each high-frequency subband graph in the nth layer, Represents the high-frequency subband graphs in the nth layer, Represents the pixel located in the i-th row and j-th column in the currently calculated high-frequency subband image. Pixel value, Represents the element value of the i-th row and j-th column of the weight matrix corresponding to the high-frequency subband graph currently calculated, N n Indicates the total number of pixels of the high-frequency subband map currently calculated.

5. The video sharpness evaluation method according to claim 3, wherein: The calculation formula of the high-frequency energy value corresponding to each layer of the high-frequency sub-band graph is as follows: Among them, E n Represents the high-frequency energy value of the high-frequency subband graph of the nth layer, Represents the high-frequency energy value of the n-th level high-frequency subband graph, Represents the high-frequency energy value of the vertical high-frequency subband graph of the nth layer, It represents the high-frequency energy value of the diagonal high-frequency subband graph of the nth layer, and α represents the energy weight coefficient.

6. The video sharpness evaluation method according to claim 3, wherein: The calculation formula of the total high-frequency energy value is as follows: Among them, Index represents the total value of high-frequency energy, E n It represents the high-frequency energy value of the n-th layer high-frequency subband image, and N represents that the grayscale image adopts N-level discrete wavelet transform.

7. The video sharpness assessment method according to any one of claims 1 to 6, wherein: The step of calculating the saliency of the grayscale image to obtain a saliency map includes: Each pixel of the grayscale image is traversed, and during the traversal process, the sum of the color distance values ​​between the current pixel and other pixels is used as a saliency value to calculate a saliency map.

8. The video sharpness assessment method according to any one of claims 1 to 6, wherein: The step of calculating the saliency of the grayscale image to obtain a saliency map includes: Obtain color values ​​corresponding to pixels in the grayscale image, and count the occurrence frequency of each color value; The significance value corresponding to each color value is calculated based on the occurrence frequency corresponding to each color value and the color distance between each color value and other color values; The saliency value corresponding to each color value is assigned to the corresponding pixel to obtain a saliency map.

9. A video sharpness assessment device, wherein: include: A high-frequency sub-band determination module is configured to convert the video image frame to be evaluated into a grayscale image, and perform discrete wavelet transform on the grayscale image to obtain a high-frequency sub-band image; a saliency determination module configured to calculate the saliency of the grayscale image to obtain a saliency map, and Scaling the saliency map to convert it into a weight matrix of the same size as the high-frequency subband map; The high-frequency energy calculation module is configured to calculate the total high-frequency energy value corresponding to the video image frame based on the weight matrix and the high-frequency sub-band graph to represent the sharpness evaluation result of the video image frame.

10. A video sharpness assessment device, comprising: one or more processors; A storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the video sharpness assessment method described in any one of claims 1-8.

11. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the video sharpness assessment method according to any one of claims 1 to 8 when executed by a computer processor.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video sharpness assessment method according to any one of claims 1 to 8 is implemented.

Images