Image quality evaluation method, device, equipment and computer program product

Abstract

The invention discloses an image quality evaluation method, device and equipment and a computer program product, and the method comprises the steps: carrying out the brightness region decomposition of an HDR image, and determining a plurality of brightness regions of the HDR image; multi-level spatial feature extraction is performed on each brightness region, and a first feature map of each brightness region in M stages is determined, so that features of a dynamic range extreme region can be extracted, features of a standard dynamic range region can be extracted, and unique distortion of an HDR image can be accurately sensed; and further performing feature recombination on the first feature maps of the plurality of brightness regions in the same stage, determining second feature maps of M stages, the second feature maps being capable of effectively representing global spatial features of the HDR image in one stage, and performing image quality evaluation according to the second feature maps of the HDR image in M stages. The quality evaluation accuracy of the HDR image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image quality evaluation method, device, equipment and computer program product. Background Art

[0002] High Dynamic Range (HDR) is a technology used to enhance image contrast and brightness range. The difference between HDR images and traditional Standard Dynamic Range (SDR) images is that HDR images do not have a defined maximum and minimum brightness value, and their brightness value is linearly related to the physical brightness value of the real scene. Therefore, HDR images can provide richer colors and more realistic scenes in high-exposure areas and low-exposure areas, and can better reflect the visual effects in the real environment.

[0003] HDR image quality evaluation is an important step to ensure that HDR image content can present high-quality visual effects on different devices and scenes. Since the representation of pixel values ​​in HDR images is proportional to the absolute brightness of the scene, the image quality evaluation method for SDR is designed for a limited brightness range and uniformly perceived pixel values. Therefore, the quality evaluation method for SDR images is not suitable for directly evaluating HDR images.

[0004] At present, most of the image quality evaluation methods for HDR use the full-reference (FR) image quality evaluation method, which requires the original image as a reference to evaluate the quality of the HDR image, limiting its application scenarios. Summary of the invention

[0005] The embodiments of the present application are intended to provide an image quality assessment method, apparatus, device and computer program product.

[0006] The technical solution of this application is implemented as follows:

[0007] In a first aspect, a method for evaluating image quality is provided, comprising:

[0008] Decomposing the high dynamic range (HDR) image by brightness region to determine multiple brightness regions of the HDR image;

[0009] Perform spatial feature extraction on each brightness region of the HDR image, and determine the first feature map of each brightness region of the HDR image in M ​​stages, where M is a positive integer;

[0010] Performing feature recombination on first feature maps of multiple brightness regions of the HDR image at the same stage to determine second feature maps of the HDR image at M stages;

[0011] A prediction quality score of the HDR image is determined according to the second feature maps of the HDR image at the M stages.

[0012] In a second aspect, an image quality evaluation device is provided, comprising:

[0013] A decomposition module, used for performing brightness region decomposition on a high dynamic range (HDR) image to determine multiple brightness regions of the HDR image;

[0014] A first feature extraction module is used to extract spatial features of each brightness area of ​​the HDR image and determine a first feature map of each brightness area of ​​the HDR image in M ​​stages;

[0015] A recombining module, used for performing feature recombining on first feature maps of multiple brightness areas of the HDR image at the same stage, and determining second feature maps of the HDR image at M stages;

[0016] The determination module is used to determine the prediction quality score of the HDR image according to the second feature maps of the HDR image in the M stages.

[0017] In a third aspect, an image quality assessment device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0018] Wherein, when the processor is configured to run a computer program, the steps of the aforementioned method are executed.

[0019] In a fourth aspect, a computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented.

[0020] In the embodiments of the present application, an image quality assessment method, apparatus, device and computer program product are provided. The method determines multiple brightness regions of the HDR image by performing brightness region decomposition on the HDR image; performs multi-level spatial feature extraction on each brightness region, and determines the first feature map of each brightness region at M stages, so that the features of the extreme dynamic range region can be extracted, and the features of the standard dynamic range region can be extracted, and the distortion unique to the HDR image can be accurately perceived; further, the first feature maps of multiple brightness regions at the same stage are feature reorganized to determine the second feature maps of M stages, and the second feature map of each stage can effectively characterize the spatial features of the corresponding level in the global range, and the image quality is evaluated according to the second feature maps of the M stages of the HDR image, which can improve the quality evaluation accuracy of the HDR image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of a first process of the image quality evaluation method in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of an HDR image in an embodiment of the present application;

[0023] Figure 3 This is a second flow chart of the image quality evaluation method in an embodiment of the present application;

[0024] Figure 4 Schematic diagram of the third process of the image quality evaluation method in the embodiment of the present application;

[0025] Figure 5 A schematic diagram of the composition framework of an image quality evaluation model in an embodiment of the present application Figure 1 ;

[0026] Figure 6 Schematic diagram of the composition framework of the residual module in the embodiment of the present application;

[0027] Figure 7 A schematic diagram of the composition framework of an image quality evaluation model in an embodiment of the present application Figure 2 ;

[0028] Figure 8 This is a schematic diagram of the composition structure of the image quality evaluation device in the embodiment of the present application;

[0029] Fig. 9 Schematic diagram of the composition structure of the image quality evaluation device in the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0031] Figure 1 Schematic diagram of the first process of the image quality evaluation method in the embodiment of the present application. Figure 1 As shown, the method may specifically include:

[0032] Step 101: performing brightness region decomposition on the HDR image to determine multiple brightness regions of the HDR image;

[0033] Compared with SDR images, HDR images tend to be brighter in the bright areas and darker in the dark areas, and the extreme contrast between bright and dark areas is more obvious. Detail loss often occurs in different brightness areas, such as Figure 2As shown. In order to make the image quality assessment model sensitive to the distortion specific to HDR content, an embodiment of the present application provides a method for decomposing an HDR image into different regions based on brightness, detecting distortion in extreme regions of the high dynamic range by extracting features of different brightness regions respectively, and effectively retaining SDR-like features in the middle range. This method has proven to be very effective. This is in stark contrast to previous methods that used nonlinearity to enhance information in two extreme brightness regions by suppressing the mid-range, which was sensitive to specific distortion in local regions of HDR content but resulted in a lack of global information of HDR content in the model.

[0034] In some embodiments, performing brightness region decomposition on an HDR image to determine multiple brightness regions of the HDR image includes: determining one or more brightness thresholds of the HDR image based on brightness information of the HDR image; and performing brightness region decomposition on the HDR image based on the one or more brightness thresholds to determine multiple brightness regions of the HDR image.

[0035] The brightness threshold is used to decompose the brightness of the HDR image to determine different brightness areas. The brightness threshold is the upper or lower brightness limit of a brightness area. The brightness threshold can be pre-set, or determined based on the current environment brightness information, or determined based on the brightness information of the image.

[0036] In some embodiments, a logarithmic transformation is used to decompose the HDR image into different areas based on brightness. Exemplarily, one or more brightness thresholds of the HDR image are determined based on the brightness information of the HDR image, including: determining the logarithmic mean of the brightness samples of the HDR image; determining the logarithmic value of the maximum brightness of the HDR image, and the logarithmic value of the minimum brightness of the HDR image; determining the brightness decomposition parameters of the HDR image based on the logarithmic mean of the brightness samples of the HDR image, the logarithmic value of the maximum brightness, and the logarithmic value of the minimum brightness; determining the first brightness threshold and the second brightness threshold of the HDR image based on the brightness decomposition parameters of the HDR image, and the maximum brightness and minimum brightness of the HDR image.

[0037] The brightness samples may be brightness components of all pixels in the HDR image, or may be brightness components of some reference pixels in the HDR image.

[0038] The brightness decomposition parameter is a parameter used to determine the brightness threshold of the HDR image. Exemplarily, the brightness decomposition parameter of the HDR image, and the maximum brightness and minimum brightness of the HDR image are used as input variables of the first function, and the output variable is used as the first brightness threshold; the brightness decomposition parameter of the HDR image, and the maximum brightness and minimum brightness of the HDR image are used as input variables of the second function, and the output variable is used as the second brightness threshold.

[0039] The first brightness threshold and the second brightness threshold are used to decompose the brightness of the HDR image to determine different brightness areas. Exemplarily, if the brightness value of the current pixel is less than or equal to the first brightness threshold, it is determined that the current pixel belongs to the first brightness area; if the brightness value of the current pixel is greater than the first brightness threshold and less than or equal to the second brightness threshold, it is determined that the current pixel belongs to the second brightness area; if the brightness value of the current pixel is greater than the second brightness threshold, it is determined that the current pixel belongs to the third brightness area.

[0040] In some embodiments, the method further comprises: converting the HDR image from the RGB color space to the XYZ color space; or converting the HDR image from the RGB color space to the YUV color space. Performing brightness region decomposition on the HDR image based on the brightness component of the HDR image.

[0041] Step 102: extracting spatial features of each brightness region of the HDR image, and determining first feature maps of each brightness region of the HDR image in M ​​stages, where M is a positive integer;

[0042] In some embodiments, a static spatial feature extraction module is used to perform multi-stage spatial feature extraction on each brightness region of the HDR image, and different levels of features gradually extracted in different layers of the network are determined, that is, the first feature map of M stages. The first feature map represents the spatial features of a brightness region of the HDR image at one stage.

[0043] The static spatial feature extraction module is used to extract multi-level spatial features of each brightness region. The static spatial feature extraction module includes M stages, each of which is used to extract spatial features of one level. For example, M is an integer greater than 1, such as 2, 3, 4, etc. M can also be 1.

[0044] In some embodiments, the static spatial feature extraction module may be a pre-trained static spatial feature extraction module, which already has excellent performance in spatial feature extraction and can be directly used for multi-level spatial features in the image quality evaluation process.

[0045] Exemplarily, the static spatial feature extraction module is a pre-trained ResNeXt-101 network. The ResNeXt-101 network has demonstrated excellent performance in the multi-stage feature extraction process of images through its unique group convolution structure and residual connection mechanism. From the primary stage to the advanced stage, the network gradually extracts features of different levels of the image, providing strong support for subsequent image processing and recognition tasks.

[0046] Step 103: performing feature reorganization on the first feature maps of multiple brightness areas of the HDR image at the same stage to determine the second feature maps of the HDR image at M stages;

[0047] The second feature map represents the global spatial features of the HDR image at one stage. The first feature map contains the spatial features of different brightness areas at one stage. After feature recombination, the global spatial features of the HDR image at each stage can be constructed.

[0048] For example, when M is 4 and each brightness area contains the first feature maps of four stages, the first feature maps of the three brightness areas of high, medium and low in the jth stage are feature reorganized to obtain the second feature map of the HDR image in the jth stage; further, the image quality is evaluated using the spatial features in the global range of the four stages, while taking into account both the global distortion features and the multi-level spatial features, which can improve the accuracy of the quality evaluation of the HDR image.

[0049] In some embodiments, feature reorganization is performed on first feature maps of multiple brightness regions of an HDR image at the same stage to determine second feature maps of the HDR image at M stages, including: based on a channel attention mechanism, processing the first feature maps of the brightness region at each stage to obtain intermediate feature maps of the brightness region at each stage; merging the intermediate feature maps of multiple brightness regions at the same stage to obtain second feature maps of the HDR image at M stages.

[0050] Based on the channel attention mechanism, the first feature map of the brightness area at each stage is processed, and the feature channels with rich information and the recalibrated features are further emphasized to obtain a more comprehensive representation of the static spatial features in each brightness area. The intermediate feature maps of multiple brightness areas after feature calibration at the same stage are merged to obtain a more comprehensive representation of the static spatial features of the entire HDR image at each stage.

[0051] In some embodiments, the first feature map of the decomposed brightness area is reorganized through a residual aggregation (RA) module to construct the final static spatial features of each stage, that is, the second feature map.

[0052] The RA module includes one or more channel attention (CA) modules and a merging module. The CA module is used to process the first feature map of the brightness area at each stage to obtain an intermediate feature map; the merging module is used to merge the intermediate feature maps of multiple brightness areas after feature calibration at the same stage.

[0053] Step 104: Determine a prediction quality score of the HDR image according to the second feature maps of the HDR image at the M stages.

[0054] Since the second feature map represents the global spatial features of the HDR image at one stage, performing image quality evaluation on the second feature maps of the HDR image at M stages can improve the quality evaluation accuracy of the HDR image and enhance the performance of the HDR image.

[0055] The image quality assessment method provided in the embodiment of the present application can be applied to video quality assessment (Video Quality Assessment, VQA) tasks, and can also be used for single-frame image quality assessment (Image Quality Assessment, IQA) tasks.

[0056] In the case where the image quality assessment method is used to perform video image quality assessment tasks, such as Figure 3 As shown, determining the prediction quality score of the HDR image according to the second feature map of the HDR image in the M stages may include:

[0057] Step 301: performing motion perception on second feature maps of multiple HDR images in an HDR video at M stages, and determining third feature maps of the HDR images at M stages;

[0058] The HDR image is any HDR image or HDR image sample in the HDR video to be evaluated. Based on the predicted quality scores of all HDR images or some HDR image samples in the HDR video, the predicted quality score of the HDR video is determined.

[0059] In some embodiments, motion perception is performed based on second feature maps of the current HDR image and its reference HDR image in the HDR video at the same stage to determine third feature maps of the current HDR image at M stages.

[0060] Among them, the current HDR image and its reference HDR image can be the previous frame image, or the next frame image, or images at other reference positions. Motion perception is performed by comparing the difference in the second feature maps of the two frames at the same stage to obtain a third feature map. The third feature map characterizes the motion perception characteristics of the current HDR image relative to the reference HDR image, captures the time domain characteristics of continuous frames of the HDR video, and enables interactive communication along the time dimension at the scale of frame distortion.

[0061] Step 302: performing feature aggregation on the second feature map and the third feature map of the HDR image at the same stage to determine the fourth feature map of the HDR image at M stages;

[0062] By performing feature aggregation on the second feature map representing the spatial feature and the third feature map representing the temporal feature, a fourth feature map representing the temporal and spatial features of the HDR image is obtained.

[0063] Step 303: Determine a predicted quality score of the HDR video according to the fourth feature maps of the multiple HDR images in the M stages.

[0064] In some embodiments, a predicted quality score of an HDR video is determined based on fourth feature maps of multiple HDR images at M stages, including: position encoding the fourth feature maps of the multiple HDR images at the same stage to determine a fifth feature map of the multiple HDR images at the same stage; performing spatiotemporal feature extraction on the fifth feature maps of the multiple HDR images at the same stage to determine a sixth feature map of the multiple HDR images at the M stages; fusing the sixth feature maps of the HDR images at the M stages to determine a seventh feature map of the HDR images; and determining the predicted quality score of the HDR video based on the seventh feature map of the multiple HDR images.

[0065] In order to be able to perceive the positions of different images in the HDR video, the embodiment of the present application can also use position coding to record the position of each image, so that the model can capture the relative position relationship between different images in the image sequence, thereby more accurately understanding the input image sequence. For example, sinusoidal position coding is used to record the position of each image.

[0066] The fifth feature map can be understood as a feature map after position encoding. The fifth feature map is subjected to spatiotemporal feature extraction and recalibration to obtain a more comprehensive representation of the spatiotemporal features of each HDR image at each stage, i.e., the sixth feature map. Furthermore, the sixth feature maps of each HDR image at all stages are fused to obtain the seventh feature map of each HDR image, which represents the spatiotemporal features of the HDR image.

[0067] In some embodiments, the predicted quality score of the HDR video is determined based on the seventh feature map of multiple HDR images, or the predicted quality score of the HDR video is determined based on the predicted quality scores of multiple HDR images, and the predicted quality score of each HDR image is determined based on the seventh feature map of the HDR image.

[0068] In some embodiments, a predicted quality score of the HDR image is determined based on the seventh feature map of the HDR image by a Multi-Layer Perceptron (MLP).

[0069] VQA methods for HDR videos are an important part of ensuring that HDR video content can present high-quality visual effects on different devices and scenarios. Since the representation of pixel values ​​in HDR videos is proportional to the absolute brightness of the scene, VQA methods for SDR videos are designed for limited brightness ranges and uniformly perceived pixel values. Therefore, VQA methods for SDR videos are not suitable for directly evaluating HDR video quality.

[0070] Currently, most VQA methods for HDR videos use full-reference (FR) VQA methods, which require the original video as a reference and evaluate the HDR video quality by comparing the difference between the processed video and the original video, which limits its application scenarios.

[0071] However, by adopting the technical solution provided in the embodiment of the present application, there is no need to use the original video as a reference. Based on brightness decomposition and reorganization, multi-level spatial features can be extracted, and interactive communication can be performed on the scale of frame distortion along the time dimension, effectively learning the interaction between spatial distortion features and motion perception features, and improving the prediction accuracy of HDR video quality.

[0072] In some embodiments, the method includes: inputting the HDR image into an image quality assessment model, outputting a predicted quality score of the HDR image, the image quality assessment model being used to perform the steps of the image quality assessment method, the image quality assessment model including one or more modules;

[0073] In some embodiments, during the training process of the image quality assessment model, the method further includes: calculating a loss function based on the predicted quality score and the reference quality score; and optimizing the parameters of the first module of the image quality assessment model based on the loss function.

[0074] The loss function includes, but is not limited to, one or more of the sum of absolute difference (SAD), sum of absolute transformed difference (SATD), sum of squared error (SSE), mean absolute difference (MAD), mean absolute error (MAE), mean squared error (MSE), etc.

[0075] The first module includes some or all modules in the image quality assessment model. When the first module includes some modules in the image quality assessment model, the image quality assessment model also includes a second module, and the second module can be a pre-trained module. During the training process of the image quality assessment model, the parameters of the second module remain unchanged, and only the parameters of the first module are updated.

[0076] In the case where the image quality assessment method is used to perform image quality assessment tasks, such as Figure 4 As shown, determining the prediction quality score of the HDR image according to the second feature map of the HDR image in the M stages may include:

[0077] Step 401: extracting features from the second feature maps of the HDR image at the M stages, and determining the eighth feature maps of the HDR image at the M stages;

[0078] Step 402: Fusing the eighth feature maps of the HDR image in the M stages to determine a ninth feature map of the HDR image;

[0079] The second feature maps of the HDR image at the M stages are subjected to feature extraction and the spatial features are calibrated to obtain a more comprehensive representation of the spatial features of the HDR image at each stage, namely, the eighth feature map. Furthermore, the eighth feature maps of the HDR image at all stages are fused to obtain the ninth feature map of the HDR image, which represents the spatial features of the HDR image.

[0080] Step 403: Determine a predicted quality score of the HDR image according to the ninth feature map of the HDR image.

[0081] In some embodiments, a predicted quality score of the HDR image is determined based on a ninth feature map of the HDR image by the MLP.

[0082] In some embodiments, the second feature maps of the HDR image at M stages are fused to determine a tenth feature map of the HDR image; and a predicted quality score of the HDR image is determined based on the tenth feature map of the HDR image.

[0083] It is understandable that for IQA tasks, the continuous frame motion perception extraction function can be deleted to achieve objective quality evaluation of single-frame HDR images.

[0084] By adopting the technical solution provided in the embodiment of the present application, there is no need to use the original image as a reference. Based on brightness decomposition and recombination, multi-level spatial features can be extracted, spatial distortion features can be effectively learned, and the prediction accuracy of HDR image quality can be improved.

[0085] In order to better reflect the purpose of this application, based on the above embodiments of this application, further examples are given by taking the VQA task as an example.

[0086] For VQA tasks, the embodiment of the present application provides a reference-free VQA model based on brightness decomposition and recombination, which can provide excellent performance for HDR videos, called HDR-DRVQA, which is used to implement the steps of the image quality evaluation method provided in the embodiment of the present application. The overall framework diagram is as follows: Figure 5 shown.

[0087] HDR-DRVQA consists of five parts: brightness-based video image decomposition, static spatial feature (such as blur, overexposure and compression artifacts) extractor, video image region reconstruction module, dynamic motion perception extraction module and multi-layer perceptron regression quality score module. In one example, a pre-trained static spatial feature extractor ResNeXt-101 network is used to extract a hierarchical representation of spatial distortion in a decomposed HDR video image, and the extractor has been shown to have excellent performance in spatial feature extraction. Based on the Transformer encoder's powerful ability to capture long-range dependencies, the embodiment of the present application improves the design of its token to make it suitable for video quality assessment tasks, enabling it to communicate interactively along the temporal dimension at the scale of frame distortion, and effectively learn the interaction between spatial distortion features and motion perception features. A detailed description of each part is given below.

[0088] 1. Image Decomposition

[0089] Compared with SDR images, HDR images tend to be brighter in highlights and darker in shadows, and the extreme contrast between light and dark areas is more obvious. Detail loss often occurs in different brightness areas, such as Figure 2 As shown. In order to make the image quality assessment model sensitive to the distortion specific to HDR content, an embodiment of the present application provides a method for decomposing an HDR image into different regions based on brightness using a logarithmic transformation, detecting distortion in extreme regions of the high dynamic range by extracting features of different brightness regions respectively, and effectively retaining SDR-like features in the middle range. This method has proven to be very effective. This is in stark contrast to previous methods that used nonlinearity to enhance information in two extreme brightness regions by suppressing the mid-range, which was sensitive to specific distortion in local regions of HDR content but resulted in a lack of global information of HDR content in the model.

[0090] The embodiment of the present application introduces a brightness decomposition strategy based on logarithmic calculation for preprocessing HDR images. By decomposing the HDR image into different brightness regions, the information in the extreme areas of the dynamic range is emphasized, while the middle area of ​​the main standard (similar to SDR) feature information is effectively retained. Since the local brightness / color extreme areas usually occupy a small spatial area, by dividing the high / low brightness areas for targeted processing, it is possible to effectively prevent the model from ignoring the distorted feature information in the local brightness extreme areas.

[0091] Exemplarily, the application of the brightness decomposition strategy is as follows: The process of converting the RGB space to the XYZ space utilizes the linear tristimulus value data. This can be achieved by converting them to the XYZ space, which can be achieved by implementing M sRGB The following equation illustrates this process:

[0092]

[0093] Among them, the color matrix M sRGB It is defined by the International Commission on Illumination (CIE) as follows:

[0094]

[0095] Then, the brightness decomposition parameters are calculated using the logarithmic mean and logarithmic values ​​associated with the HDR image brightness value Y Brightness decomposition parameters The calculation method is as follows:

[0096]

[0097] Among them, Y max is the maximum brightness value, logYmax is the logarithmic value of the maximum brightness, Y min is the minimum brightness value, logY min is the logarithmic value of the maximum brightness, Y av Y is the logarithmic mean of all brightness in the HDR image. av The exact calculation method is as follows:

[0098]

[0099] Where W is the total number of pixels in the image, Y is the brightness channel of the image in XYZ space, and θ is a small positive number to avoid calculation anomalies caused by black pixels.

[0100] Finally, the parameters are decomposed according to brightness Calculate the first brightness threshold L of the low brightness area respectively low and the second brightness threshold L of the high brightness area hig h. Used to decompose the brightness region of the HDR image. The first brightness threshold L low The first function and the second brightness threshold L hig The specific expression of the second function of h is as follows:

[0101]

[0102] Among them, λ1 is a hyperparameter that controls the ratio of different brightness partitions, L max and L min They are the maximum and minimum values ​​of the video image after normalization between 0 and 1.

[0103] L low It can be understood as the upper limit of brightness in the low brightness area, L hig h can be understood as the lower limit of the brightness of the high brightness threshold, which will be less than L lowThe area larger than L is decomposed into low brightness areas. hig h area is decomposed into high brightness areas, and the area between L low and L hig The brightness between h is decomposed into medium brightness areas and respectively input into the static space feature extraction module f for distortion information extraction.

[0104] 2. Static spatial feature extraction

[0105] For VQA tasks, static spatial features are considered to focus on appearance degradation, while static spatial features are expected to indicate appearance degradation (such as blur, overexposure, and compression artifacts), which is actually the task of IQA, especially for scenes in the wild. Therefore, the ResNeXt-101 network pre-trained based on image quality assessment is used as a static spatial feature extractor to efficiently extract static spatial features based on the brightness decomposition of video images. Then, the features of the decomposed video image are recombined and fused into the static spatial features of the final video image through the RA module. The main reason for using an image quality-based extractor is that a model pre-trained in an image quality database with more labeled data can greatly reduce the risk of overfitting problems compared to learning an extractor from videos with insufficient labeled data.

[0106] For the input data X of T frame, T}, and the HDR image x i Data representing an HDR image of any frame in T frames, where i is a frame index or frame identifier. i The brightness decomposition strategy is used to decompose the brightness into three areas: low brightness area x il , medium brightness area x im and high brightness area x i h. Then, x il 、x im and x i h are fed to the static spatial feature extraction module f with pre-trained weights θ. Therefore, the image x i The first feature map S of the jth stage i,j It can be obtained as follows:

[0107]

[0108] Where j = 1, 2, ..., M, and are respectively i The first feature maps of the corresponding low brightness area, medium brightness area and high brightness area at the jth stage.

[0109] For the image quality assessment task, the HDR image x iThe image data representing the frame to be evaluated, where i is the frame index or frame identifier.

[0110] Then, the first feature map of the decomposed brightness area is reorganized through the residual aggregation (RA) module to construct the final static spatial features of each stage, namely the second feature map. This helps to obtain a complete static spatial feature representation of the HDR image for quality score regression. In an example, the specific architecture of the RA module is as follows Figure 6 As shown. Represents image x i The set of features of different brightness regions in the jth stage of i Representation of different brightness regions at stage j and It is expressed as:

[0111]

[0112] Among them, CA is the channel attention module, FC is the embedding function with two fully connected layers (FullConnection), Indicates feature concatenation. Figure 6 As shown in FIG. 1 , the CA module at least includes: a convolutional layer (Conv), an activation layer (Relu), a squeeze and excitation (Squeeze and Excite, SE) module, and an element sum (Element Sum), wherein the SE module at least includes: a global pooling layer (Global Pooling), a convolutional layer (Conv F U ), activation layer (Relu), convolutional layer (Conv F D ), activation function (Sigmoid) and element-wise product (Element Product).

[0113] This application introduces a Squeeze and Excite (SE) module in the CA module to explicitly model the CA module, so that the module further emphasizes information-rich feature channels and recalibrates features to obtain a more comprehensive representation of static spatial features in each brightness area of ​​the video image.

[0114] Among them, the input F of the CA module in They are and The output of the CA module F out Expressed as

[0115]

[0116] Here, three CA modules are directly inserted into the RA module to improve the calibration of static spatial feature information in each of the three brightness regions. Therefore, the final HDR image x i The second feature map is expressed as:

[0117]

[0118] in, is the image x i The second feature map at the j-th stage includes global spatial features of the HDR image.

[0119] 3. Motion-aware extraction of consecutive frames

[0120] The quality of the video is usually seriously degraded due to dynamic degradation such as camera shake and light flicker. On the other hand, frame difference has almost the same visual appearance as the optical flow map, and frame difference is simple and effective in representing motion perception. Therefore, this application performs motion perception by calculating the difference of the second feature map of consecutive frames, and determines the third feature map containing the video motion feature. The calculation formula is as follows:

[0121]

[0122] in, is the image x i Compared to image x i-1 The third feature map at the jth stage.

[0123] The extracted multi-stage static spatial features and dynamic perception features are aggregated through feature cascade operations, and the feature dimensions of different stages are reduced to a fixed length C through a fully connected (FC) layer. The fourth feature map can be expressed as:

[0124]

[0125] Among them, FC j represents the j-th FC layer, which is used to reduce the feature dimension of the j-th stage.

[0126] Further, the fourth feature maps of the multiple HDR images at the M stages are input into a multi-layer perceptron regression quality score module to output a predicted quality score of the HDR video. The multi-layer perceptron regression quality score module includes a Transformer encoder and a Multi-Layer Perceptron (MLP).

[0127] In order to model the long distance of HDR video, this application introduces a Transformer encoder to model the long-distance correlation based on the acquisition of multi-scale HDR video features. Due to computational complexity and scale ambiguity, this application designs a layered Transformer encoder that takes the spatiotemporal distortion features from the same stage as input, rather than simultaneously inputting the Transformer encoder of all spatiotemporal distortion features from different stages. Features at different layers contain different contextual information, making it easier for the Transformer encoder to capture long-range spatiotemporal interactions from all parties. The multi-scale processing scheme also cooperates with the HVS mechanism to provide more stable and accurate quality modeling. In order to enable the Transformer encoder to perceive the position, sinusoidal position encoding is also used as a position recorder for each frame.

[0128] In order to be able to perceive the positions of different images in the HDR video, the embodiment of the present application can also use position coding to record the position of each image, for example, using sinusoidal position coding to record the position of each frame. j )The formula is as follows:

[0129]

[0130] Among them, [·] is the token connection, and denote the input features and position embedding of layer j, respectively, X j Represents the fifth feature map of the jth stage after position encoding.

[0131] The Transformer encoder includes a normalization layer (Layer Norm, LN), a multi-head self-attention (Multi-headSelf-Attention, MSA) and an MLP. The Transformer encoder can be expressed as

[0132]

[0133] Where d = {1, 2, ..., D} is the index of the Transformer encoder and the total number of D. In addition, is the intermediate token of the dth token in the jth stage. After obtaining the output embeddings of different stages, a fusion operation is performed by adding the feature levels, and the final quality score is calculated by regressing the quality mark using MLP, which is calculated as follows

[0134]

[0135] in, to Represents the sixth feature map of M stages, Xi Represents the seventh feature map of the HDR image, X i As input to MPL, output Scoring the predicted quality of HDR videos.

[0136] 4. Loss Function

[0137] In the process of training the HDR-DRVQA model provided in this application, the MAE loss function is also applied to optimize the parameters of the Transformer encoder and MLP. The predicted quality score is expressed as Denote the reference quality score as Where n is the index of the batch size B, and the reference quality score can be the subjective quality score. The mathematical expression of the loss function is as follows

[0138]

[0139] For the IQA task, the embodiment of the present application provides a reference-free IQA model based on brightness decomposition and recombination, which can provide excellent performance for HDR images. The overall framework diagram is as follows: Figure 7 As shown, it can be seen that compared with the VQA task, the IQA task can delete the continuous frame motion perception extraction function and realize objective quality evaluation of single-frame HDR images.

[0140] The corresponding continuous frame motion perception extraction step can be replaced by:

[0141] Convert the HDR image to i The second characteristic map Input to the Transformer encoder to obtain the eighth feature map of the HDR image in M ​​stages to The ninth feature map X is obtained by adding the feature levels to perform the fusion operation. i , and use MLP to regress the quality markers to calculate the final quality score

[0142] The image quality evaluation method provided in the embodiment of the present application and the IQA / VQA method in the related technology are evaluated using the existing HDR video subjective quality dataset LIVE-HDR (the LIVE-HDR dataset contains 279 HDR video samples and 31 reference HDR video samples). The related technologies specifically include: SDR image quality evaluation method RAPIQUE; HDR image quality evaluation method HIGRADE; HDR video quality evaluation methods TLVQM, HDR BVQM and VSFA. The results are shown in Table 1.

[0143] Metrics SRCC PLCC RAPIQUE 0.4553 0.4864 HIGRADE 0.7088 0.6827 TLVQM 0.5781 0.5552 HDR BVQM 0.6020 0.5844 VSFA 0.7127 0.6918 HDR-DRVQA (this application) 0.8150 0.8244

[0144] Among them, the Spearman Rank Order Correlation Coefficient (SRCC) is used to evaluate the monotonicity of the model prediction results, that is, whether the predicted value and the actual value maintain a consistent increase or decrease trend. The value range of SRCC is -1 to 1, -1 indicates a complete negative correlation, and 1 indicates a complete positive correlation.

[0145] The Pearson Linear Correlation Coefficient (PLCC) is mainly used to evaluate the accuracy of model prediction, that is, the degree of linear correlation between the predicted value and the actual value. The value range of PLCC is -1 to 1, where -1 indicates a complete negative correlation and 1 indicates a complete positive correlation.

[0146] By exploring the relationship between brightness regions and HDR content, an HDR-DRVQA model for decomposition and reorganization of high dynamic range (HDR) content is proposed. Specifically, the HDR image content is decomposed into different brightness regions to explicitly extract the perceptual features that appear in HDR. Then, a residual aggregation (RA) module is proposed to reorganize multi-region features to extract static spatial distortion representations. In addition, a local residual module RA and an improved hierarchical feature aggregation Transformer encoder are used to explicitly extract motion-perceptual feature representations to exploit the long-range dependencies of spatial and temporal embeddings. Experimental results show that the HDR-DRVQA model provided in the embodiment of the present application has significantly improved performance on the HDR video benchmark, and is highly correlated with human judgment of HDR video quality.

[0147] To implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides an image quality evaluation device, such as Figure 8 As shown, the image quality evaluation device 80 includes:

[0148] The decomposition module 801 is used to perform brightness region decomposition on the high dynamic range (HDR) image to determine multiple brightness regions of the HDR image;

[0149] A first feature extraction module 802 is used to extract spatial features of each brightness area of ​​the HDR image, and determine a first feature map of each brightness area of ​​the HDR image in M ​​stages;

[0150] A recombining module 803 is used to perform feature recombining on first feature maps of multiple brightness areas of the HDR image at the same stage to determine second feature maps of the HDR image at M stages;

[0151] The determination module 804 is configured to determine a prediction quality score of the HDR image according to the second feature maps of the HDR image at the M stages.

[0152] In some embodiments, the decomposition module 801 is used to determine one or more brightness thresholds of the HDR image according to the brightness information of the HDR image; and perform brightness region decomposition on the HDR image according to the one or more brightness thresholds to determine multiple brightness regions of the HDR image.

[0153] In some embodiments, the decomposition module 801 is used to determine the logarithmic mean of the brightness samples of the HDR image; determine the logarithmic value of the maximum brightness of the HDR image and the logarithmic value of the minimum brightness of the HDR image; determine the brightness decomposition parameters of the HDR image based on the logarithmic mean of the brightness samples of the HDR image, the logarithmic value of the maximum brightness and the logarithmic value of the minimum brightness; determine the first brightness threshold and the second brightness threshold of the HDR image based on the brightness decomposition parameters of the HDR image and the maximum brightness and minimum brightness of the HDR image.

[0154] In some embodiments, the first feature extraction module 802 is a pre-trained static spatial feature extraction module.

[0155] In some embodiments, the reorganization module 803 is used to process the first feature map of the brightness area at each stage based on the channel attention mechanism to obtain the intermediate feature map of the brightness area at each stage; merge the intermediate feature maps of multiple brightness areas at the same stage to obtain the second feature maps of the HDR image at M stages.

[0156] In some embodiments, the image quality assessment device 80 further includes: a second feature extraction module ( Figure 8 (not shown) is used for, when the image quality assessment method is used to perform a video image quality assessment task, performing motion perception on the second feature maps of multiple HDR images in the HDR video at M stages, and determining the third feature maps of the HDR images at the M stages; performing feature aggregation on the second feature map and the third feature map of the HDR image at the same stage, and determining the fourth feature map of the HDR image at the M stages;

[0157] The determination module 804 is configured to determine a predicted quality score of the HDR video according to the fourth feature maps of the plurality of HDR images at the M stages.

[0158] In some embodiments, the second feature extraction module is used to perform motion perception based on the second feature maps of the current HDR image and its reference HDR image in the HDR video at the same stage, and determine the third feature maps of the current HDR image at M stages.

[0159] In some embodiments, the determination module 804 is used to perform position encoding on the fourth feature maps of multiple HDR images at the same stage, and determine the fifth feature maps of the multiple HDR images at the same stage; perform spatiotemporal feature extraction on the fifth feature maps of the multiple HDR images at the same stage, and determine the sixth feature maps of the multiple HDR images at M stages; fuse the sixth feature maps of the HDR images at the M stages to determine the seventh feature map of the HDR image; and determine the predicted quality score of the HDR video based on the seventh feature maps of the multiple HDR images.

[0160] In some embodiments, the determination module 804 is used to perform feature extraction on the second feature map of the HDR image at M stages to determine the eighth feature map of the HDR image at M stages when the image quality assessment method is applied to a single-frame image quality assessment task; fuse the eighth feature map of the HDR image at M stages to determine the ninth feature map of the HDR image; and determine the predicted quality score of the HDR image based on the ninth feature map of the HDR image.

[0161] In some embodiments, the image quality assessment device 80 further includes: inputting the HDR image into an image quality assessment model, outputting a predicted quality score of the HDR image, the image quality assessment model being used to perform the steps of the image quality assessment method, the image quality assessment model comprising one or more modules;

[0162] The image quality assessment device 80 also includes: a training module, which is used to calculate a loss function according to the predicted quality score and the reference quality score during the training process of the image quality assessment model; and optimize the parameters of the first module of the image quality assessment model according to the loss function.

[0163] The first module is the determination module 804, which includes a conversion encoder and a multi-layer perceptron.

[0164] In practical applications, the above device can be a terminal device for implementing the image quality evaluation method, or a chip applied to the terminal device. In the present application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the image quality evaluation method provided in any of the above embodiments. The technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solution in the image quality evaluation method, and this application will not repeat them one by one.

[0165] Based on the hardware implementation of each unit in the above-mentioned image quality evaluation device, the embodiment of the present application also provides an image quality evaluation device, such as Fig. 9 As shown, the image quality assessment device 90 includes: a processor 901 and a memory 902 configured to store a computer program that can be run on the processor;

[0166] The processor 901 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0167] Of course, in practical applications, Fig. 9 As shown, the various components in the image quality evaluation device 90 are coupled together through a bus system 903. It can be understood that the bus system 903 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 903 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus system X03 in the figure.

[0168] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0169] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0170] The image quality evaluation device may be a chip, and optionally, the chip may further include an input interface, wherein the processor may control the input interface to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0171] Optionally, the chip may further include an output interface, wherein the processor may control the output interface to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0172] Optionally, the chip can be applied to the terminal device in the embodiments of the present application, and the chip can implement the corresponding processes implemented by the terminal device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0173] The image quality assessment device may also be a terminal device for implementing the image quality assessment method.

[0174] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of the image quality assessment device to complete the steps of the aforementioned method.

[0175] The embodiments of the present application also provide a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods in the embodiments of the present application.

[0176] Optionally, the computer program product can be applied to the image quality assessment device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the image quality assessment device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0177] The embodiment of the present application also provides a computer program.

[0178] Optionally, the computer program can be applied to the image quality evaluation device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the image quality evaluation device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0179] It should be understood that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0180] It should be understood that the terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "have", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0181] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0182] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0183] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0184] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0185] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0186] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for evaluating image quality, characterized in that: The method comprises: Performing brightness region decomposition on a high dynamic range (HDR) image to determine a plurality of brightness regions of the HDR image; Performing spatial feature extraction on each brightness region of the HDR image to determine first feature maps of each brightness region of the HDR image in M ​​stages, where M is a positive integer; Performing feature recombination on first feature maps of multiple brightness regions of the HDR image at the same stage to determine second feature maps of the HDR image at M stages; Determine a prediction quality score of the HDR image according to the second feature maps of the HDR image at the M stages.

2. The method according to claim 1, characterized in that The step of performing brightness region decomposition on a high dynamic range (HDR) image to determine a plurality of brightness regions of the HDR image comprises: Determining one or more brightness thresholds of the HDR image according to the brightness information of the HDR image; Perform brightness region decomposition on the HDR image according to the one or more brightness thresholds to determine a plurality of brightness regions of the HDR image.

3. The method according to claim 2, characterized in that The step of determining one or more brightness thresholds of the HDR image according to the brightness information of the HDR image includes: Determining a logarithmic mean of luminance samples of the HDR image; Determine a logarithmic value of the maximum brightness of the HDR image and a logarithmic value of the minimum brightness of the HDR image; Determine a brightness decomposition parameter of the HDR image according to the logarithmic mean of brightness samples of the HDR image, the logarithmic value of the maximum brightness, and the logarithmic value of the minimum brightness; A first brightness threshold and a second brightness threshold of the HDR image are determined according to the brightness decomposition parameter of the HDR image, and the maximum brightness and the minimum brightness of the HDR image.

4. The method according to claim 1, characterized in that: The step of performing feature recombination on first feature maps of multiple brightness regions of the HDR image at the same stage to determine second feature maps of the HDR image at M stages includes: Based on the channel attention mechanism, the first feature map of the brightness area at each stage is processed to obtain the intermediate feature map of the brightness area at each stage; The intermediate feature maps of the multiple brightness areas at the same stage are merged to obtain second feature maps of the HDR image at M stages.

5. The method according to claim 1, characterized in that In the case where the image quality assessment method is used to perform a video image quality assessment task, determining the predicted quality score of the HDR image according to the second feature maps of the HDR image at the M stages includes: Performing motion perception on second feature maps of multiple HDR images in the HDR video at M stages, and determining third feature maps of the HDR images at the M stages; Performing feature aggregation on the second feature map and the third feature map of the HDR image at the same stage to determine fourth feature maps of the HDR image at M stages; Determine a predicted quality score of the HDR video according to the fourth feature maps of the multiple HDR images at the M stages.

6. The method according to claim 5, characterized in that The performing motion perception on the second feature maps of the plurality of HDR images in the HDR video at the M stages to determine the third feature maps of the HDR images at the M stages includes: Motion perception is performed according to second feature maps of a current HDR image and a reference HDR image in the HDR video at the same stage, and third feature maps of the current HDR image at M stages are determined.

7. The method according to claim 5, characterized in that Determining the predicted quality score of the HDR video according to the fourth feature maps of the multiple HDR images at the M stages includes: Performing position encoding on fourth feature maps of the multiple HDR images at the same stage, and determining fifth feature maps of the multiple HDR images at the same stage; Performing spatiotemporal feature extraction on fifth feature maps of the multiple HDR images at the same stage to determine sixth feature maps of the multiple HDR images at M stages; Fusing the sixth feature maps of the HDR image at the M stages to determine a seventh feature map of the HDR image; Determining a predicted quality score of the HDR video based on the seventh feature map of the plurality of HDR images.

8. The method according to claim 1, characterized in that When the image quality assessment method is applied to an image quality assessment task, determining a predicted quality score of the HDR image according to the second feature maps of the HDR image at the M stages includes: Performing feature extraction on the second feature maps of the HDR image at the M stages to determine an eighth feature map of the HDR image at the M stages; Fusing the eighth feature maps of the HDR image at the M stages to determine a ninth feature map of the HDR image; A prediction quality score of the HDR image is determined according to a ninth feature map of the HDR image.

9. The method according to any one of claims 1 to 8, characterized in that: The method comprises: Inputting the HDR image into an image quality assessment model and outputting a predicted quality score of the HDR image, wherein the image quality assessment model is used to perform the steps of the image quality assessment method, and the image quality assessment model includes one or more modules; During the training process of the image quality assessment model, the method further includes: Calculating a loss function based on the predicted quality score and the reference quality score; According to the loss function, the parameters of the first module of the image quality assessment model are optimized.

10. An image quality evaluation device, characterized in that: The image quality evaluation device comprises: A decomposition module, used for performing brightness region decomposition on a high dynamic range (HDR) image to determine a plurality of brightness regions of the HDR image; A first feature extraction module is used to extract spatial features of each brightness area of ​​the HDR image, and determine a first feature map of each brightness area of ​​the HDR image in M ​​stages; A recombining module, used for performing feature recombining on first feature maps of multiple brightness areas of the HDR image at the same stage, and determining second feature maps of the HDR image at M stages; A determination module is used to determine a prediction quality score of the HDR image according to the second feature maps of the HDR image in the M stages.

11. An image quality evaluation device, characterized in that: The image quality assessment device comprises: a processor and a memory configured to store a computer program that can be run on the processor. Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 9 when running the computer program.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images