Image quality evaluation method and device, electronic equipment and storage medium

Abstract

This disclosure relates to an image quality assessment method, apparatus, electronic device, and storage medium. The method includes: extracting features from an acquired image to be evaluated, determining at least one feature map of the image to be evaluated, performing global moment pooling on the feature map, determining pooling data of the feature map based on the first moment and at least one Nth moment of feature points in each region of the feature map, and then determining the quality assessment result of the image to be evaluated based on the pooling data of the feature map. Embodiments of this disclosure can better capture image texture and obtain more accurate image quality assessment results.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image quality evaluation method and apparatus, electronic device and storage medium. Background Technology

[0002] Image quality assessment, a fundamental concept in image processing, has wide applicability in many fields, such as image compression, narrowband high-definition image transmission, video encoding and decoding, and video surveillance. Among related technologies, image quality can be determined by comparing the image to be evaluated with a reference image, based on evaluation metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). However, when a reference image is unavailable, blind image quality assessment—a method that does not require reference image information, such as neural network-based blind image quality assessment methods—is gaining increasingly widespread application.

[0003] However, using the above image evaluation methods to assess image quality is not sensitive enough to the material content of the image being evaluated and cannot reflect the true actual image quality. Summary of the Invention

[0004] This disclosure proposes a technical solution for image quality evaluation.

[0005] According to one aspect of this disclosure, an image quality assessment method is provided, comprising: extracting features from an acquired image to be evaluated to determine at least one feature map of the image to be evaluated; performing global moment pooling on the feature map, and determining pooling data of the feature map based on the first moment and at least one Nth moment of feature points in each region of the feature map, where N is an integer greater than 1; and determining the quality assessment result of the image to be evaluated based on the pooling data of the feature map.

[0006] In one possible implementation, before performing global moment pooling on the feature map, the method further includes: performing saliency detection on the image to be evaluated to determine a saliency map of the image to be evaluated, the saliency map including saliency information of the image to be evaluated; performing weighted processing on the feature map based on the saliency map to determine a weighted feature map; the global moment pooling on the feature map includes: performing global moment pooling on the weighted feature map.

[0007] In one possible implementation, the step of performing global moment pooling on the feature map, and determining the pooling data of the feature map based on the first moment and at least one Nth moment of the feature points in each partial region of the feature map, includes: calculating the first moment of the feature points in each partial region of the feature map to obtain the first-order pooling data of the feature map; calculating at least one Nth moment of the feature points in each partial region of the feature map based on the first-order pooling data to obtain at least one Nth-order pooling data of the feature map; and determining the pooling data of the feature map based on the first-order pooling data and the at least one Nth-order pooling data.

[0008] In one possible implementation, the first-order moment of feature points in each region of the feature map is calculated to obtain the first-order pooling data of the feature map, including: determining at least one first window data of the feature map based on the pooling kernel size and pooling step size of the feature map; determining the first-order moment corresponding to each first window data by the average pixel value of the feature points in each first window data; and determining the first-order pooling data of the feature map based on at least one of the first-order moments of the feature map.

[0009] In one possible implementation, based on the first-order pooling data, at least one Nth-order moment is calculated for feature points in each region of the feature map to obtain at least one Nth-order pooling data of the feature map, including: determining at least one second window data of the feature map based on the pooling kernel size and pooling step size of the feature map; determining the first-order moment corresponding to each second window data in the first-order pooling data; determining the Nth-order moment corresponding to each second window data based on each second window data and the first-order moment corresponding to each second window data; and determining the Nth-order pooling data of the feature map based on at least one Nth-order moment of the feature map.

[0010] In one possible implementation, determining the pooling data of the feature map based on the first-order pooling data and the at least one N-order pooling data includes: normalizing the first-order pooling data and the at least one N-order pooling data respectively to determine the normalized first-order pooling data and the normalized at least one N-order pooling data; merging the normalized first-order pooling data and the normalized at least one N-order pooling data to determine the pooling data of the feature map.

[0011] In one possible implementation, determining the quality evaluation result of the image to be evaluated based on the pooling data of the feature map includes: performing an affine transformation on the pooling data of the feature map to determine the affine transformation result of the pooling data; and determining the quality evaluation result of the image to be evaluated based on the affine transformation result of the pooling data.

[0012] In one possible implementation, the method is applied to an image quality assessment network. The training process of the image quality assessment network includes: acquiring a training dataset, which includes multiple sample data and evaluation information of the sample data; inputting the sample data into the image quality assessment network for processing to obtain the quality assessment result of the sample data; iteratively training the image quality assessment network based on the evaluation information of the sample data in the training dataset, the quality assessment result of the sample data, and a preset loss function to obtain a trained image quality assessment network; wherein, the image quality assessment network includes at least one pooling layer for performing global moment pooling on the feature maps of the sample data to determine the pooling data of the feature maps.

[0013] According to one aspect of this disclosure, an image quality evaluation apparatus is provided, comprising: a first determining module, configured to extract features from an acquired image to be evaluated and determine at least one feature map of the image to be evaluated; a second determining module, configured to perform global moment pooling processing on the feature map and determine pooling data of the feature map based on the first moment and at least one Nth moment of feature points in each partial region of the feature map, where N is an integer greater than 1; and a third determining module, configured to determine the quality evaluation result of the image to be evaluated based on the pooling data of the feature map.

[0014] In one possible implementation, the apparatus further includes a fourth determining module, configured to: perform saliency detection on the image to be evaluated before performing global moment pooling on the feature map, and determine a saliency map of the image to be evaluated, the saliency map including saliency information of the image to be evaluated; perform weighted processing on the feature map according to the saliency map, and determine a weighted feature map; the second determining module is configured to: perform global moment pooling on the weighted feature map.

[0015] In one possible implementation, the second determining module is configured to: calculate the first moment of feature points in each region of the feature map to obtain the first-order pooling data of the feature map; calculate at least one N-order moment of feature points in each region of the feature map based on the first-order pooling data to obtain at least one N-order pooling data of the feature map; and determine the pooling data of the feature map based on the first-order pooling data and the at least one N-order pooling data.

[0016] In one possible implementation, the first-order moment of feature points in each region of the feature map is calculated to obtain the first-order pooling data of the feature map, including: determining at least one first window data of the feature map based on the pooling kernel size and pooling step size of the feature map; determining the first-order moment corresponding to each first window data by the average pixel value of the feature points in each first window data; and determining the first-order pooling data of the feature map based on at least one of the first-order moments of the feature map.

[0017] In one possible implementation, based on the first-order pooling data, at least one Nth-order moment is calculated for feature points in each region of the feature map to obtain at least one Nth-order pooling data of the feature map, including: determining at least one second window data of the feature map based on the pooling kernel size and pooling step size of the feature map; determining the first-order moment corresponding to each second window data in the first-order pooling data; determining the Nth-order moment corresponding to each second window data based on each second window data and the first-order moment corresponding to each second window data; and determining the Nth-order pooling data of the feature map based on at least one Nth-order moment of the feature map.

[0018] In one possible implementation, determining the pooling data of the feature map based on the first-order pooling data and the at least one N-order pooling data includes: normalizing the first-order pooling data and the at least one N-order pooling data respectively to determine the normalized first-order pooling data and the normalized at least one N-order pooling data; merging the normalized first-order pooling data and the normalized at least one N-order pooling data to determine the pooling data of the feature map.

[0019] In one possible implementation, the third determining module is used to: perform an affine transformation on the pooling data of the feature map to determine the affine transformation result of the pooling data; and determine the quality evaluation result of the image to be evaluated based on the affine transformation result of the pooling data.

[0020] In one possible implementation, the apparatus is applied to an image quality assessment network. The apparatus further includes a training module for: acquiring a training dataset, which includes multiple sample data and evaluation information of the sample data; inputting the sample data into the image quality assessment network for processing to obtain a quality assessment result for the sample data; iteratively training the image quality assessment network based on the evaluation information of the sample data in the training dataset, the quality assessment result of the sample data, and a preset loss function to obtain a trained image quality assessment network; wherein the image quality assessment network includes at least one pooling layer for performing global moment pooling on the feature maps of the sample data to determine the pooling data of the feature maps.

[0021] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0022] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0023] In this embodiment, feature extraction is performed on the acquired image to be evaluated to determine at least one feature map of the image. Global moment pooling is then applied to the feature map. Based on the first moment and at least one Nth moment of feature points within each region of the feature map, pooling data for the feature map is determined. Furthermore, the quality evaluation result of the image to be evaluated is determined based on the pooling data of the feature map. In this way, global moment pooling is used in the image quality evaluation method to calculate the first moment and at least one Nth moment of feature points within each region of the feature map. Pooling data is then obtained based on these first moments and at least one Nth moment. This pooling data can better capture image texture, significantly improving the sensitivity of the image quality evaluation method to image texture and resulting in more accurate image quality evaluation results.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0026] Figure 1 A flowchart illustrating an image quality evaluation method according to an embodiment of the present disclosure is shown.

[0027] Figure 2 A schematic diagram of global moment pooling processing according to an embodiment of the present disclosure is shown.

[0028] Figure 3 A block diagram of an image quality evaluation apparatus according to an embodiment of the present disclosure is shown.

[0029] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0030] Figure 5 A block diagram of another electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0031] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0032] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0033] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0034] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0035] Figure 1 A flowchart illustrating an image quality evaluation method according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the method is applied to an image quality assessment network, and the image quality assessment method includes:

[0036] In step S11, feature extraction is performed on the acquired image to be evaluated to determine at least one feature map of the image to be evaluated;

[0037] In step S12, global moment pooling is performed on the feature map. Based on the first moment and at least one Nth moment of the feature points in each region of the feature map, the pooling data of the feature map is determined, where N is an integer greater than 1.

[0038] In step S13, the quality evaluation result of the image to be evaluated is determined based on the pooling data of the feature map.

[0039] In one possible implementation, the image quality evaluation method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server.

[0040] In one possible implementation, an image quality assessment network can be pre-trained to evaluate the image quality of the image to be evaluated and determine the quality assessment result. The pre-trained image quality assessment network may include at least one of the following: Convolutional Neural Networks (CNN), Deep Convolutional Neural Networks (DCNN), Deep Neural Networks (DNN), Back Propagation Neural Network (BP), or Backbone Neural Network.

[0041] For example, when the image quality assessment network includes a convolutional neural network (CNN), a lightweight network (MobileNet) can be selected as the base model for the CNN. Other network structures can be added to MobileNet to form a CNN, which is then trained. Because MobileNet is used as part of the CNN, and because MobileNet is small and has fast data processing speed, the training speed is also faster. At the same time, the trained image quality assessment network also has the advantages of small size and fast data processing speed, making it more suitable for deployment in embedded devices.

[0042] Here, the network structure of the image quality evaluation network described above is only an example. The network structure may include multiple convolutional layers, multiple pooling layers, fully connected layers, etc. The specific construction method and structure of the network can be determined according to the actual situation, and will not be elaborated here. The above example does not constitute a limitation on the embodiments of this disclosure.

[0043] In one possible implementation, in step S11, the image to be evaluated can be input into a pre-trained image quality evaluation network. The convolutional layers of the pre-trained image quality evaluation network are used to extract features from the image to be evaluated, resulting in one or more feature maps of the image to be evaluated. The feature map can be a multi-channel feature map composed of multiple features extracted using multiple convolutional kernels. The feature map may include at least one of the feature information of the image to be evaluated, such as color feature information, texture feature information, shape feature information, and spatial feature information.

[0044] In step S11, the feature map of the image to be evaluated is obtained. The feature map can be pooled by the pooling layer of the image quality evaluation network to remove redundant information in the feature map, reduce the consumption of computing resources, and improve the evaluation efficiency of the image quality evaluation network. In addition, the pooling process is also conducive to strengthening the invariance of image features, so that the image quality evaluation network can increase its robustness to image offset, rotation, scale changes, etc., and reduce the probability of overfitting in the image quality evaluation network.

[0045] In related technologies, Global Average Pooling (GAP) is used to pool feature maps, which calculates the mean of feature points in each region of the feature map. GAP pooling performs very well in high-level computer vision tasks such as classification; however, its performance is significantly reduced in low-level computer vision tasks such as image quality assessment, as the mean of each region of the feature map of the image to be evaluated cannot reflect the texture information of the image.

[0046] For example, suppose feature map A and feature map B have different materials. Feature map A is a 4×4 striped material, with odd-numbered rows being entirely white (i.e., the pixel value of the feature points in odd-numbered rows is 254) and even-numbered rows being entirely black (the pixel value of the feature points in even-numbered rows is 0). This can be represented as:

[0047] Feature map B is a uniform material of size 4×4. Feature map B is entirely gray (i.e., the pixel value of each feature point within the feature map is 127), which can be represented as:

[0048] GAP pooling of feature maps (e.g., feature map A and feature map B) involves dividing the feature map into multiple regions (e.g., each region is 2×2 in size, which corresponds to the pooling kernel size), calculating the average value of each region, and combining the average values ​​calculated for each region to obtain the pooled data of the feature map.

[0049] For example, if we use a pooling kernel of size 2×2 and stride 2 to perform GAP pooling on feature map A, we can divide feature map A into 4 regions, namely: The average value 127 of each region can be calculated separately, and the average values ​​127 of each region can be combined to obtain the pooled data of feature map A.

[0050] Similarly, performing GAP pooling on feature map B can divide it into 4 regions, namely: The average value of 127 can be calculated for each region separately. By combining the average values ​​of 127 from each region, the pooled data of feature map B can be obtained.

[0051] The pooled data A1 of feature map A obtained by GAP pooling is exactly the same as the pooled data B1 of feature map B. It is clear that the GAP pooling method in related technologies cannot distinguish the texture of feature map A and feature map B, and cannot effectively capture the texture information of the image.

[0052] In view of this, in order to improve the sensitivity of capturing material information, in step S12, global moment pooling can be used instead of GAP pooling. Global moment pooling can be used to calculate the first moment and at least one Nth moment of feature points in each region of the feature map, and obtain pooling data based on the first moment and at least one Nth moment, where N is an integer greater than 1.

[0053] The Nth order moment, also known as a higher-order moment, can be a second-order moment, a third-order moment, a fourth-order moment, etc. The pooling data can be determined based on the first-order moment and any single higher-order moment (i.e., one or more moments from the second to the Nth order). In practical applications, to achieve better results, the pooling data can include at least first-order and second-order moments. The pooling data disclosed herein can be determined by moments of various orders, and this disclosure does not impose specific limitations on this.

[0054] To achieve a balance between computational complexity and the ability to capture image texture information, pooling data can be obtained based on the first and second moments. The following explanation uses pooling data obtained from the first and second moments as an example to illustrate global moment pooling. It should be understood that the pooling data in this embodiment is not limited to this.

[0055] Assuming we use the same 2×2 pooling kernel with a stride of 2, performing global moment pooling on feature map A allows us to calculate the first moment (i.e., the mean of the feature points is 127) for each region (each region is 2×2) in feature map A, thus obtaining the first-order pooled data of feature map A. Then, the second moment (i.e., the variance of the difference between the pixel value of each feature point and the corresponding first moment in each region) can be calculated for the feature points in each region of feature map A. We can obtain the second-order pooling data of feature map A. The first-order pooling data C1 and the second-order pooling data C2 of feature map A can be merged to obtain the pooling data A2 = C1 + C2 of feature map A.

[0056] Similarly, using a pooling kernel of size 2×2 and stride 2, global moment pooling is performed on feature map B. For example, the first moment (i.e., the mean of the feature points is 127) can be calculated for the feature points in each region (each region is 2×2) of feature map B to obtain the first-order pooling data of feature map B. Then, the second moment (i.e., the variance of the difference between the pixel value of each feature point and the corresponding first moment in each region) can be calculated for the feature points in each region of feature map B. The second-order pooling data of feature map B can be obtained. The first-order pooling data C1' and the second-order pooling data C2' of feature map B can be merged to obtain the pooling data B2 = C1' + C2' of feature map B.

[0057] Although the first-order pooling data C1 of feature map A is the same as the first-order pooling data C1' of feature map B, the second-order pooling data C2 of feature map A is significantly different from the second-order pooling data C2' of feature map B. Therefore, the pooling data A2 = C1 + C2 of feature map A is significantly different from the pooling data B2 = C1' + C2' of feature map B, which can effectively distinguish the material information of feature map A and feature map B.

[0058] As can be seen, the global moment pooling processing of this embodiment is superior to the GAP pooling processing in related technologies in capturing material information, and can better capture the material information of the feature map.

[0059] It should be understood that pooling data can be composed of pooling data from level 1 to level N. The larger the value of N, the better its ability to capture image material information. Correspondingly, the amount of computation is also greater. In practical applications, different values ​​of N can be selected according to different application scenarios, such as N=4. This disclosure does not limit this.

[0060] After the pooling data of the feature map is determined in step S12, in step S13, the pooling data of the feature map can be processed by affine transformation, nonlinear activation, etc., using the fully connected layer of the image quality evaluation network to determine the quality evaluation result of the image to be evaluated.

[0061] Through steps S11 to S13, global moment pooling is used in the image quality assessment method to calculate the first moment and at least one Nth moment for feature points in each region of the feature map, and pooling data is obtained based on the first moment and at least one Nth moment. This pooling data can better capture the image material, which is beneficial to significantly improve the sensitivity of the image quality assessment method to the image material and obtain more accurate image quality assessment results.

[0062] The image quality evaluation method of this disclosure will now be described in detail.

[0063] In one possible implementation, the image quality assessment method of this disclosure can be applied to an image quality assessment network. The training process of the image quality assessment network includes: acquiring a training dataset, which includes multiple sample data and evaluation information of the sample data; inputting the sample data into the image quality assessment network for processing to obtain the quality assessment result of the sample data; and iteratively training the image quality assessment network based on the evaluation information of the sample data in the training set, the quality assessment result of the sample data, and a preset loss function to obtain a trained image quality assessment network. The image quality assessment network includes at least one pooling layer for performing global moment pooling on the feature maps of the sample data to determine the pooling data of the feature maps.

[0064] For example, the image quality assessment network of this disclosure embodiment can be a neural network of any structure, such as a deep convolutional neural network including a VGG16 network, an Inception-v2 network, etc., which includes multiple convolutional layers, multiple pooling layers, and multiple fully connected layers. The pooling layers in the neural network can adopt the global moment pooling processing proposed in this disclosure embodiment, instead of the global uniform pooling processing (or max pooling processing, etc.) in related technologies, so as to improve the sensitivity of the image quality assessment network to the image material and make the quality assessment results of the image quality assessment network more accurate.

[0065] The training dataset in this disclosure can be a large dataset downloaded from the Internet, such as the ImageNet dataset (a large visualization database used for research on visual object recognition software), in which the sample data is labeled with evaluation information; the training dataset can also be a training dataset composed of multiple images taken by an image acquisition device, in which evaluation information can be manually labeled on the sample images. This disclosure does not limit the method of obtaining the training dataset.

[0066] During each round of iterative training, each sample data can be input into the image quality assessment network for processing to obtain the quality assessment result of each sample data (e.g., the predicted category information of each sample image). The evaluation information of each sample data in the training set and the predicted quality assessment results of the sample data can be input into a preset loss function. Based on the loss output of the loss function, the network parameters of the image quality assessment network are adjusted until the network parameters of the image quality assessment network converge, thus obtaining the trained image quality assessment network.

[0067] In this context, it is assumed that the training dataset may include multiple sample data, and the evaluation information of each sample data may be category information labeled with the quality of the image or image quality score. This disclosure does not limit the number of sample data in the training dataset or the form of the evaluation information of the sample data.

[0068] In this way, a well-trained image quality assessment network can be obtained. Compared with pooling layers that use GAP pooling (or max pooling, etc.) in related techniques, pooling layers that use global moment pooling can improve the sensitivity of the image quality assessment network to image material, thereby improving the accuracy of its predicted image quality assessment results.

[0069] Next, the trained image quality assessment network can be used to predict the quality assessment results of the image to be assessed.

[0070] In step S11, feature extraction can be performed on the image to be evaluated based on multiple convolutional layers of the image quality evaluation network to determine at least one feature map of the image to be evaluated.

[0071] To further improve the image quality evaluation effect of the embodiments of this disclosure, saliency maps can also be used for pooling based on global attention to improve the sensitivity of the image quality evaluation method of the embodiments of this disclosure to the subjective attention region of the human eye.

[0072] In one possible implementation, before step S12, saliency detection can be performed on the image to be evaluated to determine a saliency map of the image to be evaluated, the saliency map including saliency information of the image to be evaluated; based on the saliency map, the feature map is weighted to determine a weighted feature map.

[0073] For example, GAP pooling in related technologies is insensitive to the subjective attention regions of the human eye. For instance, for any feature map H with uniform subjective quality, its foreground region is P (e.g., the region where the target is located), and its background region is Q (e.g., the region outside the target). The foreground region of feature map H can be blurred to obtain feature map H1, and the background region can be blurred in the same way to obtain feature map H2. Assuming that the foreground region P and the background region Q have the same area, related technologies using GAP pooling to pool feature maps H1 and H2 will produce the same pooling result, failing to distinguish between feature maps H1 and H2. Therefore, in related technologies, GAP pooling cannot reflect the subjective attention regions of the human eye.

[0074] In view of this, saliency detection can be performed on the image to be evaluated to determine the saliency map S of the image to be evaluated. The saliency map S can represent the saliency information of the image to be evaluated. The size (length × width) of the saliency map S can be the same as the size of the image to be evaluated. The pixel value of any point on the saliency map S can correspond to the saliency information of the pixel at the same position in the image to be evaluated. The larger the pixel value of any point on the saliency map S, the higher the saliency at the corresponding position in the image to be evaluated.

[0075] Alternatively, the saliency map S of the image to be evaluated, determined by performing saliency detection on the image to be evaluated, can also be a binary map, where salient regions correspond to a value (e.g., 1) and non-salient regions correspond to a value (e.g., 0). This disclosure does not limit the specific values ​​of salient and non-salient regions.

[0076] Alternatively, the saliency map S of the image to be evaluated, determined by performing saliency detection on the image, can also be a saliency map comprising multiple regions, each region corresponding to a saliency value. The saliency value of any region on the saliency map S corresponds to the saliency information of the same region in the image to be evaluated; the larger the saliency value, the higher the saliency of the corresponding region in the image to be evaluated.

[0077] Alternatively, the saliency map S of the image to be evaluated, determined by performing saliency detection, can also be a histogram. The horizontal axis of the histogram represents the distribution of pixel values, and the vertical axis represents the saliency value corresponding to each pixel value. For example, assuming the pixel values ​​in the image to be evaluated range from 0 to 255, pixels with different values ​​can correspond to different saliency values. The saliency value of a pixel value on the histogram corresponds to the saliency information of pixels with the same value in the image to be evaluated. The larger the saliency value corresponding to a pixel value on the histogram, the higher the saliency of pixels with the same pixel value in the image to be evaluated.

[0078] It should be understood that this disclosure does not limit the form of the saliency map of the image to be evaluated, nor the form of the saliency information included in the saliency map.

[0079] In the process of saliency detection of the image to be evaluated, multiple saliency detection models tailored to different scenarios or needs can be connected via a software interface. These models then perform saliency detection on the image to be evaluated, resulting in a saliency map. For example, saliency detection models can include visual attention-based models such as Pyramid Constrained Self-Attention (PCSA). This disclosure does not limit the structure or category of saliency detection models.

[0080] After obtaining the saliency map S of the image to be evaluated, it can be weighted and normalized with the feature map F of the image to be evaluated determined in step S11 to obtain the processed feature map FN = F × S / MEAN(S). Here, MEAN is the mean function, which can be used to calculate the mean of the saliency map S.

[0081] In contrast to a method that is computationally inefficient with both floating-point and fixed-point calculations, namely FN = SUM(F*S) / SUM(S), where the SUM operation (summation operation) leads to overflow in floating-point and fixed-point calculations, the method in this embodiment does not perform the SUM operation during the acquisition of the feature map FN. This is more favorable for both floating-point and fixed-point numbers and facilitates the training and deployment of image quality assessment networks based on backpropagation.

[0082] In this way, the image quality assessment network can acquire subjective attention capabilities that are independent of training, thereby increasing its sensitivity to the subjective attention areas of the human eye and effectively reducing the probability of poor quality assessment results for images with blurred backgrounds.

[0083] After obtaining the feature map with saliency weighting, global moment pooling can be performed on the weighted feature map in step S12 to determine the pooling data of the feature map, so that the global moment pooling process has subjective attention capability and improves the sensitivity to the subjective region of human eyes.

[0084] In one possible implementation, step S12 may include steps S121 to S123:

[0085] In step S121, the first moment of the feature points in each region of the feature map can be calculated to obtain the first-order pooling data of the feature map;

[0086] In one possible implementation, step S121 may include: determining at least one first window data of the feature map based on the pooling kernel size and pooling step size of the feature map; determining the average pixel value of the feature points in each first window data as the first moment corresponding to each first window data; and determining the first-order pooling data of the feature map based on at least one of the first moments of the feature map.

[0087] For example, suppose the pooling kernel size of feature map F is H×W. The pooling kernel can slide in feature map F with a stride L. Each slide can extract the first window of data F of feature map F. ij And based on the first window data F ij The first window data F ij Feature point F in ij The average pixel value of (p, q) is determined as the data F of the first window. ij The corresponding first moment It can be represented as follows:

[0088]

[0089] In formula (1), H represents the first window data F. ij The number of feature points in the vertical direction, W represents the data in the first window F. ij The number of feature points in the horizontal direction, H×W represents the data F of the first window. ij The total number of feature points. F ij (p,q) represents the data F in the first window. ij The feature point located at the p-th row and q-th column.

[0090] Using formula (1), the window data F retrieved in each sliding motion can be calculated. ij First moment Until the pooling kernel finishes sliding on the feature map F, the first moments obtained from each iteration can be... The convolution kernels are combined into a matrix according to their sliding order to obtain the first-order pooling data GMP of the feature maps. 1 .

[0091] This method facilitates the rapid and accurate determination of the first-order pooling data of the feature map.

[0092] In step S122, based on the first-order pooling data, at least one N-order moment is calculated for feature points in each region of the feature map to obtain at least one N-order pooling data of the feature map.

[0093] In one possible implementation, step S122 may include: determining at least one second window data of the feature map based on the pooling kernel size and pooling step size of the feature map; determining a first moment corresponding to each second window data in the first-order pooling data; determining an Nth moment corresponding to each second window data based on each second window data and the first moment corresponding to each second window data; and determining Nth-order pooling data of the feature map based on at least one Nth moment of the feature map.

[0094] It should be understood that in the calculation process of N-order pooling data, the pooling kernel size and pooling step size of the feature map can be the same as those of the feature map in the calculation process of first-order pooling data. This is beneficial for window correspondence, and makes it easier to use the first-order moment of first-order pooling data in the calculation process of N-order pooling data, thereby improving computational efficiency.

[0095] For example, suppose the pooling kernel size of feature map F is H×W. The pooling kernel can slide in feature map F with a stride L. Each slide can extract a second window of data F from feature map F. ij Using formula (1), the data F of the second window can be determined. ij The corresponding first moment Data F in the second window ij Each feature point F ij The pixel values ​​of (p,q) and the first moment The mean of the Nth power of the difference is determined by the second window data F. ij Nth order moment It can be represented as follows:

[0096]

[0097] In formula (2), H represents the second window data F. ij The number of feature points in the vertical direction, W represents the second window data F. ij The number of feature points in the horizontal direction, H×W represents the data F of the first window. ij The total number of feature points. F ij (p,q) represents the data F in the second window. ij The feature point located at the p-th row and q-th column is... Representative and second window data F ij The corresponding first moment, where N represents the order of the moment, can be any integer greater than 1. This disclosure does not impose any restrictions on the specific value of N.

[0098] Using formula (2), the second window data F retrieved each time can be calculated. ij Nth order moment Until the pooling kernel finishes sliding on the feature map F, the Nth order moments obtained from each iteration can be... The convolution kernels are combined into a matrix according to their sliding order to obtain the Nth-order pooled data GMP of the feature map F. N .

[0099] This method facilitates the rapid and accurate determination of the Nth-order pooling data of the feature map.

[0100] In step S123, the pooling data of the feature map is determined based on the first-order pooling data and the at least one N-order pooling data.

[0101] In one possible implementation, step S123 may include: normalizing the first-order pooling data and the at least one N-order pooling data respectively to determine the normalized first-order pooling data and the normalized at least one N-order pooling data; merging the normalized first-order pooling data and the normalized at least one N-order pooling data to determine the pooling data of the feature map.

[0102] For example, through steps S121 to S122, the first-order pooled data GMP of feature map F can be obtained. 1 and at least one N-order pooled data GMP N (i.e., second-order pooling data GMP) 2 ~N-order pooled data GMP N (one or more of them), where N is any integer greater than 1.

[0103] GMP can be applied to first-order pooled data separately. 1 and at least one N-order pooled data GMP N Normalization is performed, that is, the data is mapped to the range of 0 to 1, and the normalized first-order pooling data GMP is determined. 1 and at least one normalized Nth-order pooled data GMP N .

[0104] The normalization process can employ methods such as batch normalization, maximum normalization, layer normalization, linear function normalization (Min-Max Scaling), mean standardization (Z-Score Standardization), decimal scaling, and exponential normalization, and this disclosure does not impose any restrictions on these methods.

[0105] GMP can be used to process normalized first-order pooled data. 1 and at least one normalized Nth-order pooled data GMPN Perform merging (e.g., channel merging) to determine the pooled data for feature map F.

[0106] By normalizing and merging pooling data of each order in this way, the training failure caused by gradient explosion during backpropagation of the image quality assessment network due to the calculation of higher-order exponents can be reduced.

[0107] The global matrix pooling process of this disclosure embodiment will be described below with a specific example.

[0108] Figure 2 A schematic diagram of global moment pooling processing according to an embodiment of this disclosure is shown. Figure 2 As shown, assuming the pooling kernel size is 2×2 and the step size is 2, four first window data can be determined based on the pooling kernel size of 2×2 and the pooling step size of 2 of the feature map.

[0109] like Figure 2 As shown, for first-order pooled data, the first window of data can be retrieved through the first sliding motion of the pooling kernel. The data for the first window can be obtained according to formula (1). The corresponding first moment

[0110] The first window of data can be retrieved by sliding the pooling kernel a second time. The data for the first window can be obtained according to formula (1). The corresponding first moment

[0111] The first window of data can be retrieved by sliding the pooling kernel a third time. The data for the first window can be obtained according to formula (1). The corresponding first moment

[0112] The first window of data can be retrieved by the fourth sliding of the pooling kernel. The data for the first window can be obtained according to formula (1). The corresponding first moment

[0113] The first moments obtained from the above steps can be combined into a matrix to obtain the first-order pooling data of the feature map, i.e.:

[0114] like Figure 2 As shown, for second-order (i.e., N=2) pooling data, the second window of data can be extracted through the first sliding motion of the pooling kernel. The second window data can be obtained according to formula (2). The corresponding second moment

[0115] The second window of data can be retrieved by sliding the pooling kernel a second time. The second window data can be obtained according to formula (2). The corresponding second moment

[0116] The second window of data can be retrieved by sliding the pooling kernel a third time. The second window data can be obtained according to formula (2). The corresponding second moment

[0117] The second window data can be retrieved by the fourth sliding of the pooling kernel. The second window data can be obtained according to formula (2). The corresponding second moment

[0118] The second moments obtained from the above steps can be combined into a matrix to obtain the second-order pooling data of the feature map, i.e.:

[0119] like Figure 2 As shown, similarly, based on formula (2), third-order (i.e., N=3) pooling data can be obtained. and fourth-order (i.e., N=4) pooling data

[0120] It should be understood that during the above calculation of pooling data of each order, the operation of window data can be performed in parallel, and this disclosure does not impose any restrictions on this.

[0121] Having obtained first-order pooled data to fourth-order pooled data, we can perform channel merging on the first-order pooled data to obtain pooled data for the feature map.

[0122] During the merging process, the higher-order exponent operations in formula (2) may cause gradient explosion during backpropagation optimization. For example, as Figure 2 As shown, comparing the pooling data of different orders reveals that the values ​​of the first-order and second-order pooling data are very large, while the values ​​of the third-order and fourth-order pooling data are very small. Directly merging the large first-order and second-order pooling data with the small third-order and fourth-order pooling data may cause numerical problems during the backpropagation optimization of the image quality assessment network. This could easily lead to gradient explosion, causing instability in the image quality assessment network and significant changes in the loss of the image quality assessment network during each training iteration, hindering effective learning from the training data. Therefore, it is advisable to first normalize the pooling data of each order to be merged, and then merge the normalized pooling data to obtain the merged pooling data.

[0123] Table 1 shows the ablation study results of embodiments of this disclosure, namely, studying the performance of the image quality assessment network by deleting or adding parts of the network structure of the image quality assessment network.

[0124] Table 1

[0125] Evaluation indicators SRCC PLCC GAP Pooling 0.864 0.879 Pooled data GMP(2) 0.886 0.896 Pooled data GMP(4) without normalization NaN NaN Pooled data GMP(4) batch normalization 0.880 0.892 Pooled data GMP(4) maximum normalization 0.880 0.899 Pooled data GMP(4) layer-by-layer normalization 0.890 0.900

[0126] In Table 1, the Spearman Rank-order Correlation Coefficient (SRCC) is used to evaluate the nonlinear correlation of image quality assessment networks. The Pearson Linear Correlation Coefficient (PLCC) is used to evaluate the linear correlation of image quality assessment networks. SRCC and PLCC can be used as evaluation metrics for image quality assessment networks; the closer the SRCC and PLCC values ​​are to 1, the better the performance of the image quality assessment network.

[0127] Table 1 shows the performance of the image quality assessment network using GAP pooling processing in related technologies, and the performance of the image quality assessment network using global moment pooling processing according to the embodiments of this disclosure under different normalization processing methods. Here, pooling data GMP(2) represents pooling data composed of first-order and second-order pooling data, and pooling data GMP(4) represents pooling data composed of first-order to fourth-order pooling data.

[0128] As shown in Table 1, the image quality assessment method based on global moment pooling is significantly better than the image quality assessment method based on GAP pooling in related technologies.

[0129] However, without normalizing the pooled data GMP(4), numerical problems that may occur during the backpropagation optimization of the image quality assessment network (such as gradient explosion when calculating higher-order exponents) can lead to the failure of training the image quality assessment network and make it impossible to calculate its SRCC and PLCC values ​​(as shown in "NaN" in Table 1).

[0130] To reduce gradient explosion during higher-order exponent computation, which can lead to training failures, various normalization methods can be employed to normalize the pooling data at different orders. These include batch normalization, maximum normalization (MaxNorm), and layer normalization. Maximum normalization and layer normalization offer similar performance, while batch normalization incurs a significant performance penalty. Consequently, the SRCC and PLCC values ​​obtained through maximum and layer normalization are superior to those obtained through batch normalization.

[0131] The pooling data is obtained in step S12, and the quality evaluation result of the image to be evaluated can be determined in step S13 based on the pooling data.

[0132] In one possible implementation, step S13 may include: performing an affine transformation on the pooling data of the feature map to determine the affine transformation result of the pooling data; and determining the quality evaluation result of the image to be evaluated based on the affine transformation result of the pooling data.

[0133] For example, an affine layer (a fully connected layer) in an image quality assessment network can be used to perform an affine transformation on the pooled data of the feature map to obtain the affine transformation result. The affine transformation can include one or more combinations of translation transformation, scale transformation, flip transformation, rotation transformation, and shear transformation, and this disclosure is not limited thereto.

[0134] Once the affine transformation result is determined, the quality assessment result of the image to be evaluated can be determined based on this result. For example, assuming the affine transformation result can be calculated as a subjective evaluation score in the channel domain using an affine layer from normalized pooling data, this score can include Mean Opinion Score (MOS) and Differential Mean Opinion Score (DMOS). The image quality assessment result can be determined based on the subjective evaluation score; the higher the subjective evaluation score, the better the corresponding quality assessment result.

[0135] In this way, the quality assessment result of the image to be evaluated can be determined efficiently and quickly.

[0136] To better illustrate the effectiveness of the image quality assessment method of the present disclosure embodiments, a comparative experiment can be conducted using a small-scale synthetic dataset with reference images and distortion information, such as the LIVE dataset and the CSIQ dataset, to compare the image quality assessment methods in related technologies with the image quality assessment method of the present disclosure embodiments (as shown in the table).

[0137] Table 2

[0138]

[0139] In Table 2, SRCC and PLCC are evaluation metrics for image quality assessment networks. The closer the SRCC and PLCC values ​​are to 1, the better the performance of the image quality assessment network. BIECON, PQR, SGDNet, HyperIQA, and NemgIQA networks are image quality assessment methods based on deep convolutional neural networks (DCNNs) in related technologies, and they employ the GAP pooling method. Among them, the NemgIQA network has the best SRCC and PLCC values.

[0140] Therefore, as shown in Table 2, the NemgIQA network with the best evaluation metrics SRCC and PLCC can be selected, and global moment pooling can be used to replace GMP pooling, so that the pooling layers in the NemgIQA network have the function of global moment pooling on the feature map. Among them, GMP(2) represents that the pooling data in the network is composed of first-order pooling data and second-order pooling data, and pooling data GMP(4) represents that the pooling data in the network is composed of first-order pooling data to fourth-order pooling data.

[0141] In Table 2, the SRCC and PLCC values ​​corresponding to “NemgIQA GMP(2)” and “NemgIQA GMP(4)” are significantly better than the SRCC and PLCC values ​​corresponding to “NemgIQA”. It can be seen that the image quality evaluation method based on global moment pooling is significantly better than the image quality evaluation method based on GAP pooling in related technologies.

[0142] Therefore, the embodiments of this disclosure, through a weighted processing procedure based on the saliency map of the image to be evaluated, not only significantly improve the sensitivity of the image quality assessment method to the subjective attention region of the human eye, but also are numerically friendly to floating / fixed point numbers, facilitating the training and deployment of models based on backpropagation. Furthermore, the embodiments of this disclosure also replace the global uniform pooling (GAP) processing in related techniques with global moment pooling, significantly improving the sensitivity of the image quality assessment method to image texture. In particular, regarding the numerical problems in calculating the gradient of higher-order moments, the higher-order pooling data can be normalized first, and then merged to obtain the pooling data. This normalization method, which is numerically friendly to the backpropagation gradient calculation, can greatly reduce the probability of numerical problems that may occur in the calculation process of global moment pooling and backpropagation optimization.

[0143] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0144] In addition, this disclosure also provides an image quality evaluation device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any of the image quality evaluation methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0145] Figure 3 A block diagram of an image quality evaluation apparatus according to an embodiment of the present disclosure is shown, such as Figure 3 As shown, the device includes:

[0146] The first determining module 31 is used to extract features from the acquired image to be evaluated and determine at least one feature map of the image to be evaluated.

[0147] The second determining module 32 is used to perform global moment pooling processing on the feature map, and determine the pooling data of the feature map based on the first moment and at least one Nth moment of the feature points in each region of the feature map, where N is an integer greater than 1.

[0148] The third determining module 33 is used to determine the quality evaluation result of the image to be evaluated based on the pooling data of the feature map.

[0149] In one possible implementation, the apparatus further includes a fourth determining module, configured to: perform saliency detection on the image to be evaluated before performing global moment pooling on the feature map, and determine a saliency map of the image to be evaluated, the saliency map including saliency information of the image to be evaluated; perform weighted processing on the feature map according to the saliency map, and determine a weighted feature map; the second determining module 32 is configured to: perform global moment pooling on the weighted feature map.

[0150] In one possible implementation, the second determining module 32 is configured to: calculate the first moment of feature points in each region of the feature map to obtain the first-order pooling data of the feature map; calculate at least one N-order moment of feature points in each region of the feature map based on the first-order pooling data to obtain at least one N-order pooling data of the feature map; and determine the pooling data of the feature map based on the first-order pooling data and the at least one N-order pooling data.

[0151] In one possible implementation, the first-order moment of feature points in each region of the feature map is calculated to obtain the first-order pooling data of the feature map, including: determining at least one first window data of the feature map based on the pooling kernel size and pooling step size of the feature map; determining the first-order moment corresponding to each first window data by the average pixel value of the feature points in each first window data; and determining the first-order pooling data of the feature map based on at least one of the first-order moments of the feature map.

[0152] In one possible implementation, based on the first-order pooling data, at least one Nth-order moment is calculated for feature points in each region of the feature map to obtain at least one Nth-order pooling data of the feature map, including: determining at least one second window data of the feature map based on the pooling kernel size and pooling step size of the feature map; determining the first-order moment corresponding to each second window data in the first-order pooling data; determining the Nth-order moment corresponding to each second window data based on each second window data and the first-order moment corresponding to each second window data; and determining the Nth-order pooling data of the feature map based on at least one Nth-order moment of the feature map.

[0153] In one possible implementation, determining the pooling data of the feature map based on the first-order pooling data and the at least one N-order pooling data includes: normalizing the first-order pooling data and the at least one N-order pooling data respectively to determine the normalized first-order pooling data and the normalized at least one N-order pooling data; merging the normalized first-order pooling data and the normalized at least one N-order pooling data to determine the pooling data of the feature map.

[0154] In one possible implementation, the third determining module 33 is used to: perform an affine transformation on the pooling data of the feature map to determine the affine transformation result of the pooling data; and determine the quality evaluation result of the image to be evaluated based on the affine transformation result of the pooling data.

[0155] In one possible implementation, the apparatus is applied to an image quality assessment network. The apparatus further includes a training module for: acquiring a training dataset, which includes multiple sample data and evaluation information of the sample data; inputting the sample data into the image quality assessment network for processing to obtain a quality assessment result for the sample data; iteratively training the image quality assessment network based on the evaluation information of the sample data in the training dataset, the quality assessment result of the sample data, and a preset loss function to obtain a trained image quality assessment network; wherein the image quality assessment network includes at least one pooling layer for performing global moment pooling on the feature maps of the sample data to determine the pooling data of the feature maps.

[0156] This method is specifically technically related to the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects that improve the internal performance of computer systems in accordance with natural laws.

[0157] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0158] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0159] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0160] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0161] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0162] Figure 4 This diagram illustrates a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other terminal devices.

[0163] Reference Figure 4 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0164] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0165] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0166] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0167] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0168] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0169] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0170] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0171] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, 3G, 4G, LTE, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID), Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0172] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0173] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0174] Figure 5 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 5 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0175] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0176] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0177] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0178] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0179] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0180] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0181] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0182] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0183] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0185] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0186] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0187] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0188] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0189] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image quality assessment method, characterized in that, include: Feature extraction is performed on the acquired image to be evaluated to determine at least one feature map of the image to be evaluated; Global moment pooling is performed on the feature map. Based on the first moment and at least one Nth moment of the feature points in each region of the feature map, the pooling data of the feature map is determined. The Nth moment is determined based on the Nth power of the difference between the pixel value of each feature point in each region and the corresponding first moment, where N is an integer greater than 1. The quality evaluation result of the image to be evaluated is determined based on the pooling data of the feature map.

2. The method according to claim 1, characterized in that, Before performing global moment pooling on the feature map, the method further includes: The image to be evaluated is subjected to saliency detection to determine the saliency map of the image to be evaluated, and the saliency map includes the saliency information of the image to be evaluated; Based on the saliency map, the feature map is weighted to determine the weighted feature map; The global moment pooling process on the feature map includes: Global moment pooling is applied to the weighted feature map.

3. The method according to claim 1 or 2, characterized in that, The step of performing global moment pooling on the feature map, based on the first moment and at least one Nth moment of feature points in each region of the feature map, determines the pooling data of the feature map, including: The first moment of the feature points in each region of the feature map is calculated to obtain the first-order pooled data of the feature map. Based on the first-order pooling data, at least one N-order moment is calculated for feature points in each region of the feature map to obtain at least one N-order pooling data of the feature map. The pooling data of the feature map is determined based on the first-order pooling data and the at least one N-order pooling data.

4. The method according to claim 3, characterized in that, Calculate the first moment of feature points within each region of the feature map to obtain the first-order pooled data of the feature map, including: Based on the pooling kernel size and pooling step size of the feature map, at least one first window of data for the feature map is determined; The average pixel value of each feature point in the first window data is used to determine the first moment corresponding to each first window data. The first-order pooling data of the feature map is determined based on at least one of the first-order moments of the feature map.

5. The method according to claim 3, characterized in that, Based on the first-order pooling data, at least one Nth-order moment is calculated for feature points in each region of the feature map to obtain at least one Nth-order pooling data for the feature map, including: Based on the pooling kernel size and pooling step size of the feature map, at least one second window of data for the feature map is determined; Determine the first moment corresponding to each second window data in the first-order pooled data; Based on each second window data and the first moment corresponding to each second window data, determine the Nth moment corresponding to each second window data; The Nth-order pooling data of the feature map is determined based on at least one of the Nth-order moments of the feature map.

6. The method according to claim 3, characterized in that, The step of determining the pooling data of the feature map based on the first-order pooling data and the at least one N-order pooling data includes: Normalize the first-order pooled data and the at least one N-order pooled data respectively to determine the normalized first-order pooled data and the normalized at least one N-order pooled data. The normalized first-order pooling data and the normalized at least one N-order pooling data are combined to determine the pooling data of the feature map.

7. The method according to claim 1, characterized in that, The step of determining the quality evaluation result of the image to be evaluated based on the pooling data of the feature map includes: Perform an affine transformation on the pooled data of the feature map to determine the affine transformation result of the pooled data; The quality evaluation result of the image to be evaluated is determined based on the affine transformation result of the pooling data.

8. The method according to claim 1, characterized in that, The method is applied to an image quality assessment network, the training process of which includes: Obtain a training dataset, which includes multiple sample data and evaluation information for the sample data; The sample data is input into the image quality assessment network for processing to obtain the quality assessment result of the sample data. The image quality evaluation network is iteratively trained based on the evaluation information of the sample data in the training dataset, the quality evaluation results of the sample data, and the preset loss function to obtain a trained image quality evaluation network. The image quality assessment network includes at least one pooling layer for performing global moment pooling on the feature maps of the sample data to determine the pooling data of the feature maps.

9. An image quality evaluation device, characterized in that, include: The first determining module is used to extract features from the acquired image to be evaluated and determine at least one feature map of the image to be evaluated. The second determining module is used to perform global moment pooling on the feature map. Based on the first moment and at least one Nth moment of the feature points in each region of the feature map, the pooling data of the feature map is determined. The Nth moment is determined based on the Nth power of the difference between the pixel value of each feature point in each region and the corresponding first moment, where N is an integer greater than 1. The third determining module is used to determine the quality evaluation result of the image to be evaluated based on the pooling data of the feature map.

10. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images