A method and apparatus for predicting visual perceptual quality of an image, a terminal and a medium

Abstract

This invention provides a method, apparatus, terminal, and medium for predicting the visual perception quality of images. The method includes: acquiring an original image and a compressed image, and inputting them into a pre-trained deep learning prediction model. The deep learning prediction model includes a feature extraction module for parallel processing of the original and compressed images to extract multi-level features; a residual learning module for calculating the difference features at each level and residually connecting the difference features with the difference features processed by the residual blocks; a feature aggregation and pooling module based on multi-head attention for aggregating and pooling the features after residual concatenation based on a preset attention mechanism; and a regression prediction module based on MLP-Mixer for fusing the internal space and channel correlation of the aggregated feature vectors and outputting human satisfaction ratio and machine satisfaction ratio. This invention processes the simultaneously input source image and distorted image using a deep learning prediction model to output human satisfaction ratio and machine satisfaction ratio.

Description

Technical Field

[0001] This invention relates to the field of image quality evaluation technology, and in particular to a method, apparatus, terminal and medium for predicting the visual perception quality of images. Background Technology

[0002] With the rapid development of digital media technology, image data is growing explosively. These images are not only viewed by human users, but also used by machine vision algorithms for analysis and decision-making.

[0003] However, images often undergo lossy compression during storage and transmission, leading to quality degradation. To optimize compression efficiency, it is essential to accurately predict the visual perception quality of compressed images. Human satisfaction ratio and machine satisfaction ratio are commonly used to evaluate the visual perception quality of compressed images. These metrics not only effectively determine the current compression quality but also serve as optimization targets, guiding the design of video encoders and improving their compression efficiency.

[0004] However, traditional image quality assessment methods in existing technologies, such as peak signal-to-noise ratio, structural similarity, and various deep learning metrics, mainly focus on predicting a single quality score. They are difficult to directly reflect probabilistic indicators based on threshold distributions, such as human satisfaction ratio or machine satisfaction ratio. Moreover, although there are specific prediction models for human satisfaction ratio and prediction models for machine satisfaction ratio, they are usually treated as independent tasks, which leads to the following problems: First, the lack of a unified modeling mechanism ignores the inherent connection between human and machine perception. Existing solutions typically train independent models for human satisfaction ratio and machine satisfaction ratio as two separate tasks. This not only increases the computational resource consumption during actual deployment but also fails to leverage the common characteristics of both when perception is severely distorted to mutually assist in improving prediction accuracy.

[0005] Secondly, the method for calculating feature differences is too simplistic and fails to capture complex distortion information. Existing technologies often use simple pixel-by-pixel subtraction when calculating feature differences between the original and compressed images. This linear operation is unable to capture the nonlinear texture distortion and structural artifacts introduced by compression, resulting in insufficient ability to discriminate subtle distortions.

[0006] Finally, the feature fusion method is coarse, introducing a lot of redundancy. When fusing multi-scale features, existing methods often use simple concatenation or global average pooling, which cannot adaptively focus on the local regions or specific frequency bands that have the greatest impact on perception quality. As a result, key information is drowned out by background noise, affecting the accuracy of prediction.

[0007] Therefore, existing technologies have shortcomings and need to be improved and developed. Summary of the Invention

[0008] In view of this, the purpose of the present invention is to provide a method, device, terminal and medium for predicting the visual perception quality of images, which can simultaneously predict the human satisfaction ratio and the machine satisfaction ratio in a single network, thereby improving the prediction accuracy.

[0009] The technical solution adopted by this invention to solve the technical problem is as follows: In a first aspect, the present invention discloses a method for predicting the visual perception quality of an image, wherein the method includes: Obtain the original image and the compressed image; The original image and the compressed image are input into a pre-trained deep learning prediction model to output the predicted human satisfaction ratio and machine satisfaction ratio. The deep learning prediction model includes a feature extraction module, a differential feature residual learning module, a feature aggregation and pooling module based on multi-head attention, and a regression prediction module based on MLP-Mixer. Furthermore, the feature extraction module is used to process the original image and the compressed image in parallel to extract multi-level original image features and multi-level compressed image features; the differential feature residual learning module is used to calculate the differential features corresponding to each level between the original image features and the compressed image features, and to perform residual connection between the differential features and the differential features processed by the residual block to obtain the enhanced differential features corresponding to each level; the feature aggregation and pooling module based on multi-head attention is used to aggregate and pool the enhanced differential features based on a preset attention mechanism to obtain the aggregated feature vector; the regression prediction module based on MLP-Mixer is used to fuse the internal space and channel correlation of the aggregated feature vector and output the predicted human satisfaction ratio and the machine satisfaction ratio.

[0010] Optionally, the feature extraction module is a neural network model for extracting multi-level features, constructed based on the CAFormer-s36 backbone network. The parallel processing of the original image and the compressed image to extract multi-level features of the original image and multi-level features of the compressed image includes: The original image and the compressed image are processed in parallel to extract the original image features from the first to the fourth level and the compressed image features from the first to the fourth level.

[0011] Optionally, the residual block comprises three convolutional layers and two nonlinear activation functions; wherein, the computational expression corresponding to the residual concatenation of the differential features with the differential features processed by the residual block is: ; in, For the first Enhanced differential features corresponding to the level For the first The differences in characteristics corresponding to the levels These are the difference features after residual block processing. It is a non-linear activation function. In order to be with the first The first convolutional layer corresponding to the level, In order to be with the first The second convolutional layer corresponding to the level, In order to be with the first The third convolutional layer corresponds to the layer, and the first convolutional layer is a 1×1 convolutional layer, the second convolutional layer is a 3×3 convolutional layer, and the third convolutional layer is a 1×1 convolutional layer.

[0012] Optionally, the step of aggregating and pooling the enhanced differential features based on a preset attention mechanism to obtain the aggregated feature vector includes: The enhanced differential features corresponding to each level are upsampled to the same resolution, and the enhanced differential features corresponding to each level are channel-stitched to obtain the stitched features. The concatenated features are subjected to layer normalization to map them into a key matrix and a value matrix in a preset attention mechanism. Based on the preset query matrix, the key matrix, and the value matrix, the result of each attention head is calculated using the preset attention calculation formula; The results of each attention head are concatenated to obtain a concatenated result, and the concatenated result is then pooled and compressed to obtain an aggregated feature vector.

[0013] Optionally, the key matrix is: And the value matrix is: ; in, The key matrix, For value matrices, Features after splicing The first learnable weight matrix, It is a learnable second weight matrix.

[0014] Optionally, the preset attention calculation formula is: ; =C / h; in, The preset query matrix is ​​defined as C, which is the sum of the number of feature channels at all levels, and h is the number of attention heads.

[0015] Optionally, the pre-trained deep learning prediction model is trained by an end-to-end training method that minimizes the L1 loss between the predicted value and the true label.

[0016] Secondly, the present invention also discloses an image visual perception quality prediction device, wherein the device comprises: The image acquisition module is used to acquire raw and compressed images; The human-machine visual perception computing module is used to input the original image and the compressed image into a pre-trained deep learning prediction model to output the predicted human satisfaction ratio and machine satisfaction ratio. The deep learning prediction model includes a feature extraction module, a differential feature residual learning module, a feature aggregation and pooling module based on multi-head attention, and a regression prediction module based on MLP-Mixer. Furthermore, the feature extraction module is used to process the original image and the compressed image in parallel to extract multi-level original image features and multi-level compressed image features; the differential feature residual learning module is used to calculate the differential features corresponding to each level between the original image features and the compressed image features, and to perform residual connection between the differential features and the differential features processed by the residual block to obtain the enhanced differential features corresponding to each level; the feature aggregation and pooling module based on multi-head attention is used to aggregate and pool the enhanced differential features based on a preset attention mechanism to obtain the aggregated feature vector; the regression prediction module based on MLP-Mixer is used to fuse the internal space and channel correlation of the aggregated feature vector and output the predicted human satisfaction ratio and the machine satisfaction ratio.

[0017] Thirdly, the present invention discloses a terminal, comprising: a memory, a processor, and a visual perception quality prediction program for an image stored in the memory and executable on the processor, wherein the visual perception quality prediction program for the image, when executed by the processor, implements the steps of the visual perception quality prediction method for the image as described above.

[0018] Fourthly, the present invention discloses a computer-readable storage medium storing a computer program that can be executed to implement the steps of the image visual perception quality prediction method as described above.

[0019] This invention provides a method, apparatus, terminal, and medium for predicting the visual perception quality of images. The method includes: acquiring an original image and a compressed image; inputting the original image and the compressed image into a pre-trained deep learning prediction model to output predicted human satisfaction ratio and machine satisfaction ratio; wherein the deep learning prediction model includes a feature extraction module, a differential feature residual learning module, a feature aggregation and pooling module based on multi-head attention, and a regression prediction module based on MLP-Mixer; and the feature extraction module is used to perform parallel processing on the original image and the compressed image to extract multi-level features from the original image. The system employs a multi-level compressed image feature set. The differential feature residual learning module calculates the differential features corresponding to each level between the original image features and the compressed image features, and performs residual concatenation between the differential features and the differential features processed by residual blocks to obtain enhanced differential features corresponding to each level. The feature aggregation and pooling module based on multi-head attention aggregates and pools the enhanced differential features based on a preset attention mechanism to obtain aggregated feature vectors. The regression prediction module based on MLP-Mixer fuses the internal space and channel correlation of the aggregated feature vectors and outputs the predicted human satisfaction ratio and the machine satisfaction ratio. Therefore, this invention uses a unified deep learning prediction model that includes a feature extraction module, a differential feature residual learning module, a feature aggregation and pooling module based on multi-head attention, and a regression prediction module based on MLP-Mixer to process simultaneously input original and compressed images, and directly outputs the predicted human satisfaction ratio and machine satisfaction ratio of the compressed image. That is, it can simultaneously predict human satisfaction ratio and machine satisfaction ratio in a single network. Through a unified network structure, it effectively utilizes the complementarity of human and machine perception, significantly reduces prediction error, improves prediction accuracy, and the construction of a unified deep neural network architecture can significantly reduce the number of parameters deployed in the model and the inference time. Attached Figure Description

[0020] Figure 1 This is a flowchart of a preferred embodiment of the image visual perception quality prediction method in this invention; Figure 2 This is a schematic diagram of the composition structure of a specific differential feature residual learning module disclosed in this invention; Figure 3 This is a schematic diagram of the composition structure of a feature aggregation and pooling module based on multi-head attention disclosed in this invention; Figure 4 This is a schematic diagram of a specific compressed image visual perception quality prediction model structure disclosed in this invention; Figure 5This is a functional principle block diagram of a preferred embodiment of the image visual perception quality prediction device of the present invention. Figure 6 This is a functional principle block diagram of a preferred embodiment of the terminal in this invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0022] Currently, with the rapid development of digital media technology, image data is growing explosively. These images are not only viewed by human users, but also used by machine vision algorithms for analysis and decision-making.

[0023] However, images often undergo lossy compression during storage and transmission, leading to quality degradation. To optimize compression efficiency, it is essential to accurately predict the visual perception quality of compressed images. Human satisfaction ratio and machine satisfaction ratio are commonly used to evaluate the visual perception quality of compressed images. These metrics not only effectively determine the current compression quality but also serve as optimization targets, guiding the design of video encoders and improving their compression efficiency.

[0024] However, traditional image quality assessment methods in existing technologies, such as peak signal-to-noise ratio, structural similarity, and various deep learning metrics, mainly focus on predicting a single quality score. They are difficult to directly reflect probabilistic indicators based on threshold distributions, such as human satisfaction ratio or machine satisfaction ratio. Moreover, although there are specific prediction models for human satisfaction ratio and prediction models for machine satisfaction ratio, they are usually treated as independent tasks, which leads to the following problems: First, the lack of a unified modeling mechanism ignores the inherent connection between human and machine perception. Existing solutions typically train independent models for human satisfaction ratio and machine satisfaction ratio as two separate tasks. This not only increases the computational resource consumption during actual deployment but also fails to leverage the common characteristics of both when perception is severely distorted to mutually assist in improving prediction accuracy.

[0025] Secondly, the method for calculating feature differences is too simplistic and fails to capture complex distortion information. Existing technologies often use simple pixel-by-pixel subtraction when calculating feature differences between the original and compressed images. This linear operation is unable to capture the nonlinear texture distortion and structural artifacts introduced by compression, resulting in insufficient ability to discriminate subtle distortions.

[0026] Finally, the feature fusion method is coarse, introducing a lot of redundancy. When fusing multi-scale features, existing methods often use simple concatenation or global average pooling, which cannot adaptively focus on the local regions or specific frequency bands that have the greatest impact on perception quality. As a result, key information is drowned out by background noise, affecting the accuracy of prediction.

[0027] To this end, this application provides a visual perception quality prediction scheme for images, which can simultaneously predict human satisfaction ratio and machine satisfaction ratio in a single network, thereby improving prediction accuracy.

[0028] Please see Figure 1 , Figure 1 This is a flowchart of the image visual perception quality prediction method in this invention. For example... Figure 1 As shown, the image visual perception quality prediction method according to the embodiments of the present invention includes: Step S11: Obtain the original image and the compressed image.

[0029] In this embodiment, the original image and the corresponding compressed image are obtained. The compressed image is a distorted image formed by lossy compression of the original image during storage and transmission.

[0030] Step S12: Input the original image and the compressed image into a pre-trained deep learning prediction model to output the predicted human satisfaction ratio and machine satisfaction ratio. The deep learning prediction model includes a feature extraction module, a differential feature residual learning module, a multi-head attention-based feature aggregation and pooling module, and an MLP-Mixer-based regression prediction module. The feature extraction module processes the original image and the compressed image in parallel to extract multi-level original image features and multi-level compressed image features. The differential feature residual learning module calculates the differential features corresponding to each level between the original image features and the compressed image features, and performs residual concatenation between the differential features and the differential features processed by residual blocks to obtain enhanced differential features corresponding to each level. The multi-head attention-based feature aggregation and pooling module aggregates and pools the enhanced differential features based on a preset attention mechanism to obtain aggregated feature vectors. The MLP-Mixer-based regression prediction module fuses the internal space and channel correlation of the aggregated feature vectors to output the predicted human satisfaction ratio and machine satisfaction ratio.

[0031] It should be noted that the deep learning prediction model can be an end-to-end unified deep learning prediction model built on a deep neural network architecture, used to predict the joint human-machine visual perception quality score, namely the human satisfaction ratio and the machine satisfaction ratio. The pre-trained deep learning prediction model is trained end-to-end by minimizing the L1 loss between the predicted value and the true label. Here, the human satisfaction ratio represents the proportion of individuals who cannot perceive compression distortion; the machine satisfaction ratio represents the proportion of machines whose recognition results in the compressed image are consistent with the original image. Using these two metrics to evaluate the visual perception quality of compressed images can not only effectively determine the current compression quality but also serve as optimization targets to guide the design of video encoders, improve compression efficiency, and achieve compression efficiency optimization.

[0032] It is understandable that a unified deep learning prediction model can simultaneously input the original image and its corresponding compressed image, and output the predicted human-machine joint visual perception quality score, i.e., the human satisfaction ratio and the machine satisfaction ratio. In this deep learning prediction model, a shared feature extraction module can acquire multi-scale features of the original and compressed images; a differential feature residual learning module deeply mines the nonlinear differences between the original and compressed images; a feature aggregation and pooling module based on multi-head attention adaptively selects and fuses key features; and finally, an MLP-Mixer-based regression prediction module predicts the human and machine satisfaction ratios. In this embodiment, the unified network structure effectively utilizes the complementarity of human and machine perception, significantly reducing prediction errors and improving prediction accuracy. Furthermore, the unified deep neural network architecture design significantly reduces the number of parameters and inference time required for model deployment, providing accurate joint quality evaluation metrics for rate-distortion optimization of image coding systems.

[0033] In one specific implementation, the feature extraction module can be a neural network model built on the CAFormer-s36 backbone network for extracting multi-level features. Specifically, in this embodiment, the original image and the compressed image are processed in parallel to extract multi-level original image features and multi-level compressed image features. This can include: processing the original image and the compressed image in parallel to extract the first to fourth levels of original image features and the first to fourth levels of compressed image features. It is understood that the feature extraction module can be the CAFormer-s36 backbone network with a hybrid convolution and attention mechanism. This backbone network has undergone large-scale pre-training and can extract features (i.e., feature maps) at four different scales. These features contain rich features ranging from low-level texture details to high-level semantic information. For example, low-level features reflect the texture details of the image, which are crucial for predicting human satisfaction ratio; while high-level features reflect the semantic characteristics of the image and significantly contribute to predicting machine satisfaction ratio. Specifically, the features output from the first n layers of the backbone network can be identified as low-level features, which contain image information, and the features output from the last m layers of the backbone network can be identified as high-level features, which contain semantic information of the image.

[0034] It should be noted that the differential feature residual learning module, based on operations such as local spatial modeling, channel reorganization, and nonlinear activation, obtains more expressive differential features, i.e., enhanced differential features, through residual learning. Specifically, for each level of features output by the feature extraction module, the initial difference between the original image features and the compressed image features is first calculated to obtain the corresponding differential features. These differential features are then fed into a residual block containing a multi-layer convolutional structure. For example, this residual block sequentially processes the differential features by performing a 1x1 convolutional layer for channel mixing, a 3x3 convolutional layer for extracting spatial structural differences, and a nonlinear activation function. Finally, the differential features processed by the residual block are added to the initial difference to obtain the enhanced differential features. This allows the network to learn more discriminative and structured enhanced differential features than simple subtraction, effectively capturing the complex artifacts and structural distortions introduced by the compression process. In particular, the differential feature residual learning module uses a residual block structure containing convolution and nonlinear activation to replace the traditional linear feature subtraction, greatly enhancing the model's ability to capture subtle nonlinear differences such as compression artifacts, thereby improving the sensitivity of quality prediction.

[0035] In one specific implementation, the residual block comprises three convolutional layers and two nonlinear activation functions; wherein, the computational expression corresponding to the residual concatenation of the differential features with the differential features processed by the residual block is: ; in, For the first Enhanced differential features corresponding to the level For the first The differences in characteristics corresponding to the levels These are the difference features after residual block processing. It is a non-linear activation function. In order to be with the first The first convolutional layer corresponding to the level, In order to be with the first The second convolutional layer corresponding to the level, In order to be with the first The third convolutional layer corresponds to the first level, which is a 1×1 convolutional layer, the second is a 3×3 convolutional layer, and the third is a 1×1 convolutional layer. Furthermore, when the feature extraction module outputs features from four different levels, It can be 1, 2, 3, 4, that is... =1, 2, 3, 4.

[0036] For example, see Figure 2 As shown, the differential feature residual learning module first calculates the original map feature F0 and the compressed map feature F... qk Different features between different levels The features are then fed into a 1×1 convolutional layer for differential feature recombination to enable cross-channel interaction, allowing information-related channels to be more closely linked to similar locations. Next, a 3×3 convolutional layer is used to increase the receptive field, thereby modeling the perception of differential patterns in local spatial locations. Then, another 1×1 convolutional layer is used to recombine and integrate the feature channels. Finally, the differential features processed by the residual block are joined with the original differential features using a residual connection to ensure the smoothness and stability of the learning process, resulting in the final enhanced differential features. In this process, after the first two convolutional layers, the GELU activation function is used to enhance nonlinear expressive power, while its soft thresholding property is used to suppress noise response. Through an end-to-end learning process, the parameters of the convolutional layers are automatically adjusted, which can highlight the importance of each element in the differential features.

[0037] Therefore, the differential feature residual learning module explicitly enhances the learning of differential features, enabling the network to better utilize information from both the source and distorted images and obtain a more expressive differential feature. This is more helpful for predicting SUR (Satisfied User Ratio) and SMR (Satisfied Machine Ratio). Simultaneously, the differential feature residual learning module can automatically adjust the importance of features at different locations and in different channels through learnable weights, reducing the impact of meaningless noise in the features and improving the overall efficiency of feature utilization.

[0038] In one specific implementation, the enhanced differential features are aggregated and pooled to obtain an aggregated feature vector based on a preset attention mechanism. Specifically, this may include: upsampling the enhanced differential features corresponding to each level to the same resolution, and concatenating the channels of the enhanced differential features corresponding to each level to obtain concatenated features; performing layer normalization on the concatenated features to map the concatenated features to the key matrix and value matrix in the preset attention mechanism; calculating the result of each attention head based on the preset query matrix, key matrix, and value matrix, and using a preset attention calculation formula; concatenating the results of each attention head to obtain a concatenated result, and pooling and compressing the concatenated result to obtain an aggregated feature vector.

[0039] Understandably, directly concatenating the enhanced differential features at multiple levels and different resolutions output by the differential feature residual learning module would result in excessively high dimensionality and a large amount of redundant information. Therefore, a feature aggregation and pooling module based on multi-head attention can aggregate and compress these multi-level enhanced differential features into a single representation that simultaneously reflects the perceptual characteristics of both the Human Visual System (HVS) and the Machine Visual System (MVS), while also reducing the amount of feature data. Specifically, firstly, all levels of enhanced differential features are adjusted to a uniform spatial resolution and concatenated. Then, a set of learnable query vectors is used, and the multi-head attention mechanism allows these query vectors to interact with the concatenated features. This process enables the deep learning prediction model to automatically learn and focus on the regions in the image most sensitive to quality assessment (such as regions with complex textures or salient objects), while suppressing background noise. The output of the attention mechanism undergoes linear mapping and pooling compression and dimensionality reduction, ultimately generating a compact global feature vector rich in key information—the aggregated feature vector. The feature aggregation and pooling module based on multi-head attention can intelligently filter high-value information and remove redundancy from multi-scale, high-dimensional features, solving the problems of dimensionality curse and noise interference in multi-level feature fusion, and realizing adaptive extraction of key perceptual features.

[0040] It should be noted that the key matrix is: And the value matrix is: ;in, The key matrix, For value matrices, Features after splicing The first learnable weight matrix, It is a learnable second weight matrix.

[0041] The default formula for calculating attention is: ; =C / h; where is a preset query matrix, C is the sum of the number of feature channels at all levels, and h is the number of attention heads.

[0042] For example, as shown in Figure 3 for the multi-level enhanced differential features , by upsampling the high-level features, retaining the local detail information of the low-level features and the spatial structure information of all features, and then pooling and compressing them to reduce the overall data volume. That is, first uniformly upsample (Upsampling) the multi-level enhanced differential features to the same resolution H×W, and splice them in the channel dimension to obtain an implicit feature , that is, the spliced feature, where C is the sum of the number of feature channels at all levels. Then, perform a layer normalization (LayerNorm) on the spliced feature , which can balance the differences of features from different levels and different distributions, making the overall training more stable. Then, determine the spliced feature as a feature sequence containing N = H×W spatial tokens, and then map it to the key matrix K and value matrix V in the preset attention mechanism through two linear layers respectively. And in order to better aggregate information from different levels in the tokens and at the same time reduce the noise in the features or the redundancy caused by the correlation of features at different levels, a learnable query matrix can be used, that is , where Y is much smaller than N. This learnable query matrix Q can complete the adaptive screening and more efficient fusion of features by reducing the number of tokens, and can focus on the feature patterns that are most critical for the SUR and SMR prediction tasks. Preset the number of attention heads as h, calculate the results of each attention head, and finally splice the results of the h attention heads together, and then pool them through a linear layer into the final compact representation , where Z < C, further reducing the data volume of the representation, thus completing the aggregation and compression of different-level differential features.

[0043] It should be noted that each attention head can learn differentiated attention patterns for different levels, different local regions, and different channels. Through adaptive weighting, each level, position, and channel is dynamically filtered, which not only preserves the low-level feature differences that HVS can more easily perceive, but also does not ignore the high-level concepts that are key to MVS, and can also model the correlation between the two. Secondly, the learnable query matrix and pooling operation, the former can reduce the overall data volume by setting it to contain only a relatively low number of tokens, and can also enable the network to learn the differential feature information that is most important for SUR and SMR prediction tasks.

[0044] In one specific implementation, when the aggregated feature vector is input to the MLP-Mixer-based regression prediction module, the MLP-Mixer-based regression prediction module includes alternating spatial mixing layers and channel mixing layers, as well as two independent fully connected regression heads. Through the alternating spatial mixing layers and channel mixing layers, the spatial and channel correlations within the aggregated feature vector are further integrated, and through the two independent fully connected regression heads, the predicted human satisfaction ratio and machine satisfaction ratio are output respectively.

[0045] As can be seen, in this embodiment of the invention, a unified deep learning prediction model that includes a feature extraction module, a differential feature residual learning module, a feature aggregation and pooling module based on multi-head attention, and a regression prediction module based on MLP-Mixer processes the simultaneously input original image and compressed image, and directly outputs the predicted human satisfaction ratio and machine satisfaction ratio of the compressed image. That is, it can simultaneously predict the human satisfaction ratio and machine satisfaction ratio in a single network. Through the unified network structure, it effectively utilizes the complementarity of human and machine perception, significantly reduces prediction error, improves prediction accuracy, and the construction of a unified deep neural network architecture significantly reduces the number of parameters deployed in the model and the inference time.

[0046] For example, unlike existing technologies that model independently, a unified deep neural network architecture for joint human-machine prediction can simultaneously predict both human and machine satisfaction ratios within a single network. See also Figure 4 As shown, the original image I0 and the compressed image I0 are... qk The input is fed into a pre-trained deep learning prediction model, where it first passes through a feature extractor (which acts as a feature extraction module) to extract multi-level original image features F0 and multi-level compressed image features F. qk Then, for each level of features output by the feature extraction module, the DFRL (Difference Feature Residual Learning) module first calculates the initial difference between the features of the original image and the features of the compressed image. The differential features are then fed into a residual block containing multiple convolutional layers within the DFRL module. This residual block processes the differential features sequentially through a 1x1 convolutional layer performing channel mixing, a 3x3 convolutional layer extracting spatial structural differences, and a non-linear activation function. Finally, the differential features processed by the residual block are added to the initial difference to obtain the enhanced differential features, which are then processed by MHAAP (Multi-Head Attention Aggregation and...). The pooling module (feature aggregation and pooling based on multi-head attention) can aggregate and compress multi-level enhanced differential features into a single representation that reflects both HVS and MVS perceptual characteristics with a smaller feature data volume—the aggregated feature vector. This aggregated feature vector is then input into an MLP-Mixer-based regression prediction module. This module includes alternating spatial and channel mixing layers, as well as two independent fully connected regression heads. The alternating spatial and channel mixing layers further integrate the spatial and channel correlations within the aggregated feature vector. The two independent fully connected regression heads output the predicted SMR and SUR, respectively. Each fully connected regression head includes layer normalization, a linear layer, and an activation function. This technical solution leverages the inherent correlation between human and machine visual perception to achieve mutual promotion between tasks, simultaneously outputting both human and machine satisfaction ratios, improving prediction accuracy, and significantly reducing deployment costs.

[0047] It should be noted that the prediction accuracy on typical SUR and SMR datasets is shown in Table 1. The prediction error is represented by the mean absolute error, with a value range of 0 to 1, and the smaller the value, the better.

[0048] Table 1

[0049] As shown in Table 1, the SUR and SMR prediction methods proposed in this invention achieve excellent prediction results. Specifically, on the SUR-SMR dataset containing the first four datasets, the SUR prediction error is 0.0570 and the SMR prediction error is 0.0331 using the SUR and SMR prediction methods proposed in this invention.

[0050] In one embodiment, such as Figure 5 As shown, based on the above-described method for predicting the visual perception quality of images, the present invention also provides a device for predicting the visual perception quality of images, comprising: Image acquisition module 11 is used to acquire raw images and compressed images; The human-machine visual perception computing module 12 is used to input the original image and the compressed image into a pre-trained deep learning prediction model to output the predicted human satisfaction ratio and machine satisfaction ratio. The deep learning prediction model includes a feature extraction module, a differential feature residual learning module, a feature aggregation and pooling module based on multi-head attention, and a regression prediction module based on MLP-Mixer. Furthermore, the feature extraction module is used to process the original image and the compressed image in parallel to extract multi-level original image features and multi-level compressed image features; the differential feature residual learning module is used to calculate the differential features corresponding to each level between the original image features and the compressed image features, and to perform residual connection between the differential features and the differential features processed by the residual block to obtain the enhanced differential features corresponding to each level; the feature aggregation and pooling module based on multi-head attention is used to aggregate and pool the enhanced differential features based on a preset attention mechanism to obtain the aggregated feature vector; the regression prediction module based on MLP-Mixer is used to fuse the internal space and channel correlation of the aggregated feature vector and output the predicted human satisfaction ratio and the machine satisfaction ratio.

[0051] Furthermore, it is worth noting that the working process of the image visual perception quality prediction device provided in this embodiment is the same as the working process of the image visual perception quality prediction method described above, and will not be repeated here. For details, please refer to the working process of the image visual perception quality prediction method described above.

[0052] Figure 6 A schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0053] When the processor 502 executes the program, it implements the image visual perception quality prediction method provided in the above embodiments.

[0054] Furthermore, the terminal also includes: Communication interface 503 is used for communication between memory 501 and processor 502.

[0055] The memory 501 is used to store computer programs that can run on the processor 502.

[0056] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0057] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0058] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0059] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0060] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for predicting the visual perception quality of images.

[0061] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0062] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0063] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can read and execute instructions from and from an instruction execution system, apparatus or device).

[0064] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0065] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for predicting the visual perception quality of an image, characterized in that, The method includes: Obtain the original image and the compressed image; The original image and the compressed image are input into a pre-trained deep learning prediction model to output the predicted human satisfaction ratio and machine satisfaction ratio. The deep learning prediction model includes a feature extraction module, a differential feature residual learning module, a feature aggregation and pooling module based on multi-head attention, and a regression prediction module based on MLP-Mixer. Furthermore, the feature extraction module is used to process the original image and the compressed image in parallel to extract multi-level original image features and multi-level compressed image features; the differential feature residual learning module is used to calculate the differential features corresponding to each level between the original image features and the compressed image features, and to perform residual connection between the differential features and the differential features processed by the residual block to obtain the enhanced differential features corresponding to each level; the feature aggregation and pooling module based on multi-head attention is used to aggregate and pool the enhanced differential features based on a preset attention mechanism to obtain the aggregated feature vector; the regression prediction module based on MLP-Mixer is used to fuse the internal space and channel correlation of the aggregated feature vector and output the predicted human satisfaction ratio and the machine satisfaction ratio.

2. The method for predicting the visual perception quality of an image according to claim 1, characterized in that, The feature extraction module is a neural network model built on the CAFormer-s36 backbone network for extracting multi-level features. The parallel processing of the original image and the compressed image to extract multi-level features of the original image and multi-level features of the compressed image includes: The original image and the compressed image are processed in parallel to extract the original image features from the first to the fourth level and the compressed image features from the first to the fourth level.

3. The method for predicting the visual perception quality of an image according to claim 1, characterized in that, The residual block contains three convolutional layers and two nonlinear activation functions; the computational expression corresponding to the residual concatenation between the differential features and the differential features processed by the residual block is as follows: ； in, For the first Enhanced differential features corresponding to the level For the first The differences in characteristics corresponding to the levels These are the difference features after residual block processing. It is a non-linear activation function. In order to be with the first The first convolutional layer corresponding to the level, In order to be with the first The second convolutional layer corresponding to the level, In order to be with the first The third convolutional layer corresponds to the layer, and the first convolutional layer is a 1×1 convolutional layer, the second convolutional layer is a 3×3 convolutional layer, and the third convolutional layer is a 1×1 convolutional layer.

4. The method for predicting the visual perception quality of an image according to claim 1, characterized in that, The process of aggregating and pooling the enhanced differential features based on a preset attention mechanism to obtain the aggregated feature vector includes: The enhanced differential features corresponding to each level are upsampled to the same resolution, and the enhanced differential features corresponding to each level are channel-stitched to obtain the stitched features. The concatenated features are subjected to layer normalization to map them into a key matrix and a value matrix in a preset attention mechanism. Based on the preset query matrix, the key matrix, and the value matrix, the result of each attention head is calculated using the preset attention calculation formula; The results of each attention head are concatenated to obtain a concatenated result, and the concatenated result is then pooled and compressed to obtain an aggregated feature vector.

5. The method for predicting the visual perception quality of an image according to claim 4, characterized in that, The key matrix is: And the value matrix is: ; in, The key matrix, For value matrices, Features after splicing The first learnable weight matrix, It is a learnable second weight matrix.

6. The method for predicting the visual perception quality of an image according to claim 5, characterized in that, The preset attention calculation formula is: ； =C / h; in, The preset query matrix is ​​defined as C, which is the sum of the number of feature channels at all levels, and h is the number of attention heads.

7. The method for predicting the visual perception quality of an image according to any one of claims 1 to 6, characterized in that, The pre-trained deep learning prediction model is trained in an end-to-end manner by minimizing the L1 loss between the predicted value and the true label.

8. A device for predicting the visual perception quality of an image, characterized in that, The device includes: The image acquisition module is used to acquire raw and compressed images; The human-machine visual perception computing module is used to input the original image and the compressed image into a pre-trained deep learning prediction model to output the predicted human satisfaction ratio and machine satisfaction ratio. The deep learning prediction model includes a feature extraction module, a differential feature residual learning module, a feature aggregation and pooling module based on multi-head attention, and a regression prediction module based on MLP-Mixer. Furthermore, the feature extraction module is used to process the original image and the compressed image in parallel to extract multi-level original image features and multi-level compressed image features; the differential feature residual learning module is used to calculate the differential features corresponding to each level between the original image features and the compressed image features, and to perform residual connection between the differential features and the differential features processed by the residual block to obtain the enhanced differential features corresponding to each level; the feature aggregation and pooling module based on multi-head attention is used to aggregate and pool the enhanced differential features based on a preset attention mechanism to obtain the aggregated feature vector; the regression prediction module based on MLP-Mixer is used to fuse the internal space and channel correlation of the aggregated feature vector and output the predicted human satisfaction ratio and the machine satisfaction ratio.

9. A terminal, characterized in that, include: The image includes a memory, a processor, and a visual perception quality prediction program for an image stored in the memory and executable on the processor, wherein the visual perception quality prediction program for an image, when executed by the processor, implements the steps of the visual perception quality prediction method for an image as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed to implement the steps of the image visual perception quality prediction method as described in any one of claims 1 to 7.

Images