A local region mask based full-reference image quality assessment method and system

Abstract

The application discloses a full-reference image quality evaluation method and system based on a local region mask, relates to the technical field of image quality evaluation, and specifically comprises the following steps: local region information extraction is respectively performed on a distorted image and a reference image to obtain an initial local region feature map; multi-stage feature extraction is performed on the initial local region feature map to obtain an initial deep feature map; a local region mask is generated based on the initial local region feature map of the reference image, and a local region deep feature is generated according to the mask; an enhanced local region deep feature is generated according to the local region deep feature by using a self-attention mechanism and a cross-attention mechanism; the enhanced local region deep feature is weighted and aggregated by using a weight matrix to generate a weighted fusion local region deep feature, the weighted fusion local region deep feature is spliced with the initial local region feature map, and a quality score is calculated. The application makes the predicted quality score closer to the perception result of the human eye, and provides an evaluation standard more in line with the human visual characteristics for image quality evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image quality evaluation, and more particularly to a full-reference image quality evaluation method and system based on local region masks. BACKGROUND

[0002] Image quality evaluation is a fundamental and critical task in the field of image processing and computer vision, and its importance is self-evident. This technology is widely used in image / video encoding, super-resolution reconstruction, image / video visual quality enhancement, and other fields, providing strong support for objective measurement and optimization of image quality. The core is to build a model that can simulate the perception mechanism of the human visual system, and through digital calculation, the overall quality of the image is quantitatively evaluated, so as to realize objective and accurate image perception quality evaluation.

[0003] In the full-reference image quality evaluation method, the algorithm compares the human eye perception difference between the reference image (i.e. the original high-quality image) and the image to be evaluated (i.e. the distorted image that may have been processed or compressed) to estimate the quality score of the distorted image. This method is intuitive and accurate, and plays an irreplaceable role in image compression encoding evaluation, image processing algorithm test benchmark setting, and image processing task optimization target setting.

[0004] However, traditional full-reference image quality evaluation methods often focus on global quality evaluation and may not be sensitive to changes in local regions of the image. In practical applications, certain regions of the image may have a more important impact on overall quality perception, such as faces, text, or key objects. Therefore, how to effectively extract features of important regions and improve the accuracy of image quality evaluation is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application provides a full-reference image quality evaluation method and system based on local region masks, which overcomes the above-mentioned defects.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] A full-reference image quality evaluation method based on local region masks, the specific steps are:

[0008] Obtain a distorted image and a reference image;

[0009] Perform local region information extraction on the distorted image and the reference image respectively to obtain initial local region distortion feature maps and initial local region reference feature maps;

[0010] performing multi-stage feature extraction on the distorted image and the reference image respectively and generating an initial distorted depth feature map and an initial reference depth feature map;

[0011] generating a plurality of local area masks based on the initial local area reference feature map, wherein the plurality of local area masks are respectively used to generate a plurality of local area distorted depth features and a plurality of local area reference depth features from the initial distorted depth feature map and the initial reference depth feature map;

[0012] generating enhanced local area reference depth features based on a self-attention mechanism by fusing the plurality of local area reference depth features, and generating enhanced local area distorted depth features based on a cross-attention mechanism by taking the enhanced local area reference depth features as guide features and according to the plurality of local area distorted depth features;

[0013] generating a weight matrix according to the enhanced local area reference depth features, and performing weighted aggregation on the enhanced local area reference depth features and the enhanced local area distorted depth features respectively by using the weight matrix to generate weighted fusion local area reference depth features and weighted fusion local area distorted depth features;

[0014] splicing the weighted fusion local area reference depth features, the weighted fusion local area distorted depth features, the initial local area distorted feature map and the initial local area reference feature map, and calculating a quality score.

[0015] Optionally, after obtaining the distorted image and the reference image, the distorted image and the reference image need to be preprocessed.

[0016] Optionally, the step of obtaining the initial local area distorted feature map or the initial local area reference feature map comprises:

[0017] inputting the distorted image or the reference image into a local area sampling network, and generating the initial local area distorted feature map or the initial local area reference feature map through multiple iterations.

[0018] Optionally, the step of obtaining the initial distorted depth feature map or the initial reference depth feature map comprises:

[0019] extracting depth features of the distorted image or the reference image according to a depth extraction model, and selecting a plurality of stages of the depth features as pre-depth feature maps;

[0020] up-sampling the pre-depth feature maps to the size of the distorted image or the reference image to generate the initial distorted depth feature map or the initial reference depth feature map.

[0021] Optionally, the obtaining step of the local region distortion depth feature or the local region reference depth feature is:

[0022] The local region mask is generated according to the initial local region reference feature map, and the number of the local region mask is the same as the number of the local region;

[0023] The initial distortion depth feature map or the initial reference depth feature map is multiplied by each of the local region mask to obtain the selected local region distortion feature or the selected local region reference feature;

[0024] The selected local region distortion feature or the selected local region reference feature is globally average-pooled to obtain the local region distortion depth feature or the local region reference depth feature.

[0025] Optionally, the obtaining step of the weighted fusion local region reference depth feature or the weighted fusion local region distortion depth feature is:

[0026] The weight matrix is generated according to the enhanced local region reference depth feature using a multi-layer perception and a Softmax function;

[0027] The weighted fusion local region reference depth feature or the weighted fusion local region distortion depth feature is generated by weighted averaging the enhanced local region reference depth feature or the enhanced local region distortion depth feature at different stages according to the weight matrix.

[0028] Optionally, the quality score obtaining step is:

[0029] The initial local region distortion feature map and the weighted fusion local region distortion depth feature or the initial local region reference feature map and the weighted fusion local region reference depth feature are spliced in the channel dimension to obtain a distortion image splicing feature or a reference image splicing feature;

[0030] The distortion image splicing feature and the reference image splicing feature are spliced in the channel dimension to obtain an overall splicing image feature;

[0031] The quality score is calculated based on the overall splicing image feature.

[0032] A local region mask-based full-reference image quality evaluation system, comprising:

[0033] An image acquisition module is configured to acquire a distortion image and a reference image;

[0034] A local region division module is configured to extract local region information from the distortion image and the reference image to obtain an initial local region distortion feature map and an initial local region reference feature map.

[0035] a deep feature extraction module configured to perform multi-stage feature extraction on the distorted image and the reference image respectively and generate an initial distorted deep feature map and an initial reference deep feature map;

[0036] a deep feature processing module configured to generate a plurality of local region masks based on the initial local region reference feature map, and generate a plurality of local region distorted deep features and a plurality of local region reference deep features from the initial distorted deep feature map and the initial reference deep feature map respectively based on the plurality of local region masks;

[0037] a feature enhancement module configured to generate enhanced local region reference deep features by fusing a plurality of local region reference deep features based on a self-attention mechanism, and generate enhanced local region distorted deep features based on the enhanced local region reference deep features as a guide feature according to a plurality of the local region distorted deep features based on a cross-attention mechanism;

[0038] a weighted fusion module configured to generate a weight matrix according to the enhanced local region reference deep features, and perform weighted aggregation on the enhanced local region reference deep features and the enhanced local region distorted deep features respectively by using the weight matrix to generate weighted fusion local region reference deep features and weighted fusion local region distorted deep features;

[0039] a quality evaluation module configured to splice the weighted fusion local region reference deep features, the weighted fusion local region distorted deep features, the initial local region distorted feature map and the initial local region reference feature map, and calculate a quality score.

[0040] According to the technical solution described above, the present disclosure provides a full-reference image quality evaluation method and system based on local region masks, which has the following beneficial effects compared with the prior art:

[0041] 1. Improving the accuracy of image quality evaluation: the present disclosure can accurately extract the local region features and deep features of the reference image and the distorted image by using the local region mask technology, and combine the self-attention mechanism and the cross-attention mechanism to make the model pay more attention to the key local regions in the image, while using global information to guide the learning of each local region; not only does it introduce information from other local regions, but also enables the model to extract more fine-grained features, thereby improving the accuracy and reliability of image quality evaluation;

[0042] 2. Enhance the sensitivity of the model to local details: the deep features extracted by the local region mask, combined with the pooling operation, can fuse the features of each pixel point in the local region, providing a good foundation for subsequent feature extraction; it makes the model more sensitive to the local details of the image, can capture more rich image information, and helps to improve the fineness and recognition of quality evaluation;

[0043] 3. Improve the stability of quality score prediction: the invention generates a weight matrix using the local region deep features of the reference image, and performs weighted averaging on the reference features and distorted features to obtain the final weighted feature map; it adaptively generates the weight coefficients of the local region features in different scales in the image, so as to more accurately reflect the quality status of different regions in the image; improve the stability of quality score prediction, so that the prediction result is more reliable;

[0044] 4. Optimize the computational efficiency and generalization ability: the invention combines the advantages of local region features and global information, and realizes effective fusion and extraction of features through attention mechanism, which not only improves the computational efficiency, but also makes the model have stronger generalization ability; it can be applied to different types of distorted images and reference images, and provides a more general and effective solution for image quality evaluation;

[0045] 5. Quality score prediction closer to human eye perception: the invention concatenates the extracted weighted features and the initial local region features, and then performs supervised learning through a multi-layer perception machine to regress the quality score between the reference image and the distorted image. This method makes the predicted quality score closer to the perception result of the human eye, and provides an evaluation standard that is more consistent with the characteristics of human vision for image quality evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0047] Figure 1 The flowchart of the full reference image quality evaluation method based on local region mask provided by the present application;

[0048] Figure 2 The architecture diagram of the full reference image quality evaluation method based on local region mask provided by the present application. DETAILED DESCRIPTION

[0049] Clearly, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0050] The embodiment discloses a full-reference image quality evaluation method based on local region masks, as shown in the following formula (1), and the specific steps are as follows. Figure 1

[0051] S1, obtaining a distorted image and a reference image;

[0052] S2, extracting local region information from the distorted image and the reference image respectively to obtain an initial local region distortion feature map and an initial local region reference feature map;

[0053] S3, performing multi-stage feature extraction on the distorted image and the reference image respectively to generate an initial distorted depth feature map and an initial reference depth feature map;

[0054] S4, generating a plurality of local region masks based on the initial local region reference feature map, and generating a plurality of local region distortion depth features and a plurality of local region reference depth features based on the plurality of local region masks, the initial distorted depth feature map and the initial reference depth feature map;

[0055] S5, generating an enhanced local region reference depth feature based on a self-attention mechanism by fusing a plurality of local region reference depth features, and generating an enhanced local region distortion depth feature based on a cross-attention mechanism by taking the enhanced local region reference depth feature as a guide feature and according to a plurality of local region distortion depth features;

[0056] S6, generating a weight matrix according to the enhanced local region reference depth feature, and performing weighted aggregation on the enhanced local region reference depth feature and the enhanced local region distortion depth feature respectively by using the weight matrix to generate a weighted fusion local region reference depth feature and a weighted fusion local region distortion depth feature;

[0057] S7, splicing the weighted fusion local region reference depth feature, the weighted fusion local region distortion depth feature, the initial local region distortion feature map and the initial local region reference feature map, and calculating a quality score.

[0058] In an embodiment, in S1, after obtaining the distorted image and the reference image, the distorted image and the reference image need to be preprocessed.

[0059] ​Further, in S1, the distorted image and the reference image are first obtained, and then the images are uniformly adjusted to a size of 224x224 pixels using image processing software or library, ensuring consistency of subsequent processing and effectiveness of comparison.

[0060] As shown in FIG. 1, in an embodiment, in S2, the initial local region distorted feature map or the initial local region reference feature map is obtained by: Figure 2

[0061] The distorted image or the reference image is input into the local region sampling network, and the initial local region distorted feature map or the initial local region reference feature map is generated through multiple iterations.

[0062] Further, in S2, the reference image and the distorted image uniformly adjusted to the same size are input into the local region sampling network for local region segmentation of the reference image and the distorted image, and the corresponding local region images of the image are generated through multiple iterations (in this embodiment, the number of iterations is set to 10), and the extracted initial local region feature map (including the initial local region distorted feature map or the initial local region reference feature map) is output, which contains the texture, color and shape information of the image.

[0063] In S3, the initial distorted depth feature map or the initial reference depth feature map is obtained by:

[0064] S31, extracting the depth features of the distorted image or the reference image according to the depth extraction model, and selecting the depth features of multiple stages as the pre-depth feature map;

[0065] S32, upsampling the pre-depth feature map to the size of the distorted image or the reference image to generate the initial distorted depth feature map or the initial reference depth feature map.

[0066] Further, the depth extraction model, i.e., the pre-trained VGG-16 neural network model, is used to extract the depth features of the reference image and the distorted image, and the output feature maps of the 2nd, 3rd and 4th stages in the VGG-16 network are taken out as the pre-depth feature map, which represent different levels of visual features of the image. Then, the channel numbers of the output feature maps of the three stages are unified to D through a 1x1 convolution layer, and the feature maps are upsampled to the original image size (224*224) using the bilinear interpolation method to generate the initial depth feature map, which includes the initial distorted depth feature map and the initial reference depth feature map.

[0067] In an embodiment, in S4, the local region distorted depth feature or the local region reference depth feature is obtained by:

[0068] ​S41, generating a local region mask according to the initial local region reference feature map, the number of local region masks being the same as the number of local regions;

[0069] S42, multiplying the initial distortion depth feature map or the initial reference depth feature map by each local region mask to obtain the selected local region distortion feature or the selected local region reference feature;

[0070] S43, performing global average pooling on the selected local region distortion feature or the selected local region reference feature to obtain the local region distortion depth feature or the local region reference depth feature.

[0071] Further, in S4, a local region mask is generated according to the initial local region reference feature map. For each local region, a local region mask is generated, which allows passing through the current local region and masking all other local regions. Then, the initial depth feature map is multiplied by the mask to extract the depth feature of each local region, and the expression is:

[0072] F ij =M i ⊙D j ;

[0073] Wherein,

[0074] In the formula, M i is the i-th local region mask; D j is the initial depth feature map of the j-th stage; ⊙ represents element-wise multiplication (Hadamard multiplication); F ij is the feature of the j-th stage initial depth feature map in the i-th local region.

[0075] Further, the initial distortion depth feature map and the initial reference depth feature map are multiplied by each mask to obtain the selected local region feature, which includes the selected local region distortion feature or the selected local region reference feature;

[0076] The selected local region feature is globally averaged and pooled to obtain a feature vector with a length and width of 1, which is the feature vector corresponding to the local region, and finally all local region depth feature vectors are obtained.

[0077] In an embodiment, in S5, for the local region reference depth feature, a self-attention mechanism is used, and the global local region feature is used to guide each local region feature to learn more rich features of the reference global information, and then to fuse other local regions to obtain an enhanced local region reference depth feature guided by the global information;

[0078] For the distorted image, the enhanced local region reference depth feature is used as the guide feature, and the cross-attention technology is adopted to guide the local region distorted depth feature to learn the feature map of the comprehensive attention mechanism, that is, the enhanced local region distorted depth feature.

[0079] Further, for the reference image, the self-attention mechanism is adopted to fuse the depth features between the local regions through the global average pooling and the multi-head attention mechanism to obtain the enhanced local region reference depth feature. For the distorted image, the cross-attention mechanism is used to extract the features of the distorted image by taking the enhanced local region reference depth feature as the guide, and the expression is:

[0080] Q R =F R W Q

[0081] K R =F R W K ;

[0082] V R =F R W V

[0083]

[0084] In the formula, W Q , W K , W V , are all learnable weight matrices; F R is the local region reference depth feature; F D is the local region distorted depth feature; Q R and Q D represent the queries of the reference image and the distorted image, respectively; K R and K D and V R and V D represent the keys and values of the corresponding image, respectively. and represent the enhanced local region distorted depth feature and the enhanced local region reference depth feature, respectively.

[0085] In an embodiment, in S6, the step of obtaining the weighted fused local region reference depth feature or the weighted fused local region distorted depth feature is:

[0086] S61, using the multi-layer perception and the Softmax function to generate the weight matrix according to the enhanced local region reference depth feature;

[0087] S62, generating a weighted fusion local region reference depth feature or a weighted fusion local region distorted depth feature by weighted average of the enhanced local region reference depth features or the enhanced local region distorted depth features of different stages according to the weight matrix.

[0088] Further, the channel number of the enhanced local region depth features (including the enhanced local region reference depth features and the enhanced local region distorted depth features) is adjusted to 1 by global average pooling, and then a weight matrix is generated using a multi-layer perception and a Softmax function. The weight matrix is used to generate a weighted fusion local region depth feature (including a weighted fusion local region reference depth feature and a weighted fusion local region distorted depth feature) by weighted average of depth features of different stages. It can be expressed as:

[0089]

[0090] W = Softmax(MLP(G));

[0091]

[0092] In the formula, D is the channel number, MLP is a learnable multi-layer perception, W is the corresponding adaptive weight matrix, and are the corresponding weighted features of the reference image and the distorted image respectively; i and j represent the i-th local region and the j-th stage feature respectively.

[0093] In an embodiment, in S7, the quality score obtaining step includes:

[0094] S71, splicing the initial local region distorted feature map and the weighted fusion local region distorted depth feature or the initial local region reference feature map and the weighted fusion local region reference depth feature in the channel dimension to obtain a distorted image splicing feature or a reference image splicing feature;

[0095] S72, splicing the distorted image splicing feature and the reference image splicing feature in the channel dimension to obtain an overall splicing image feature;

[0096] S73, calculating the quality score based on the overall splicing image feature.

[0097] Further, the initial local region feature map and the weighted fused local region depth feature are spliced in the channel dimension to obtain spliced features corresponding to the reference image and the distorted image; then, the spliced features of the reference image and the distorted image are flattened, and the flattened features are spliced in the channel dimension to obtain a one-dimensional vector, which is input into a multi-layer perception machine. The multi-layer perception machine includes two hidden layers, each having 128 neurons and using a ReLU activation function. Through supervised learning, the quality score between the reference image and the distorted image is further regressed.

[0098] The embodiment also includes a local region mask-based full-reference image quality evaluation system, including:

[0099] An image acquisition module is configured to acquire a distorted image and a reference image.

[0100] A local region division module is configured to extract local region information from the distorted image and the reference image respectively to obtain an initial local region distorted feature map and an initial local region reference feature map.

[0101] A depth feature extraction module is configured to perform multi-stage feature extraction on the distorted image and the reference image respectively to generate an initial distorted depth feature map and an initial reference depth feature map.

[0102] A depth feature processing module is configured to generate a plurality of local region masks based on the initial local region reference feature map, and generate a plurality of local region distorted depth features and a plurality of local region reference depth features from the initial distorted depth feature map and the initial reference depth feature map respectively based on the plurality of local region masks.

[0103] A feature enhancement module is configured to generate an enhanced local region reference depth feature based on a self-attention mechanism by fusing the plurality of local region reference depth features, and generate an enhanced local region distorted depth feature based on the cross-attention mechanism by taking the enhanced local region reference depth feature as a guide feature and according to the plurality of local region distorted depth features.

[0104] A weighted fusion module is configured to generate a weight matrix based on the enhanced local region reference depth feature, and perform weighted aggregation on the enhanced local region reference depth feature and the enhanced local region distorted depth feature respectively by using the weight matrix to generate a weighted fused local region reference depth feature and a weighted fused local region distorted depth feature.

[0105] A quality evaluation module is configured to splice the weighted fused local region reference depth feature, the weighted fused local region distorted depth feature, the initial local region distorted feature map, and the initial local region reference feature map, and calculate a quality score.

[0106] The various embodiments described in this specification are implemented in a progressive manner, each embodiment focusing on the differences from other embodiments, and the same or similar parts between embodiments can be mutually referred to. For the apparatus disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0107] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those of ordinary skill in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A full-reference image quality assessment method based on local area mask, characterized in that: The specific steps are: Obtaining a distorted image and a reference image; Extracting local area information from the distorted image and the reference image respectively to obtain an initial local area distortion feature map and an initial local area reference feature map; Performing multi-stage feature extraction on the distorted image and the reference image respectively and generating an initial distorted depth feature map and an initial reference depth feature map; Generate a plurality of local area masks based on the initial local area reference feature map, and generate a plurality of local area distorted depth features and a plurality of local area reference depth features respectively from the plurality of local area masks and the initial distorted depth feature map and the initial reference depth feature map; Based on the self-attention mechanism, a plurality of the local area reference depth features are fused to generate an enhanced local area reference depth feature; based on the cross-attention mechanism, the enhanced local area reference depth feature is used as a guide feature to generate an enhanced local area distorted depth feature according to the plurality of the local area distorted depth features; Generating a weight matrix according to the enhanced local area reference depth feature, and using the weight matrix to perform weighted aggregation on the enhanced local area reference depth feature and the enhanced local area distorted depth feature to generate a weighted fused local area reference depth feature and a weighted fused local area distorted depth feature; The weighted fused local area reference depth feature, the weighted fused local area distorted depth feature, the initial local area distorted feature map and the initial local area reference feature map are spliced, and a quality score is calculated.

2. The full-reference image quality assessment method based on local area mask according to claim 1, characterized in that: After acquiring the distorted image and the reference image, the distorted image and the reference image need to be preprocessed.

3. The full-reference image quality assessment method based on local area mask according to claim 1, characterized in that: The steps of obtaining the initial local area distortion feature map or the initial local area reference feature map are: The distorted image or the reference image is input into a local area sampling network, and the initial local area distortion feature map or the initial local area reference feature map is generated through multiple iterations.

4. The full-reference image quality assessment method based on local area mask according to claim 1, characterized in that: The steps of obtaining the initial distorted depth feature map or the initial reference depth feature map are as follows: Extracting depth features of the distorted image or the reference image according to a depth extraction model, and selecting the depth features of a plurality of preset stages as a pre-depth feature map; The pre-depth feature map is up-sampled to the size of the distorted image or the reference image to generate the initial distorted depth feature map or the initial reference depth feature map.

5. The full-reference image quality assessment method based on local area mask according to claim 1, characterized in that: The steps of acquiring the local area distorted depth feature or the local area reference depth feature are: Generating the local region masks according to the initial local region reference feature map, wherein the number of the local region masks is the same as the number of local regions; Multiplying the initial distorted depth feature map or the initial reference depth feature map by each of the local area masks respectively to obtain a selected local area distortion feature or a selected local area reference feature; Perform global average pooling on the selected local area distortion feature or the selected local area reference feature to obtain the local area distortion depth feature or the local area reference depth feature.

6. The full-reference image quality assessment method based on local area mask according to claim 4, characterized in that: The steps of acquiring the weighted fusion local area reference depth feature or the weighted fusion local area distortion depth feature are as follows: Generate the weight matrix using a multilayer perceptron and a Softmax function according to the enhanced local area reference depth feature; The enhanced local area reference depth features or the enhanced local area distorted depth features at different stages are weighted averaged according to the weight matrix to generate the weighted fused local area reference depth features or the weighted fused local area distorted depth features.

7. The full-reference image quality assessment method based on local area mask according to claim 1, characterized in that: The quality score acquisition steps are: Splicing the initial local area distortion feature map and the weighted fusion local area distortion depth feature or the initial local area reference feature map and the weighted fusion local area reference depth feature in the channel dimension to obtain a distorted image splicing feature or a reference image splicing feature; Splicing the distorted image stitching features and the reference image stitching features in the channel dimension to obtain an overall stitching image feature; A quality score is calculated based on the overall stitched image features.

8. A full-reference image quality assessment system based on local area mask, characterized in that: include: An image acquisition module, used for acquiring a distorted image and a reference image; A local area division module is used to extract local area information from the distorted image and the reference image respectively to obtain an initial local area distortion feature map and an initial local area reference feature map; A depth feature extraction module, configured to perform multi-stage feature extraction on the distorted image and the reference image respectively and generate an initial distorted depth feature map and an initial reference depth feature map; a depth feature processing module, configured to generate a plurality of local area masks based on the initial local area reference feature map, wherein the plurality of local area masks are respectively combined with the initial distorted depth feature map and the initial reference depth feature map to generate a plurality of local area distorted depth features and a plurality of local area reference depth features; A feature enhancement module is used to fuse multiple local region reference depth features based on the self-attention mechanism to generate enhanced local region reference depth features; Based on the cross attention mechanism, the enhanced local area reference depth feature is used as a guide feature, and the enhanced local area distorted depth feature is generated according to the multiple local area distorted depth features; A weighted fusion module is used to generate a weight matrix according to the enhanced local area reference depth feature, and use the weight matrix to perform weighted aggregation on the enhanced local area reference depth feature and the enhanced local area distorted depth feature to generate a weighted fused local area reference depth feature and a weighted fused local area distorted depth feature; A quality evaluation module is used to splice the weighted fusion local area reference depth feature, the weighted fusion local area distortion depth feature, the initial local area distortion feature map and the initial local area reference feature map, and calculate a quality score.

Images