A blind image quality evaluation method and system based on neighbor searching

Abstract

The application discloses a blind image quality evaluation method and system based on a near neighbor search, and the method comprises the following steps: constructing a semantic classification module and a distortion classification module based on training tasks of semantic classification and distortion classification; given an input image, retrieving an original image with similar content from an IQA database through the semantic classification module; in a series of distorted images corresponding to the retrieved original image, retrieving an image instance with similar distortion through the distortion classification module, and repeating the above retrieval step to obtain multiple image instances; and using a semantic perception distance to aggregate subjective quality scores of the retrieved image instances into a predicted quality score of the final input image, which is used for evaluating the quality of the input image. The application does not involve any regression process for blind image quality evaluation, helps to solve the biased estimation problem of a regression-based blind image quality evaluation model for biased training samples, and has important significance for evaluating the perceived quality of an image.

Description

Technical Field

[0001] This invention belongs to the field of blind image perception quality evaluation, specifically involving a blind image quality evaluation method and system based on nearest neighbor search. Background Technology

[0002] Image Quality Assessment (IQA) aims to enable computers to judge the visual quality of images like humans do, and it is a key component in various visual tasks such as image restoration, image compression, and super-resolution. Because Blind Image Quality Assessment (BIQA) does not require access to any reference information when evaluating image quality, it has gained increasing popularity among researchers in recent years.

[0003] In developing the BIQA model, the collected raw images were downgraded using different types and intensities of distortion to synthesize a series of distorted images. Each distorted image was then evaluated by numerous human subjects and labeled with a Mean Opinion Score (MOS). Finally, a regression-based BIQA model was obtained by fitting the predicted quality scores and MOS values ​​of the training images.

[0004] Regression-based BIQA models typically consist of two parts: quality-perceived feature extraction and image quality regression. In traditional BIQA models, researchers extract handcrafted features as quality-perceived representations, such as locally normalized brightness coefficients, gradient statistics, log Gabor responses, and color. These features are then regressed to a subjective score using nonlinear functions, such as SVR (Support Vector Regression) and MVG (Multivariate Gaussian) models. Due to the limitations of handcrafted features' quality-perceived capabilities and the learning power of nonlinear mapping functions, more and more deep learning-based BIQA models have been proposed in recent years. In deep learning-based BIQA models, CNNs (Convolutional Neural Networks) are commonly used to extract quality-perceived features, and then image quality scores are obtained through quality regressors, such as MLPs (Multi-layer Perceptrons), Hypernetworks, and Visual Transformers. However, regression-based BIQA models are easily affected by the representativeness and frequency of the training samples. Specifically, the limited size of the IQA database can easily lead to biased training samples, thus affecting the accuracy of modeling the true distribution of image quality. Furthermore, regression-based models derive their loss by minimizing the average prediction error across all training samples, meaning the loss is primarily influenced by frequently occurring samples. These factors can easily lead to biased estimations of model parameters, resulting in inaccurate predictions. To mitigate this problem, the focus should be on avoiding over-reliance on specific model parameters. One possible solution is to evaluate image quality by retrieving similar instances in the feature space.

[0005] Ensuring the validity of the feature space is crucial for obtaining sufficiently similar instances. However, this can be difficult to achieve when feature representations are learned directly from MOS values. Specifically, the principle of visual occlusion emphasizes the impact of image content and distortion on HVS (Human Visual System) perceptual function. However, MOS values ​​primarily reflect the level of quality degradation without providing specific information about the type of content or distortion. Even image pairs with the same content or distortion can exhibit different image qualities. Conversely, image pairs with different content or distortion may still have comparable image quality. Therefore, BIQA models derived from regressing MOS values ​​may fail to effectively model content and distortion representations, thus affecting their ability to simulate HVS perceptual function. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a blind image quality assessment method and system based on nearest neighbor search. Based on the SC (Semantic-based Classification) module and the DC (Distortion-based Classification) module, it retrieves image instances similar to the input image from an manually annotated IQA database, and obtains the predicted quality score by aggregating the MOS values ​​of these similar image instances. This effectively solves the problem of biased parameter estimation in regression models caused by biased training samples, improves the sensitivity of regression models to biased training samples, ensures the effectiveness of the feature space, and obtains sufficiently similar image instances.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] This invention proposes a blind image quality assessment method based on nearest neighbor search, comprising the following steps:

[0009] S1. Construct semantic classification module and distortion classification module respectively based on the training tasks of semantic classification and distortion classification.

[0010] S2. Given an input image, use the semantic classification module to retrieve original images with similar content from the IQA database.

[0011] S3. Using the distortion classification module, retrieve a series of distorted images corresponding to the original image and image instances with similar distortion to the input image.

[0012] S4. To mitigate prediction bias, the semantic classification module and the distortion classification module are used to retrieve multiple image instances similar to the input image.

[0013] S5. Using semantic-aware distance, average pooling is performed on the subjective quality scores of the image instances in step S4 to obtain the final predicted quality score of the input image.

[0014] Furthermore, in step S1, the specific content of constructing the semantic classification module and the distortion classification module is as follows:

[0015] (1) The specific content of constructing the semantic classification module is: remove the last classification layer of the VGG16 network pre-trained on the ImageNet dataset.

[0016] (2) The specific content of constructing the distortion classification module is as follows: BoTNet is a hybrid design based on spatial convolution and multi-head self-attention (MHSA), which can be used as a distortion classification model. In the distortion classification network, BoTNet is used to replace the convolutional layers in the last three bottleneck blocks of ResNet with MHSA, while other network layers remain unchanged. Specifically, given a training set containing N samples, represented as *(x n ,y n )|1≤n≤N+, where x n and y n Let represent the labels of the nth input image and its true distortion category, respectively. The model parameters θ are optimized by minimizing the cross-entropy loss function, specifically as follows:

[0017]

[0018] Where C represents the number of distortion categories, This represents the predicted probability that the nth image belongs to category C. This represents the value of θ that minimizes the following expression. This represents the true probability that the nth image belongs to category C. To avoid distorted category bias, the final classification layer of the trained model is removed, and the resulting model is the distorted classification module used to extract distorted features.

[0019] Furthermore, in step S2, according to the sampling protocol for developing the IQA model, the IQA database is randomly divided into a training set and a test set, with the training set accounting for 80% and the test set accounting for 20%. The training set consists of P original images, denoted as S = {(X...} p D p M p Let X be a subset of the vectors X and P, where X ≤ 1 ≤ p ≤ P. p This is the p-th original image. It corresponds to X p Distorted images, D represents p The l-th sample, M p This represents the subjective quality score of the distorted image. The specific content extracted from the semantic classification module to retrieve the original image with similar content to the input image is as follows: S201, based on the given input image x... t and the original image X p Calculate x t and X p The content similarity between them is called semantic perceptual distance. The formula is:

[0020]

[0021] Among them, vt and v p They represent x respectively t and X p The semantic vector, |v t |、|v p | represent v respectively t v p The modulus is T, where T represents transpose.

[0022] S202, and x t The K most similar original images were retrieved in the following way:

[0023]

[0024] in, This represents a function that sorts semantic-aware distances in descending order and selects the top K images, where 1 ≤ K ≤ P.

[0025] Furthermore, in step S3, the specific content of retrieving image instances with similar distortion to the input image is as follows:

[0026] S301, Based on the given input image x t and distorted images Calculate x t and The distorted similarity between them is called the distorted perception distance. The formula is:

[0027]

[0028] Among them, wt and They represent x respectively t and The distorted feature vector.

[0029] S302, Based on the original image retrieved from the semantic classification module Obtain the distorted image set D p From D p Retrieved from x t The formula for the distorted image with the highest distortion similarity is:

[0030]

[0031] The resulting distorted image In terms of content and distortion, it is similar to x t Instances with similar perceptions.

[0032] Furthermore, in step S5, the specific content of the final predicted quality score of the input image is as follows:

[0033] Using semantically sensed distance As a weighting factor for each distorted image The MOS values ​​are averaged to give instances with higher semantic similarity a greater influence on the final prediction. The quality score of the prediction can be expressed as:

[0034]

[0035] in, Representing an image The true MOS value.

[0036] Furthermore, this invention also proposes a blind image quality assessment system based on nearest neighbor search, comprising:

[0037] A classification module training system is used to construct semantic classification modules and distortion classification modules.

[0038] The semantic classification module system is used to retrieve original images from the IQA database that have similar content to the input image.

[0039] The distortion classification module system is used to retrieve image instances with similar distortion to the input image from a series of distorted images corresponding to the retrieved original image.

[0040] The prediction quality score calculation system is used to calculate the subjective quality score of an image instance using a semantically aware distance or simple averaging strategy, and then perform average pooling on the calculated value to obtain the final predicted quality score of the input image.

[0041] Furthermore, the specific details of constructing the semantic classification module and the distortion classification module in the classification module training system are as follows:

[0042] (1) The specific content of constructing the semantic classification module is: remove the last classification layer of the VGG16 network pre-trained on the ImageNet dataset.

[0043] (2) The specific content of constructing the distortion classification module is as follows: BoTNet is a hybrid design based on spatial convolution and multi-head self-attention (MHSA), which can be used as a distortion classification model. In the distortion classification network, BoTNet is used to replace the convolutional layers in the last three bottleneck blocks of ResNet with MHSA, while other network layers remain unchanged. Specifically, given a training set {(x) containing N samples... n y n )|1≤n≤N}, where x n and y n Let represent the labels of the nth input image and its true distortion category, respectively. The model parameters θ are optimized by minimizing the cross-entropy loss function, specifically as follows:

[0044]

[0045] Where C represents the number of distortion categories, This represents the predicted probability that the nth image belongs to category C. This represents the value of θ that minimizes the following expression. This represents the true probability that the nth image belongs to class C. To avoid distorted class bias, the final classification layer of the trained model is removed, and the resulting model is used as a distortion classification module to extract distortion features.

[0046] Furthermore, in the semantic classification module system, the specific content retrieved from the original image that has similar content to the input image is as follows:

[0047] Step 1: Based on the given input image x t and the original image X p Calculate x t and X p The content similarity between them is called semantic perceptual distance. The formula is:

[0048]

[0049] Among them, v t and v p They represent x respectively t and X p The semantic vector, |v t |、|v p | represent v respectively t v p The modulus is T, where T represents transpose.

[0050] Step 2, with x t The K most similar original images were retrieved in the following way:

[0051]

[0052] in, This represents a function that sorts semantic-aware distances in descending order and selects the top K images, where 1 ≤ K ≤ P.

[0053] Furthermore, in the distortion classification module system, the specific content of retrieving image instances with similar distortion to the input image is as follows:

[0054] Step 1: Based on the given input image x t and distorted images Calculate x t and The distorted similarity between them is called the distorted perception distance. The formula is:

[0055]

[0056] Among them, wt and They represent x respectively t and The distorted feature vector.

[0057] Step 2: Based on the original images retrieved from the semantic classification module Obtain the distorted image set D p From D p Retrieved from x t The formula for the distorted image with the highest distortion similarity is:

[0058]

[0059] The resulting distorted image In terms of content and distortion, it is similar to x t Instances with similar perceptions.

[0060] Furthermore, in the prediction quality score calculation system, the specific content of the final predicted quality score of the input image is as follows: using semantically perceptual distance... As a weighting factor for each distorted image The MOS values ​​are averaged to give instances with higher semantic similarity a greater influence on the final prediction. The quality score of the prediction can be expressed as:

[0061]

[0062] in, Representing an image The true MOS value.

[0063] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0064] The method proposed in this invention establishes local relationships between adjacent instances in the feature space without relying on the entire training set. This allows the proposed model to avoid over-reliance on specific model parameters, effectively reducing prediction bias caused by biased training sets, improving the modeling accuracy of the true distribution of image quality, and alleviating the model's sensitivity to biased training samples.

[0065] Furthermore, this invention uses a semantic classification module and a distortion classification module to repeatedly retrieve image instances similar to the input image from a manually annotated IQA database, ensuring the effectiveness of the feature space to obtain instances that are as similar as possible. Attached Figure Description

[0066] Figure 1This is a schematic diagram of the blind image quality evaluation method based on nearest neighbor search according to an embodiment of the present invention. Detailed Implementation

[0067] The technical solution of the present invention will now be clearly and completely described in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0068] To achieve the above objectives, this invention proposes a blind image quality assessment method based on nearest neighbor search, such as... Figure 1 As shown, the specific steps are as follows:

[0069] S1. Randomly select 40,000 original images from the 140,000 high-resolution images published in the KADID-10K database. Use distortion synthesis code to generate a large number of distorted images to train the distortion classification network and increase the robustness of the model. Then, use data from the IQA dataset and KADID-10K to fine-tune and construct the semantic classification module and the distortion classification module respectively. The specific content is as follows:

[0070] (1) The specific content of constructing the semantic classification module is: remove the last classification layer of the VGG16 network pre-trained on the ImageNet dataset.

[0071] (2) The specific content of constructing the distortion classification module is as follows: BoTNet is a hybrid design based on spatial convolution and MHSA, which can be used as a distortion classification model. In the distortion classification network, BoTNet is used to replace the convolutional layers in the last three bottleneck blocks of ResNet with MHSA, while other network layers remain unchanged. Specifically, given a training set {(x) containing 16 samples... n y n Let x be the sum of the numbers x and n, where x ≤ 16. n and y n Let represent the labels of the nth input image and its true distortion category, respectively. The training objective is to minimize the cross-entropy loss function by optimizing the model parameters θ, specifically:

[0072]

[0073] Where C is the number of distortion categories. This represents the predicted probability that the nth image belongs to category C. This represents the value of g that minimizes the following expression. This represents the true probability that the nth image belongs to class C. To avoid distorted class bias, the final classification layer of the trained model is removed, and the resulting model is used as a distortion classification module to extract distortion features.

[0074] S2. Given an input image, retrieve original images with similar content from the KADID-10K database using the semantic classification module. According to the sampling protocol used in developing the IQA model, the KADID-10K database is randomly divided into a training set and a test set, with the training set accounting for 80% and the test set accounting for 20%. The training set consists of 81 original images, denoted as S={(X p D p M p )|1≤p≤81}, where X p This is the p-th original image. It corresponds to X p Distorted images, D represents p The l-th sample, M p This represents the subjective quality score of these distorted images.

[0075] The specific content extracted from the semantic classification module that is similar to the input image is as follows:

[0076] S201, Based on the given input image x t and the original image X p Calculate x t and X p The content similarity between them is called semantic perceptual distance. The formula is:

[0077]

[0078] Among them, v t and v p They represent x respectively t and X p The semantic vector, |v t |、|v p | represent v respectively t v p The modulus is T, where T represents transpose.

[0079] S202, and x t The K most similar original images were retrieved in the following way:

[0080]

[0081] in, This represents a function that sorts semantic-aware distances in descending order and selects the top K images, where 1 ≤ K ≤ P.

[0082] S3. A total of 10,125 distorted images correspond to the retrieved original image. The specific content of retrieving image instances with similar distortion to the input image through the distortion classification module within this series of distorted images is as follows:

[0083] S301, Based on the given input image x t and distorted images Calculate x t and The distorted similarity between them is called the distorted perception distance. The formula is:

[0084]

[0085] Among them, wt and They represent x respectively t and The distorted feature vector.

[0086] S302. Extract the retrieved original image based on the semantic classification module. Obtain the distorted image set D p From D p Retrieved from x t The formula for the distorted image with the highest distortion similarity is:

[0087]

[0088] The resulting distorted image In terms of content and distortion, it is similar to x t Instances with similar perceptions.

[0089] S4. To mitigate prediction bias, the semantic classification module and the distortion classification module are used to retrieve multiple image instances similar to the input image.

[0090] S5. Using semantic-aware distance, the subjective quality scores of the image instances in step S4 are averaged and pooled to obtain the final predicted quality score of the input image. The specific details are as follows:

[0091] Using semantically sensed distance As a weighting factor for each distorted image The MOS scores are averaged to give instances with higher semantic similarity a greater influence on the final prediction. The prediction quality score is:

[0092]

[0093] in, Representing an image The true MOS value.

[0094] This invention also proposes a blind image quality assessment system based on nearest neighbor search, comprising a classification module training system, a semantic classification module system, a distortion classification module system, a predicted quality score calculation system, and a computer program that can run on a processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention embodiment, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of this invention.

[0095] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A blind image quality assessment method based on nearest neighbor search, characterized in that, Includes the following steps: S1. Construct semantic classification and distortion classification modules based on the training tasks of semantic classification and distortion classification, respectively; specifically: (1) The specific content of constructing the semantic classification module: Remove the last classification layer of the VGG16 network pre-trained on the ImageNet dataset; (2) Specific details of constructing the distortion classification module: In the distortion classification network, BoTNet is used to replace the convolutional layers in the last three bottleneck blocks of ResNet with multi-head self-attention modules, while other network layers remain unchanged; specifically, given a training set {(x) containing N samples... n ,y n )|1≤n≤N}, where x n and y n Let represent the labels of the nth input image and its true distortion category, respectively; the model parameters θ are optimized by minimizing the cross-entropy loss function, specifically as follows: Where C represents the number of distortion categories, This represents the predicted probability that the nth image belongs to category C. This represents the value of θ that minimizes the following expression. This represents the true probability that the nth image belongs to category C; by removing the final classification layer of the training model, the resulting model is the distortion classification module; S2. Randomly divide the IQA database into a training set and a test set, with the training set accounting for 80% and the test set accounting for 20%. The training set consists of P original images, denoted as S = {(X...} p D p M p Let X be a subset of the vectors X and P, where X ≤ 1 ≤ p ≤ P. p This is the p-th original image. It corresponds to X p Distorted images, D represents p The l-th sample, M p This represents the subjective quality score of the distorted image; given an input image, the semantic classification module retrieves original images with similar content from the IQA database; specifically: S201, Based on the given input image x t and the original image X p Calculate x t and X p The content similarity between them is called semantic perceptual distance. The formula is: Among them, v t and v p They represent x respectively t and X p The semantic vector, |v t |、|v p | represent v respectively t v p The modulus length, T represents transpose; S202, and x t The K most similar original images were retrieved in the following way: in, This represents a function that sorts semantic-aware distances in descending order and selects the top K images, where 1≤K≤P; S3. Using the distortion classification module, retrieve a series of distorted images corresponding to the original image and image instances with similar distortion to the input image; S4. Use the semantic classification module and the distortion classification module to retrieve multiple image instances similar to the input image; S5. Using semantic-aware distance, average pooling is performed on the subjective quality scores of the image instances in step S4 to obtain the final predicted quality score of the input image.

2. The blind image quality assessment method based on nearest neighbor search according to claim 1, characterized in that, In step S3, the specific content of retrieving image instances with similar distortion to the input image is as follows: S301, Based on the given input image x t and distorted images Calculate x t and The distorted similarity between them is called the distorted perception distance. The formula is: Among them, w t and They represent x respectively t and Distorted feature vectors; S302, Based on the original image retrieved from the semantic classification module Obtain the distorted image set D p From D p Retrieved from x t The formula for the distorted image with the highest distortion similarity is: The resulting distorted image In terms of content and distortion, it is similar to x t Instances with similar perceptions.

3. The blind image quality assessment method based on nearest neighbor search according to claim 1, characterized in that, In step S5, the specific details of the final predicted quality score of the input image are as follows: Using semantically sensed distance As a weighting factor for each distorted image The MOS values ​​are averaged, and the predicted quality score is expressed as: in, Representing an image The true MOS value.

4. A blind image quality assessment system based on nearest neighbor search, characterized in that, include: The classification module training system is used to construct semantic classification modules and distortion classification modules; specifically: (1) The specific content of constructing the semantic classification module is: remove the last classification layer of the VGG16 network pre-trained on the ImageNet dataset; (2) In the distortion classification network, BoTNet is used to replace the convolutional layers in the last three bottleneck blocks of ResNet with multi-head self-attention modules, while other network layers remain unchanged; specifically, given a training set {(x) containing N samples... n ,y n )|1≤n≤N}, where x n and y n Let represent the labels of the nth input image and its true distortion category, respectively; the model parameters θ are optimized by minimizing the cross-entropy loss function, specifically as follows: Where C represents the number of distortion categories, This represents the predicted probability that the nth image belongs to category C. This represents the value of θ that minimizes the following expression. This represents the true probability that the nth image belongs to category C; by removing the final classification layer of the training model, the resulting model is the distortion classification module; The semantic classification module system is used to retrieve original images from the IQA database that have similar content to the input image; specifically: Step 1: Based on the given input image x t and the original image X p Calculate x t and X p The content similarity between them is called semantic perceptual distance. The formula is: Among them, v t and v p They represent x respectively t and X p The semantic vector, |v t |、|v p | represent v respectively t v p The modulus length, T represents transpose; Step 2, with x t The K most similar original images were retrieved in the following way: in, This represents a function that sorts semantic-aware distances in descending order and selects the top K images, where 1≤K≤P; The distortion classification module system is used to retrieve image instances with similar distortion to the input image from a series of distorted images corresponding to the retrieved original image; The prediction quality score calculation system is used to calculate the subjective quality score of an image instance using a semantically aware distance or simple averaging strategy, and then perform average pooling on the calculated value to obtain the final predicted quality score of the input image.

5. The blind image quality assessment system based on nearest neighbor search according to claim 4, characterized in that, In the distortion classification module system, the specific content for retrieving image instances with similar distortion to the input image is as follows: Step 1: Based on the given input image x t and distorted images Calculate x t and The distorted similarity between them is called the distorted perception distance. The formula is: Among them, w t and They represent x respectively t and Distorted feature vectors; Step 2: Based on the original images retrieved from the semantic classification module Obtain the distorted image set D p From D p Retrieved from x t The formula for the distorted image with the highest distortion similarity is: The resulting distorted image In terms of content and distortion, it is similar to x t Instances with similar perceptions.

6. The blind image quality assessment system based on nearest neighbor search according to claim 4, characterized in that, In the prediction quality score calculation system, the specific content of the final predicted quality score of the input image is as follows: using semantically perceptual distance. As a weighting factor for each distorted image The MOS values ​​are averaged, and the predicted quality score is expressed as: in, Representing an image The true MOS value.

Images