Image quality evaluation method and system based on multi-vision task fusion

Abstract

The invention provides an image quality evaluation method and system based on multi-vision task fusion, and relates to the technical field of image processing. According to the method, the fitness of different images to a specific visual task is quantified into corresponding evaluation index values, and subjective scores of a user are replaced with the evaluation index values so as to guide training of an image quality prediction model. Specifically, the method comprises the following steps: firstly, constructing a multi-task-oriented image set used for image quality evaluation, and based on the influence of each image in the image set on the performance of different visual tasks (target detection, semantic segmentation and attitude estimation), quantifying to obtain evaluation index values of the different visual tasks on each image in the image set; the method comprises the following steps: firstly, acquiring an image set, fusing to obtain a fusion evaluation index value of each image in the image set, and then training based on the images carrying the fusion evaluation index values to obtain an end-to-end image quality prediction model which can be used for image quality evaluation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a method and system for image quality assessment based on multi-vision task fusion. Background Art

[0002] Surveillance cameras are essential key equipment in the security field and are widely used in important tasks such as criminal investigation, traffic accident identification, and natural disaster site warning. However, the surveillance images obtained by surveillance cameras are often affected by factors such as weather and light, resulting in various distortion problems and seriously reducing image quality. Such degraded surveillance images will directly affect the normal operation of security services. Therefore, accurately evaluating and predicting image quality is particularly important in the security field.

[0003] Currently, image quality assessment methods are divided into two categories: subjective and objective. Traditional subjective assessment methods rely on the mean opinion score (MOS), which is based on direct human assessment of image quality. However, due to the subjective preferences of users, this method often leads to large variance and low accuracy in prediction results. With the development of neuroscience, electroencephalography (EEG) has been gradually applied to the fields of communication and signal processing. Some researchers have tried to use EEG to assist in image quality assessment. Although this method can better reflect the user's subjective perceptual experience, it requires the simultaneous collection of EEG data when the user views the image, which usually results in higher manpower and time costs.

[0004] Objective image quality assessment methods are mainly divided into two categories: full reference (FR-IQA) and no reference (NR-IQA). FR-IQA methods rely on reference images. Specifically, the method evaluates the quality of distorted images by comparing the difference between the distorted image and the ideal reference image (i.e., the original undistorted image). NR-IQA methods do not rely on reference images and are generally divided into two main research directions. One is image quality assessment for specific types of distortion, such as blur, block effect, and ringing effect; the other is image quality assessment for non-specific types, i.e., generalized distortion assessment, which comprehensively evaluates various types of distortion. When training quality assessment models, objective image quality assessment methods usually rely on users' subjective ratings of images as labels for model training to simulate the perception process of the human visual system. However, in practical applications, obtaining these subjective rating labels is often time-consuming and costly. Therefore, there is an urgent need to study an objective image quality assessment method that does not rely on human subjective ratings. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for image quality assessment based on multi-vision task fusion.

[0006] A first aspect of an embodiment of the present invention provides a method for image quality assessment based on multi-vision task fusion, the method comprising: Degrading the original distortion-free image to multiple different quality levels to obtain an image set for image quality assessment; Inputting an image set for image quality assessment into a target detection model, a semantic segmentation model, and a posture estimation model to obtain evaluation index values ​​corresponding to multiple visual tasks, wherein the multiple visual tasks include: target detection, semantic segmentation, and posture estimation; fusing the evaluation index values ​​corresponding to the multiple visual tasks to obtain a fused evaluation index value corresponding to the image set for image quality assessment; Taking the fusion evaluation index value corresponding to the image set for image quality evaluation as the true value label, based on the image set for image quality evaluation, the end-to-end image quality prediction model to be trained is trained to obtain a trained end-to-end image quality prediction model, wherein the trained end-to-end image quality prediction model is used to simulate the process of machine perception of images to perform image quality evaluation; The image to be evaluated is input into the trained end-to-end image quality prediction model to obtain an image quality evaluation result.

[0007] Optionally, the original distortion-free image is degraded to multiple different quality levels to obtain an image set for image quality assessment, including: Taking clarity, noise and brightness as degradation dimensions, the original undistorted image is subjected to four different degrees of quality degradation to obtain an image set for image quality assessment.

[0008] Optionally, the image set for image quality assessment is input into the target detection model, the semantic segmentation model, and the pose estimation model to obtain evaluation index values ​​corresponding to each of the multiple visual tasks, and the evaluation index values ​​corresponding to each of the multiple visual tasks are fused to obtain a fused evaluation index value corresponding to the image set for image quality assessment, including: Inputting an image set for image quality assessment into a target detection model, using a first evaluation index for target detection and a second evaluation index for target detection, and using the target detection model to obtain a first evaluation index value and a second evaluation index value of the image set for image quality assessment, wherein the first evaluation index for target detection is an average bounding box precision, and the second evaluation index for target detection is an average bounding box precision when IoU=0.5; Inputting an image set for image quality assessment into a semantic segmentation model, using a first evaluation index for semantic segmentation and a second evaluation index for semantic segmentation, and using the semantic segmentation model to obtain a third evaluation index value and a fourth evaluation index value of the image set for image quality assessment, wherein the first evaluation index for semantic segmentation is average precision, and the second evaluation index for semantic segmentation is average IoU; Inputting the image set for image quality assessment into the pose estimation model, using a first evaluation index for pose estimation and a second evaluation index for pose estimation, and using the pose estimation model to obtain a fifth evaluation index value and a sixth evaluation index value of the image set for image quality assessment, wherein the first evaluation index for pose estimation is the average accuracy, and the second evaluation index for pose estimation is the average accuracy of the bounding box when OKS=0.5; The first to sixth evaluation index values ​​of the image set used for image quality evaluation are fused to obtain a fused evaluation index value corresponding to the image set used for image quality evaluation.

[0009] Optionally, the end-to-end image quality prediction model to be trained includes: a feature representation module, a local feature extraction module, a global feature extraction module, a feature fusion module and a fully connected module connected in sequence; Taking the fusion evaluation index value corresponding to the image set for image quality evaluation as the true value label, the end-to-end image quality prediction model to be trained is trained based on the image set for image quality evaluation, including: The image set used for image quality assessment is input into the feature representation module to obtain a feature vector; Extract features from feature vectors through a local feature extraction module to obtain local features; The feature vector is extracted through the global feature extraction module to obtain the global feature; The local features and global features are fused through the fusion module to obtain fused features; The fused features are processed through a fully connected module to obtain the image quality prediction value; Based on the true value label and the image quality prediction value, the parameters of the end-to-end image quality prediction model to be trained are updated to obtain a trained end-to-end image quality prediction model.

[0010] Optionally, the global feature extraction module is divided into four stages of global feature extraction sub-modules, and each stage of the global feature extraction sub-module is composed of a different number of Swin Transformer blocks; The local feature extraction module is divided into four stages of local feature extraction submodules, and each stage of the local feature extraction submodule consists of a different number of ConvNeXt blocks; The number of the fusion modules is 4. At the end of each feature extraction stage, the feature fusion module fuses the global features and local features of that stage to serve as the input of the next stage.

[0011] Optionally, the local features and the global features are fused through a fusion module to obtain fused features, including: The similarity between global features and local features is calculated through the fusion module; Perform 1×1 convolution operation on the global features to generate attention weights; The global and local features are weighted and fused to obtain the fused features.

[0012] A second aspect of an embodiment of the present invention provides a system for image quality assessment based on multi-vision task fusion, the system comprising: An image generation subsystem, used for degrading the original distortion-free image to a plurality of different quality levels, to obtain an image set for image quality assessment; An index value evaluation subsystem is used to input an image set for image quality evaluation into a target detection model, a semantic segmentation model, and a posture estimation model to obtain evaluation index values ​​corresponding to multiple visual tasks, wherein the multiple visual tasks include: target detection, semantic segmentation, and posture estimation; and to fuse the evaluation index values ​​corresponding to the multiple visual tasks to obtain a fused evaluation index value corresponding to the image set for image quality evaluation; a training subsystem, configured to train an end-to-end image quality prediction model to be trained based on the image set for image quality assessment, using the fusion evaluation index value corresponding to the image set for image quality assessment as a true value label, to obtain a trained end-to-end image quality prediction model, wherein the trained end-to-end image quality prediction model is used to simulate a process in which a machine perceives an image to perform image quality assessment; The application subsystem is used to input the image to be evaluated into the trained end-to-end image quality prediction model to obtain an image quality evaluation result.

[0013] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when executed, the processor implements the method for image quality assessment based on multi-visual task fusion as described in the first aspect of the present invention.

[0014] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for image quality assessment based on multi-visual task fusion as described in the first aspect of the present invention.

[0015] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which is used by a processor to implement the steps in the method for image quality assessment based on multi-visual task fusion as described in the first aspect of the present invention.

[0016] In an embodiment of the present invention, the adaptability of different images to specific visual tasks is quantified into corresponding evaluation index values, and these evaluation index values ​​are used to replace the user's subjective ratings to guide the training of the image quality prediction model. Specifically, in an embodiment of the present invention, an image set for image quality assessment for multiple visual tasks is first constructed, and based on the impact of each image in the image set on the performance of different visual tasks (target detection, semantic segmentation, and pose estimation), the evaluation index values ​​of different visual tasks on each image in the image set are quantified, and fused to obtain the fusion evaluation index values ​​of each image in the image set. Subsequently, an end-to-end image quality prediction model is trained based on the images carrying the fusion evaluation index values, and the end-to-end image quality prediction model can be used for image quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0018] Figure 1 is a flowchart of a method for image quality assessment based on multi-visual task fusion provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of an original undistorted image and a degraded image in the method for image quality assessment based on multi-visual task fusion provided by an embodiment of the present invention; Figure 3 It is a schematic diagram of a fusion evaluation index value generation process in a method for image quality evaluation based on multi-vision task fusion provided in an embodiment of the present invention; Figure 4 is an example diagram of fusion evaluation index values ​​of different images in the method for image quality evaluation based on multi-visual task fusion provided by an embodiment of the present invention; Figure 5 It is a schematic diagram of the structure of an end-to-end image quality prediction model in the method for image quality assessment based on multi-visual task fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Traditional subjective evaluation methods, such as Mean Opinion Score (MOS Score), allow users to directly give corresponding quality scores to different images, but this process is easily affected by external factors such as user preferences, emotional state, and the evaluation environment, resulting in large variance and low precision in the evaluation results. In recent years, with the development of psychophysiology, more and more researchers have applied EEG to the fields of communication and signal processing. In related technologies, the impact of image quality degradation on user EEG signals has been studied, and the steady-state visual evoked potential (SSVEP) and event-related potential (ERP) methods have been used to capture the relationship between image, video quality and EEG. This method requires obtaining the user's EEG while watching the media material in the preliminary stage for subsequent analysis. However, the process of obtaining EEG requires a lot of time and labor costs.

[0021] According to the availability of reference images, objective image quality assessment methods can be divided into two categories: full-reference (FR-IQA) and no-reference (NR-IQA).

[0022] The full reference image quality assessment method (FR-IQA) requires the original reference image and the distorted image. Traditional FR-IQA methods mainly evaluate image quality by calculating the peak signal-to-noise ratio (PSNR) or structural similarity index (SSIM) between the reference image and the distorted image. However, PSNR only considers pixel-level errors, and although SSIM considers brightness, contrast, and structural information, it may still ignore some more complex image features and distortion types, making the evaluation results one-sided. With the proposal and development of convolutional neural networks (CNNs), more and more researchers have begun to apply CNNs to FR-IQA. CNN-based FR-IQA methods usually accept a pair of images as input: one is a reference image (a high-quality image without distortion) and the other is a distorted image to be evaluated. CNN automatically extracts multi-level features from these two images. By calculating the feature differences between the two images, the network can capture the quality difference between the distorted image and the reference image.

[0023] Considering that it is often difficult to obtain reference images in actual application scenarios, researchers have begun to explore non-reference image quality assessment (NR-IQA). This approach usually relies on large-scale image datasets with subjective ratings, and automatically extracts multi-level features from the input image through CNN, from low-level texture and edge information to high-level semantic information. Ultimately, these features are mapped to quality scores through a regression network. Since convolutional neural networks can only capture the sensory field of local images, they cannot obtain the global features of the image. One of the natural advantages of Transformer is to capture the global features of the target object. With the success of Visual Transformer and Swin Transformer in visual tasks, some researchers have begun to try to replace CNN with Visual Transformer or Swin Transformer, or combine the two for NR-IQA applications. Objective image quality assessment methods usually rely on datasets with subjective ratings. However, in practical application scenarios, obtaining subjective ratings of in-the-wild image datasets is often time-consuming and laborious. Currently, with the gradual development of machine perception, image quality assessment has begun to shift from the "human perspective" to the "machine perspective". Image quality assessment methods related to machine perception are usually oriented towards specific task performance.

[0024] In an embodiment of the present invention, a task-oriented image quality assessment study is conducted. Specifically, the adaptability of different images to specific visual tasks is quantified into corresponding evaluation index values, and these evaluation index values ​​are used to replace the user's subjective ratings to guide the training of the image quality prediction model. Specifically, in an embodiment of the present invention, an image set for image quality assessment for multiple visual tasks is first constructed, and based on the impact of each image in the image set on the performance of different visual tasks (target detection, semantic segmentation, and pose estimation), the evaluation index values ​​of different visual tasks on each image in the image set are quantified, and the fusion evaluation index values ​​of each image in the image set are fused. Subsequently, an end-to-end image quality prediction model is trained based on the images carrying the fusion evaluation index values, and the end-to-end image quality prediction model can be used for image quality assessment.

[0025] Specifically, Figure 1 As shown, Figure 1 : is a flow chart of a method for image quality assessment based on multi-visual task fusion provided by an embodiment of the present invention. The method for image quality assessment based on multi-visual task fusion provided by an embodiment of the present invention comprises the following steps: S1, the original undistorted image is degraded to multiple different quality levels to obtain an image set for image quality assessment.

[0026] Specifically, in an embodiment of the present invention, an image data set for image quality assessment is constructed. Specifically, 600 images containing humans from the CoCo2017 training set are selected as the training set, 60 images containing humans from the CoCo2017 validation set are selected as the validation set, and 60 images containing humans from the CoCo2017 test set are selected as the test set. Then, these images are degraded to four different quality levels in three dimensions of clarity, noise and brightness, respectively, to obtain an image quality assessment data set for image quality assessment. Finally, a training set, validation set and test data set for image quality assessment are constructed based on the image quality assessment data set.

[0027] In the embodiment of the present invention, images containing humans are selected based on the requirements of downstream visual tasks.

[0028] S2, inputting the image set used for image quality assessment into the target detection model, the semantic segmentation model and the posture estimation model to obtain evaluation index values ​​corresponding to each of the multiple visual tasks, wherein the multiple visual tasks include: target detection, semantic segmentation and posture estimation; fusing the evaluation index values ​​corresponding to each of the multiple visual tasks to obtain a fused evaluation index value corresponding to the image set used for image quality assessment.

[0029] In the embodiment of the present invention, a fusion evaluation index value is obtained based on the above data set as an image quality scoring index. Specifically, based on the above image quality assessment data set, object detection, semantic segmentation, and pose estimation (based on humans) are selected as specific downstream visual tasks. For each of these three visual tasks, the most advanced method is selected: Co-DETR is used for object detection, ONE-PEACE is used for semantic segmentation, and 4×RSN-50 is used for pose estimation, and corresponding quality scoring indicators are generated for all images.

[0030] S3, taking the fusion evaluation index value corresponding to the image set for image quality evaluation as the true value label, based on the image set for image quality evaluation, training the end-to-end image quality prediction model to be trained to obtain a trained end-to-end image quality prediction model, wherein the trained end-to-end image quality prediction model is used to simulate the process of machine perception of images for image quality evaluation.

[0031] S4, inputting the image to be evaluated into the trained end-to-end image quality prediction model to obtain an image quality evaluation result.

[0032] In the embodiment of the present invention, an end-to-end image quality prediction model is constructed. The model aims to achieve a direct mapping from an image to a quality score. The model first uses a Swin Transformer module and a CNN module to extract global and local features of the image, then generates a final feature map of the image through a fusion network, and finally maps these features to a quality score through a regression model.

[0033] Specifically, Figure 1 As shown, the first row shows the construction of the image quality assessment dataset and the generation of the fusion evaluation index value. During the dataset construction process, each image is degraded in three dimensions: clarity, brightness, and noise, and each dimension contains four degradation levels to construct an image dataset for quality assessment. Next, the embodiment of the present invention selects target detection, semantic segmentation, and pose estimation as downstream tasks, and obtains the fusion evaluation index value of each image by measuring the fitness of different quality images in different tasks and fusing these indicators. In this way, the first row completes the process from image to machine perception and then to image generation quality score (fusion evaluation index value). The second row demonstrates the image quality prediction network based on machine perception. In order to achieve end-to-end prediction of image quality, a prediction network is designed in the embodiment of the present invention, whose input is the image in the constructed image dataset and the output is the predicted quality score. The fusion evaluation index value generated above is used as a label for network training. Therefore, the second row completes the mapping from image to quality score by modeling the machine perception process.

[0034] In an optional implementation of the present invention, step S1 includes: using clarity, noise and brightness as degradation dimensions, respectively performing four different degrees of quality degradation on the original distortion-free image to obtain an image set for image quality assessment.

[0035] Specifically, in order to construct an image quality assessment dataset for the three major visual tasks of target detection, semantic segmentation, and human pose estimation, the embodiment of the present invention selects CoCo2017 as the reference image dataset. Taking into account the subsequent need to perform different degrees of quality degradation on the reference images, and the fact that the number of images contained in the CoCo2017 dataset is very large, the embodiment of the present invention only selects a part of them to participate in the subsequent model training, verification, and testing. In addition, since the downstream visual tasks include human pose estimation, in the embodiment of the present invention, one of the principles for selecting image sets is that these image sets must contain humans. Specifically, the embodiment of the present invention selected 600 images from the CoCo2017 training set, 60 images from the verification set, and 60 images from the test set as reference images.

[0036] In the field of security, surveillance cameras are prone to capture images with abnormal clarity, abnormal noise, and abnormal brightness at night due to jitter, electromagnetic interference, light pollution, and other reasons. To this end, in an embodiment of the present invention, based on the reference image, clarity, noise, and brightness are selected as degradation dimensions, and four different degrees of quality degradation are performed respectively. Among them, the degradation strategy of the clarity dimension is to add Gaussian blur to the image, the degradation strategy of the noise dimension is to add Gaussian noise to the image, and the degradation strategy of the brightness dimension is to linearly adjust the brightness value of each pixel in the image. Finally, an image quality assessment dataset was successfully constructed, in which the training set contains 9,000 images, and the validation set and test set each contain 900 images. Figure 2 A schematic diagram of an original undistorted image and a degraded image in an image quality assessment method based on multi-vision task fusion provided by an embodiment of the present invention is shown.

[0037] In the embodiment of the present invention, step S2 includes: S21, inputting the image set for image quality assessment into the target detection model, using the first evaluation index for target detection and the second evaluation index for target detection, and using the target detection model to obtain the first evaluation index value and the second evaluation index value of the image set for image quality assessment, wherein the first evaluation index for target detection is the average precision of the bounding box, and the second evaluation index for target detection is the average precision of the bounding box when IoU=0.5; S22, inputting the image set for image quality assessment into the semantic segmentation model, using a first evaluation index for semantic segmentation and a second evaluation index for semantic segmentation, and using the semantic segmentation model to obtain a third evaluation index value and a fourth evaluation index value of the image set for image quality assessment, wherein the first evaluation index for semantic segmentation is the average precision, and the second evaluation index for semantic segmentation is the average IoU; S23, inputting the image set for image quality assessment into the pose estimation model, using the first evaluation index for pose estimation and the second evaluation index for pose estimation, and using the pose estimation model to obtain a fifth evaluation index value and a sixth evaluation index value of the image set for image quality assessment, wherein the first evaluation index for pose estimation is the average accuracy, and the second evaluation index for pose estimation is the average accuracy of the bounding box when OKS=0.5; S24, fusing the first to sixth evaluation index values ​​of the image set for image quality evaluation to obtain a fused evaluation index value corresponding to the image set for image quality evaluation.

[0038] In the embodiment of the present invention, based on the above image set, the image quality score can be generated for a specific visual task. The fusion evaluation index value generation process is as follows: Figure 3As shown, specifically, in the embodiment of the present invention, target detection, semantic segmentation and human posture estimation are selected as downstream visual tasks to guide the generation of quality scores. Specifically, in the embodiment of the present invention, SOTA models are selected for these three visual tasks; the method used for target detection is Co-DETR, the method used for semantic segmentation is ONE-PEACE, and the method used for posture estimation is 4×RSN-50. The input of the model is an image dataset, and the output is the evaluation index corresponding to the visual task. The evaluation indicators selected for target detection are the bounding box average precision (bbox-mAP-50) and the bounding box average precision (bbox-mAP) at IoU=0.5, the evaluation indicators selected for semantic segmentation are the average accuracy (aAcc) and the average IoU (mIoU), and the evaluation indicators selected for posture estimation are the average precision (AP-50) and the average precision (AP) at OKS=0.5.

[0039] In an embodiment of the present invention, the target detection model, the semantic segmentation model and the posture estimation model are pre-trained models with target detection capabilities, semantic segmentation capabilities and posture estimation capabilities, respectively. In an embodiment of the present invention, each image in an image set for image quality assessment is input into the target detection model, the semantic segmentation model and the posture estimation model respectively to complete the corresponding visual tasks, and in the process of the model performing the visual tasks, respective evaluation indicators are obtained.

[0040] In addition, since the image quality score is expected to be expressed as a percentage in the embodiment of the present invention, the values ​​of the above performance indicators need to be converted into a range of 0-100. Then, the indicators of the three visual tasks are fused according to formula 1 to obtain the fused quality score (fusion evaluation indicator value): (1) In formula (1), q i Indicates i The quality score of the fused images is OB i 1 and OB i 2 Corresponding to two evaluation indicators for target detection; SS i 1 and SS i 2 They correspond to two evaluation indicators of semantic segmentation respectively; PE i 1 and PE i 2 They correspond to two evaluation indicators of posture estimation.

[0041] In order to improve the learning ability of the image quality prediction model, in the embodiment of the present invention, it is also proposed to normalize the quality score of the image.

[0042] Figure 4 Two original images and their generated distorted images under the same quality degradation are given, and their fusion evaluation index values ​​are given. These fusion evaluation index values ​​are determined based on the task fitness of the images in three visual tasks. Figure 4 As can be seen from the figure, the task fitness of the second column of images is significantly worse than that of the first column of images. This is because the scenes in the second column of images are more complex and contain more subtle targets.

[0043] In the embodiment of the present invention, the end-to-end image quality prediction model to be trained includes: a feature representation module, a local feature extraction module, a global feature extraction module, a feature fusion module and a fully connected module connected in sequence; the goal of the model is to extract features from the input image and finally output an evaluation result reflecting the image quality level. Specifically, the step S3 includes: S31, inputting the image set for image quality assessment into a feature representation module to obtain a feature vector; S32, extracting features from the feature vector through a local feature extraction module to obtain local features; S33, extracting features from the feature vector through a global feature extraction module to obtain global features; S34, fusing the local features and the global features through a fusion module to obtain a fusion feature; S35, processing the fused features through a fully connected module to obtain an image quality prediction value; S36, based on the true value label and the image quality prediction value, update the parameters of the end-to-end image quality prediction model to be trained to obtain a trained end-to-end image quality prediction model.

[0044] In the embodiment of the present invention, the structure of the end-to-end image quality prediction model is as follows: Figure 5 As shown. The model consists of two parts: a feature extraction network and a fully connected network. The feature extraction network consists of two sub-networks, one responsible for extracting the global features of the image, and the other responsible for extracting the local features of the image. In an embodiment of the present invention, the sub-network for extracting global features is constructed based on the Swin Transformer module, and the sub-network for extracting local features uses the ConvNext module. The global and local features extracted by the two sub-networks are integrated through a feature fusion unit, and the fused features are fed into the subsequent fully connected network.

[0045] Specifically, first, the image is cropped to 224×224 and converted into a feature vector. Further, the feature vector is input into the feature representation module (Patch Partition layer and Linear Embedding layer) for image partitioning, and the vector of each image block is mapped to a high-dimensional space to enhance the feature representation. The window size for cropping the image is 7×7, and a high-dimensional feature vector is obtained. Then, the high-dimensional feature vector is input into the global feature extraction network and the local feature extraction network for further feature extraction.

[0046] In the embodiment of the present invention, the local feature extraction module is divided into four stages of local feature extraction submodules, and each stage of the local feature extraction submodule is composed of a different number of ConvNeXt blocks; The number of the fusion modules is 4. At the end of each feature extraction stage, the feature fusion module fuses the global features and local features of that stage to serve as the input of the next stage.

[0047] The global feature extraction network is divided into 4 stages, each stage consists of a different number of Swin Transformer blocks, specifically, the number of Swin Transformer blocks in each stage is 2, 2, 9, 2 respectively. The embedding dimension of the feature is initially 96, and the feature dimension doubles after each stage. In addition, a multi-head attention mechanism is used in each stage, and the number of multi-head attention heads is 3, 6, 12, and 24 respectively. The local feature extraction network is also divided into 4 stages, each stage consists of a different number of ConvNeXt blocks, and the number of blocks is 3, 3, 6, and 3 respectively. The initial embedding dimension of the feature is 96, and the embedding dimension of the feature doubles after each stage. It is worth noting that at the end of each stage, the feature fusion unit fuses the global and local features and then serves as the input of the next stage. The feature fusion unit uses the attention mechanism to generate attention weights by calculating the similarity between global features and local features.

[0048] In an embodiment of the present invention, step S34 specifically includes: calculating the similarity between the global features and the local features through a fusion module; performing a 1×1 convolution operation on the global features to generate attention weights; and weighting and fusing the global and local features to obtain fused features.

[0049] Specifically, the feature fusion process can be completed through a 1×1 convolution operation, in which the input and output channels of each feature fusion module in the convolution operation are the feature dimensions of the corresponding stage. Finally, the local features and global features are weighted and fused using the generated attention weights. After extracting the global and local fusion features of the image, the feature vector is fed into the subsequent two-layer fully connected network, with 768 input channels and 256 output channels in the first layer and 256 input channels and 1 output channel in the second layer. The loss function used in the training of the model is the mean square error: (2) In formula (2), N is the total number of image datasets, For the i The quality score label of the image, is the quality score predicted by the model.

[0050] In an embodiment of the present invention, the quality metric of an image is generated by the task fitness of the image for the visual tasks of target detection, semantic segmentation, and posture estimation. High-quality images often have better task fitness performance than low-quality images. Taking the target detection task as an example, after the image is degraded in the dimensions of clarity, noise, and brightness, the overall performance of the task gradually decreases as the degree of degradation increases. For example, as the degree of image quality degradation increases, it is basically impossible to affect the detection of two targets (people and skateboards) with a higher pixel percentage in the image. For the target car, its low pixel percentage causes it to contain low-density effective information. When the degree of degradation reaches a certain level, it is basically impossible to complete the target detection task.

[0051] However, not all high-quality images are better than low-quality images in task fitness. This fitness is also closely related to the complexity of the semantic information in the image and the diversity of the scene. In the two images that have not been degraded, the image with more homogeneous scenes and simpler semantic information tends to have higher task fitness.

[0052] The image quality score generated in the image quality score generating section of the embodiment of the present invention will be used as the basic truth value of the quality score prediction model.

[0053] The embodiment of the present invention first constructs an image dataset for quality assessment, which contains original images and degraded images. The selected degradation dimensions are clarity, brightness and noise. Each dimension has four degradation levels. Based on the dataset, the embodiment of the present invention selects object detection, semantic segmentation and pose estimation as downstream visual tasks, simulates the machine perception process of the image, and generates an objective quality score of the image.

[0054] Furthermore, in the embodiment of the present invention, an end-to-end image quality prediction model is trained, wherein the input of the model is an image and the output is a corresponding quality score. The aforementioned quality score is used as the ground truth for model training.

[0055] In the embodiments of the present invention, the visual tasks simulating machine perception can be more diverse, and image quality assessment can be jointly learned with other related visual tasks (e.g., object recognition, image classification, etc.), and the correlation of these tasks can be used to improve the performance and robustness of the image quality assessment model. In addition, the distortion categories of images in the dataset can also be richer. In this way, by learning more types of distortion features, the model can better capture various distortion patterns, thereby improving its ability to recognize and evaluate new and unknown distortion types.

[0056] Based on the same inventive concept, an embodiment of the present invention further provides a system for image quality assessment based on multi-vision task fusion, the system comprising: An image generation subsystem, used for degrading the original distortion-free image to a plurality of different quality levels, to obtain an image set for image quality assessment; An index value evaluation subsystem is used to input an image set for image quality evaluation into a target detection model, a semantic segmentation model, and a posture estimation model to obtain evaluation index values ​​corresponding to multiple visual tasks, wherein the multiple visual tasks include: target detection, semantic segmentation, and posture estimation; and to fuse the evaluation index values ​​corresponding to the multiple visual tasks to obtain a fused evaluation index value corresponding to the image set for image quality evaluation; a training subsystem, configured to train an end-to-end image quality prediction model to be trained based on the image set for image quality assessment, using the fusion evaluation index value corresponding to the image set for image quality assessment as a true value label, to obtain a trained end-to-end image quality prediction model, wherein the trained end-to-end image quality prediction model is used to simulate a process in which a machine perceives an image to perform image quality assessment; The application subsystem is used to input the image to be evaluated into the trained end-to-end image quality prediction model to obtain an image quality evaluation result.

[0057] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps in the method for image quality assessment based on multi-visual task fusion as described in any of the above embodiments are implemented.

[0058] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for image quality assessment based on multi-visual task fusion described in any of the above embodiments.

[0059] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the method for image quality assessment based on multi-visual task fusion described in any of the above embodiments.

[0060] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0061] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0062] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0064] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0065] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0066] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0067] The above is a detailed introduction to the method for image quality assessment based on multi-visual task fusion provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for image quality assessment based on multi-vision task fusion, characterized in that: The method comprises: Degrading the original distortion-free image to multiple different quality levels to obtain an image set for image quality assessment; Inputting an image set for image quality assessment into a target detection model, a semantic segmentation model, and a posture estimation model to obtain evaluation index values ​​corresponding to multiple visual tasks, wherein the multiple visual tasks include: target detection, semantic segmentation, and posture estimation; fusing the evaluation index values ​​corresponding to the multiple visual tasks to obtain a fused evaluation index value corresponding to the image set for image quality assessment; Taking the fusion evaluation index value corresponding to the image set for image quality evaluation as the true value label, based on the image set for image quality evaluation, the end-to-end image quality prediction model to be trained is trained to obtain a trained end-to-end image quality prediction model, wherein the trained end-to-end image quality prediction model is used to simulate the process of machine perception of images to perform image quality evaluation; The image to be evaluated is input into the trained end-to-end image quality prediction model to obtain an image quality evaluation result.

2. The method for image quality assessment based on multi-visual task fusion according to claim 1, characterized in that: The original undistorted image is degraded to multiple different quality levels to obtain an image set for image quality assessment, including: Taking clarity, noise and brightness as degradation dimensions, the original undistorted image is subjected to four different degrees of quality degradation to obtain an image set for image quality assessment.

3. The method for image quality assessment based on multi-visual task fusion according to claim 1, characterized in that: The image set used for image quality assessment is input into the target detection model, the semantic segmentation model and the pose estimation model to obtain the evaluation index values ​​corresponding to the multiple visual tasks, and the evaluation index values ​​corresponding to the multiple visual tasks are fused to obtain the fused evaluation index value corresponding to the image set used for image quality assessment, including: Inputting an image set for image quality assessment into a target detection model, using a first evaluation index for target detection and a second evaluation index for target detection, and using the target detection model to obtain a first evaluation index value and a second evaluation index value of the image set for image quality assessment, wherein the first evaluation index for target detection is an average bounding box precision, and the second evaluation index for target detection is an average bounding box precision when IoU=0.5; Inputting an image set for image quality assessment into a semantic segmentation model, using a first evaluation index for semantic segmentation and a second evaluation index for semantic segmentation, and using the semantic segmentation model to obtain a third evaluation index value and a fourth evaluation index value of the image set for image quality assessment, wherein the first evaluation index for semantic segmentation is average precision, and the second evaluation index for semantic segmentation is average IoU; Inputting the image set for image quality assessment into the pose estimation model, using a first evaluation index for pose estimation and a second evaluation index for pose estimation, and using the pose estimation model to obtain a fifth evaluation index value and a sixth evaluation index value of the image set for image quality assessment, wherein the first evaluation index for pose estimation is the average accuracy, and the second evaluation index for pose estimation is the average accuracy of the bounding box when OKS=0.5; The first to sixth evaluation index values ​​of the image set used for image quality evaluation are fused to obtain a fused evaluation index value corresponding to the image set used for image quality evaluation.

4. The method for image quality assessment based on multi-visual task fusion according to claim 1, characterized in that: The end-to-end image quality prediction model to be trained includes: a sequentially connected feature representation module, a local feature extraction module, a global feature extraction module, a feature fusion module and a fully connected module; Taking the fusion evaluation index value corresponding to the image set for image quality evaluation as the true value label, the end-to-end image quality prediction model to be trained is trained based on the image set for image quality evaluation, including: The image set used for image quality assessment is input into the feature representation module to obtain a feature vector; Extract features from feature vectors through a local feature extraction module to obtain local features; The feature vector is extracted through the global feature extraction module to obtain the global feature; The local features and global features are fused through the fusion module to obtain fused features; The fused features are processed through a fully connected module to obtain the image quality prediction value; Based on the true value label and the image quality prediction value, the parameters of the end-to-end image quality prediction model to be trained are updated to obtain a trained end-to-end image quality prediction model.

5. The method for image quality assessment based on multi-visual task fusion according to claim 4 is characterized in that: The global feature extraction module is divided into four stages of global feature extraction sub-modules, and each stage of the global feature extraction sub-module consists of a different number of Swin Transformer blocks; The local feature extraction module is divided into four stages of local feature extraction submodules, and each stage of the local feature extraction submodule consists of a different number of ConvNeXt blocks; The number of the fusion modules is 4. At the end of each feature extraction stage, the feature fusion module fuses the global features and local features of that stage to serve as the input of the next stage.

6. The method for image quality assessment based on multi-visual task fusion according to claim 4, characterized in that: The local features and global features are fused through the fusion module to obtain fusion features, including: The similarity between global features and local features is calculated through the fusion module; Perform 1×1 convolution operation on the global features to generate attention weights; The global and local features are weighted and fused to obtain the fused features.

7. A system for image quality assessment based on multi-vision task fusion, characterized in that: The system comprises: An image generation subsystem, used for degrading the original distortion-free image to a plurality of different quality levels, to obtain an image set for image quality assessment; An index value evaluation subsystem is used to input an image set for image quality evaluation into a target detection model, a semantic segmentation model, and a posture estimation model to obtain evaluation index values ​​corresponding to multiple visual tasks, wherein the multiple visual tasks include: target detection, semantic segmentation, and posture estimation; and to fuse the evaluation index values ​​corresponding to the multiple visual tasks to obtain a fused evaluation index value corresponding to the image set for image quality evaluation; a training subsystem, configured to train an end-to-end image quality prediction model to be trained based on the image set for image quality assessment, using the fusion evaluation index value corresponding to the image set for image quality assessment as a true value label, to obtain a trained end-to-end image quality prediction model, wherein the trained end-to-end image quality prediction model is used to simulate a process in which a machine perceives an image to perform image quality assessment; The application subsystem is used to input the image to be evaluated into the trained end-to-end image quality prediction model to obtain an image quality evaluation result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for image quality assessment based on multi-visual task fusion described in any one of claims 1-6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for image quality assessment based on multi-visual task fusion described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the method for image quality assessment based on multi-visual task fusion described in any one of claims 1-6 are implemented.

Images