Image quality evaluation method and device, storage medium and electronic equipment

Abstract

The invention relates to an image quality evaluation method and device, a storage medium and electronic equipment, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-evaluated image; the to-be-evaluated image is input into a quality evaluation model, the quality evaluation model comprises a feature extraction module and a full connection module, and the full connection module is used for changing the size of the to-be-evaluated image; and performing feature extraction on the to-be-evaluated image based on the feature extraction module to obtain a feature extraction result, and processing the feature extraction result by using the full connection module to obtain a quality score of the to-be-evaluated image. According to the invention, the quality score of the to-be-evaluated image is obtained by using the quality evaluation model, so that the flexibility and accuracy of image quality evaluation can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image quality assessment method, device, storage medium, and electronic device. Background Art

[0002] Currently, images or videos are often distorted due to various reasons during the process of acquisition, processing, transmission and storage. Image or video distortion will greatly affect the subsequent processing and application of the image or video. Therefore, image or video quality assessment is particularly important.

[0003] In related technologies, image or video quality assessment typically relies on subjective evaluation or simple statistical metrics. However, these assessment methods often fail to accurately reflect the human eye's perception of image quality. Therefore, how to better assess image quality is a pressing technical issue. Summary of the Invention

[0004] The purpose of the present disclosure is to provide an image quality assessment method, device, storage medium and electronic device to solve the technical problems existing in the related art.

[0005] According to a first aspect of the present disclosure, a method for evaluating image quality is provided, the method comprising: Obtaining the image to be evaluated; Inputting the image to be evaluated into a quality assessment model, wherein the quality assessment model includes a feature extraction module and a fully connected module, and the fully connected module is used to change the size of the image to be evaluated; Based on the feature extraction module, feature extraction is performed on the image to be evaluated to obtain a feature extraction result, and the feature extraction result is processed by the fully connected module to obtain a quality score of the image to be evaluated.

[0006] Optionally, the fully connected module includes a global pooling layer, and the processing of the feature extraction result by using the fully connected module to obtain the quality score of the image to be evaluated includes: Pooling each channel corresponding to the feature extraction result based on the global pooling layer to obtain a target feature map; The target feature map is classified to obtain the quality score.

[0007] Optionally, the fully connected module further includes a dilated spatial pyramid pooling module, and obtaining the quality score of the image to be evaluated by using the fully connected module and the feature extraction result includes: Performing dilated convolution on the feature extraction results using the dilated spatial pyramid pooling module to obtain candidate feature maps of different scales, and performing depthwise separable convolution on the candidate feature maps of different scales to obtain a target feature map of the same size as the image; The target feature map is classified to obtain the quality score.

[0008] Optionally, the quality assessment model further includes a channel attention module, and after extracting features from the image to be assessed based on the feature extraction module and obtaining a feature extraction result, the following steps are included: Performing channel weighting on the feature extraction result based on the channel attention module to obtain a channel weighted result; The method of performing dilated convolution on the feature extraction result using the dilated spatial pyramid pooling module to obtain candidate feature maps of different scales includes: The dilated spatial pyramid pooling module is used to perform dilated convolution on the channel weighted results to obtain candidate feature maps of different scales.

[0009] Optionally, the quality assessment model further includes a spatial attention module; after extracting features from the image to be assessed based on the feature extraction module and obtaining a feature extraction result, the method further includes: Performing spatial weighting on the feature extraction result based on the spatial attention module to obtain a spatial weighted result; The method of performing dilated convolution on the feature extraction result using the dilated spatial pyramid pooling module to obtain candidate feature maps of different scales includes: The dilated spatial pyramid pooling module is used to perform dilated convolution on the spatial weighted result to obtain candidate feature maps of different scales.

[0010] Optionally, the method further includes: Acquire a sample data set, where the sample data set includes a plurality of sample images; receiving scoring information corresponding to each of the sample images, and generating a label data set based on the scoring information; The quality assessment model is trained using a training dataset consisting of the label dataset and the sample dataset.

[0011] Optionally, each of the sample images corresponds to a plurality of scoring information, and generating a label dataset from the scoring information includes: Averaging the multiple scoring information and using the average result as the label information of the corresponding sample image; The label data set is composed according to the label information of each sample image.

[0012] In a second aspect, the present disclosure provides an image quality assessment device, the device comprising: an acquisition module, configured to acquire an image to be evaluated; an input module configured to input the image to be evaluated into a quality assessment model, wherein the quality assessment model includes a feature extraction module and a fully connected module, and the fully connected module is used to change the size of the image to be evaluated; The processing module is configured to perform feature extraction on the image to be evaluated based on the feature extraction module to obtain a feature extraction result, and process the feature extraction result using the fully connected module to obtain a quality score of the image to be evaluated.

[0013] In a third aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0014] In a fourth aspect, the present disclosure provides an electronic device, the electronic device comprising: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of any one of the methods in the first aspect.

[0015] In a first aspect, the present disclosure provides a computer program product, comprising a computer program, which implements the steps of any one of the methods in the first aspect when executed by a processor.

[0016] After acquiring the image to be evaluated, the present disclosure inputs the image to be evaluated into a quality assessment model. The quality assessment model may include a feature extraction module and a fully connected module, wherein the fully connected module can be used to change the size of the image to be evaluated. On this basis, feature extraction is performed on the image to be evaluated based on the feature extraction module to obtain a feature extraction result, and the feature extraction module is processed by the fully connected module to obtain a quality score of the image to be evaluated. By utilizing the quality assessment model, the embodiment of the present disclosure can not only change the size of the image to be evaluated, but also flexibly and effectively implement the evaluation of the image, thereby improving the accuracy of image evaluation.

[0017] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 The figure is a flowchart of a method for evaluating image quality according to an exemplary embodiment.

[0019] Figure 2 The figure is a structural example diagram of a quality assessment model in an image quality assessment method according to an exemplary embodiment.

[0020] Figure 3 2 is a structural example diagram of ResNet 18 in an image quality assessment method according to an exemplary embodiment.

[0021] Figure 4 FIG. 4 is a structural example diagram of a Block module in an image quality assessment method according to an exemplary embodiment.

[0022] Figure 5 FIG. 5 is an example diagram showing five distortion levels in a method for evaluating image quality according to an exemplary embodiment.

[0023] Figure 6 FIG. 4 is an overall example diagram showing a method for evaluating image quality according to an exemplary embodiment.

[0024] Figure 7 The figure is a flowchart of another image quality assessment method according to an exemplary embodiment.

[0025] Figure 8 This is a structural example diagram of a global pooling layer in another image quality assessment method according to an exemplary embodiment.

[0026] Figure 9 3 is a structural example diagram of a dilated spatial pyramid pooling module in another image quality assessment method according to an exemplary embodiment.

[0027] Figure 10 2 is a structural example diagram of a channel attention module in another image quality assessment method according to an exemplary embodiment.

[0028] Figure 11 3 is a structural example diagram of a spatial attention module in another image quality assessment method according to an exemplary embodiment.

[0029] Figure 12 FIG. 4 is a diagram illustrating an overall structure of a quality assessment model in another image quality assessment method according to an exemplary embodiment.

[0030] Figure 13 The figure is a block diagram showing an image quality assessment according to an exemplary embodiment of the present disclosure.

[0031] Figure 14is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0033] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0034] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0035] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0036] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0037] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0038] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0039] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0040] Images and videos currently play a vital role in people's lives, being used in a wide range of fields, including entertainment, social interaction, education, healthcare, and scientific research. Furthermore, videos are composed of multiple images (video frames), making image quality crucial for these applications. For example, users have high expectations for image quality, scientific research requires accurate image data, and doctors rely on clear images for diagnoses.

[0041] However, current image quality evaluation methods usually cannot meet the above requirements. That is, existing image evaluation methods mainly rely on human subjective evaluation or use some simple statistical indicators, but these methods cannot well reflect the human eye's perception of image quality.

[0042] Furthermore, in terms of product demand, users and businesses are increasingly demanding higher image quality. Users desire clearer, more realistic images to enhance their viewing experience; businesses rely on high-quality images to better showcase their products and enhance their brand image. Some online services also require users to upload high-quality images to provide more accurate feedback. Therefore, evaluating image quality is crucial.

[0043] Related technologies often use simple, artificial statistical metrics to indicate image quality, such as signal-to-noise ratio (SNR) and mean squared error (MSE). While these methods can provide useful information in some cases, they all suffer from significant drawbacks. Specifically, these metrics assess image quality from a mathematical or statistical perspective, rather than from the perspective of human visual perception. For example, image A may appear statistically very similar to image B, yet to the human eye, the quality of images A and B may differ significantly. This is primarily due to the nonlinear nature of the human visual system, which typically has varying sensitivities to different visual characteristics. These visual characteristics can include brightness, color, and texture.

[0044] It can be seen that related technologies ignore some important visual information during the image quality assessment process. For example, the signal-to-noise ratio and mean square error usually only consider the difference in pixel values, but ignore the spatial structure of the image, which is a relatively important part of the information when the human eye evaluates image quality. In addition, this method does not take into account color information and sensitivity to noise when performing quality assessment. Among them, color is an important factor that affects the human eye's perception of image quality. Furthermore, this method is very sensitive to image noise. For example, in an image, even if only a small area is affected by noise, the signal-to-noise ratio and mean square error may change significantly, which can lead to an over-assessment or underestimation of image quality.

[0045] Although related technologies use simple models to evaluate image quality, this method also has the following disadvantages: first, the accuracy is limited, that is, simple models can only consider some characteristics of the image, such as brightness, contrast, etc., while ignoring other important characteristics, such as texture, color, etc., which will cause the evaluation results of the model to deviate greatly from the perception of the human eye; second, the ability to process complex scenes is weak, that is, simple models usually do not work well when processing complex images. For example, images contain multiple objects, multiple colors, and multiple textures. These complex scenes usually require more complex models to correctly understand and process; it cannot handle human eye perception factors, that is, simple models cannot fully consider the human eye's subjective perception of image quality, such as preferences for color, contrast, and brightness, which will cause the evaluation results of the model to be different from people's actual feelings.

[0046] In order to solve the above problems, the embodiments of the present disclosure propose a method for image quality assessment. This method can achieve more accurate assessment by introducing a quality assessment model. It is an image quality assessment method that is more in line with human eye perception. That is, by combining human subjective evaluation and machine evaluation, the embodiments of the present disclosure can take into account both human subjective feelings and machine computing capabilities.

[0047] Figure 1 A method for evaluating image quality is shown according to an exemplary embodiment. Figure 1 As shown, the image quality assessment method may include the following steps: In step S110 , an image to be evaluated is acquired.

[0048] In an embodiment of the present disclosure, the image to be evaluated may be an image whose visual quality needs to be determined. The image to be evaluated may include key information and background information, and the image to be evaluated may include multiple target objects, which may be human bodies, animals, buildings, objects, etc.

[0049] Here, the image to be evaluated can be a real-time image collected by an automated collection system, such as an image taken by a webcam or a social media mobile phone. The image to be evaluated can also be a locally pre-stored image, or an image obtained by a web crawler.

[0050] After acquiring the initial image, the embodiment of the present disclosure may preprocess the initial image, such as performing denoising, white balance adjustment, and contrast enhancement on the initial image to obtain an image to be evaluated, thereby improving the accuracy of quality assessment.

[0051] Optionally, embodiments of the present disclosure may obtain multiple candidate images, extract common features from these multiple candidate images, and then fuse the multiple candidate images based on these common features to obtain the image to be evaluated. The multiple candidate images may come from different sources, such as a first candidate image from a webcam and a second candidate image from a smartphone.

[0052] Specifically, the disclosed embodiment can pre-process the collected multiple candidate images and automatically screen out images suitable for evaluation based on specific criteria (such as resolution, size, content, etc.). On this basis, computer vision technology is used to extract features from the screened images, such as edges, corners, textures, color distribution, etc., and align images from different sources to the same coordinate system to achieve image registration, that is, perform geometric transformations such as translation, rotation, and scaling. Feature matching is performed between the registered images to find common feature points across images. Based on the matched feature points, feature maps are created for each image. These feature maps can be key information in the image. Finally, the feature maps of different images are fused, and image reconstruction and dilated convolution are performed based on the fused image to obtain the image to be evaluated.

[0053] By performing quality assessment on the fused images to be evaluated, not only can the quality of the images themselves be improved, but more accurate and reliable visual information can also be provided for a variety of applications.

[0054] In step S120 , the image to be evaluated is input into the quality assessment model.

[0055] As an optional approach, after acquiring the image to be evaluated, the embodiment of the present disclosure may input the image to be evaluated into a quality assessment model, and the quality assessment model may include a feature extraction module and a fully connected module.

[0056] As a specific implementation, the quality assessment model can be a CNN (Convolutional Neural Network) or a Transformer. Preferably, to ensure a sufficiently small model structure, the quality assessment model can adopt a CNN model. In terms of model architecture, the disclosed embodiment can adopt an encoder approach, primarily because encoders have the ability to extract and learn data features, making them more suitable for processing large-scale data.

[0057] For example, the architecture of the quality assessment model (encoder) can be as follows Figure 2 As shown, based on Figure 2 It is known that the quality assessment model may include a feature extraction module 121 and a fully connected module 122, wherein the feature extraction module 121 may implement feature extraction using a resnet18 network, a shufflenet0.5 network, a shufflenet1.0 network, a mobilenet0.5 network, or a mobilenet1.0 network.

[0058] As an example, the embodiment of the present disclosure may select resnet18 as the network for extracting features. The structure of resnet18 may be as follows: Figure 3 Based on Figure 3 As we know, resnet18 can include input layer (Image), convolution layer (conv), maximum pooling layer (max pool), residual block (Block), average pooling layer (avg pool) and fully connected layer (Fully Connected, FC).

[0059] The convolution layer can be a 7*7 structure (7*7 conv). In addition, Block can be a module. In the resnet18 of the embodiment of the present disclosure, Block can be repeated 8 times. The specific structure of the Block module can be as follows: Figure 4 Based on Figure 4 It is known that the Block module can include two consecutive convolutional layers, and these two convolutional layers can be a 3*3 structure.

[0060] In the embodiment of the present disclosure, the fully connected module 122 can be used to change the size of the image to be evaluated. Before the image to be evaluated is input into the quality assessment model, the embodiment of the present disclosure may not adjust the size of the image to be evaluated, that is, the size of the image input into the quality assessment model can be arbitrary, and the user can choose different image sizes according to personal needs.

[0061] In some embodiments, the quality assessment model can be trained using a training dataset. Therefore, before inputting the image to be assessed into the quality assessment model, embodiments of the present disclosure can first obtain a training dataset. The training dataset can include a sample dataset and a labeled dataset. The sample dataset can be a public dataset or a manufactured dataset.

[0062] For example, the public datasets may include TID2013 and KADID10K, for example, TID2013 may include 25 reference images and 3000 distorted images. The manufactured dataset may be manufactured by changing the clarity of the image. Specifically, the disclosed embodiment may utilize methods such as reducing image clarity, random blur, random noise, salt and pepper noise, slight motion blur, mosaic and block effects to change the image from the clearest to the most blurred. The change process may be divided into five distortion levels, which may be as follows: Figure 5 As shown, Figure 5 The a image in the image may be the clearest image, while the e image may be the most blurred image, and the image clarity from the a image to the e image may decrease gradually.

[0063] The public dataset and manufacturing dataset described above can be collectively referred to as a sample dataset. This sample dataset can include multiple sample images. As mentioned above, these sample images can include images of varying quality. In other words, the same image can correspond to multiple images of varying resolutions, and these images of varying resolutions can form the sample dataset.

[0064] After acquiring the sample dataset, the disclosed embodiment can receive the scoring information corresponding to each sample image and generate a label dataset from the scoring information. Based on this, a training dataset is generated from the label dataset and the sample dataset, and the training dataset is used to train the initial network to obtain a quality assessment model.

[0065] As can be seen, annotating images is the foundation of model training. Scoring images provides scoring information. The annotated data is fed into the initial network, which is then instructed to find patterns in quality assessment based on the existing training dataset.

[0066] In some embodiments, each sample image may correspond to multiple scoring information. In the process of generating a label data set from the scoring information, the embodiment of the present disclosure may average the multiple scoring information and use the average result as the label information of the corresponding sample image. Finally, a label data set may be formed based on the label information of each sample image.

[0067] Here, the labeled data set can be manually labeled based on one's own feelings. Since the quality assessment model in the embodiment of the present disclosure needs to take into account people's subjective feelings, it cannot be simply labeled using algorithms and indicators. The specific score strategy can be shown in Table 1.

[0068] Table 1

[0069] Table 1 can help users better score sample images.

[0070] As an example, the disclosed embodiment can collect about 5,000 images of various qualities, assign them to ten people for scoring, and then average the scores to obtain the final score of the image. Figure 5 For the image e shown in the figure, the scores given by 10 people are 1 point, 1 point, 2 points, 1 point, 2 points, 1 point, 1 point, 1 point, 1 point and 1 point respectively. The label information of the image e is 1.2 points.

[0071] The acquisition process of the above training data set can be implemented by the data production module.

[0072] In step S130 , feature extraction is performed on the image to be evaluated based on the feature extraction module to obtain a feature extraction result, and the feature extraction result is processed using the fully connected module to obtain a quality score of the image to be evaluated.

[0073] In some embodiments, the quality assessment model may utilize a feature extraction module to extract features from an input image to be assessed, thereby obtaining a feature extraction result, and then utilize a fully connected module to process the feature extraction result to obtain a quality score for the image to be assessed.

[0074] In other words, the image to be evaluated can be scored through feature extraction and full connectivity. Because the quality assessment model is trained on a large number of manually annotated images, it can accurately assess the quality of the image. Based on this, the disclosed embodiments can provide feedback and guidance for user operations.

[0075] It should be noted that the target of quality assessment can be either an image or a video. That is, the embodiments of the present disclosure can obtain an image to be evaluated and use a quality assessment model to evaluate and analyze the image to be evaluated to obtain a quality score corresponding to the image. Alternatively, the embodiments of the present disclosure can also obtain a video to be evaluated and use a quality assessment model to evaluate and analyze the video to obtain a quality score corresponding to the video. The specific choice of evaluating an image or a video is not explicitly limited here and can be made based on actual circumstances.

[0076] In summary, the image quality assessment method proposed in the embodiment of the present disclosure can be implemented through three modules, which can include a data production module, a model training module and a model reasoning module, as detailed in Figure 6 The embodiment of the present disclosure can be based on Figure 6 The data creation module shown generates a training dataset. Based on this, the initial network is trained using the training dataset generated by the data creation module. Specifically, the training dataset generated by the data creation module is input into the model training module, which then trains the quality assessment model. Finally, the disclosed embodiments utilize a model inference module to obtain a quality score for the image to be evaluated. During this process, the image to be evaluated is input into a pre-trained quality assessment model, and the model inference module is used to infer the image quality.

[0077] After acquiring the image to be evaluated, the embodiment of the present disclosure inputs the image to be evaluated into a quality assessment model. The quality assessment model may include a feature extraction module and a fully connected module, wherein the fully connected module can be used to change the size of the image to be evaluated. On this basis, feature extraction is performed on the image to be evaluated based on the feature extraction module to obtain a feature extraction result, and the feature extraction module is processed by the fully connected module to obtain a quality score of the image to be evaluated. By utilizing the quality assessment model, the embodiment of the present disclosure can not only change the size of the image to be evaluated, but also flexibly and effectively implement image evaluation, thereby improving the accuracy of image evaluation.

[0078] Figure 7 Another image quality assessment method according to an exemplary embodiment is shown. Figure 7 As shown, the image quality assessment method may include the following steps: In step S210 , an image to be evaluated is acquired.

[0079] In step S220 , the image to be evaluated is input into the quality assessment model.

[0080] The specific implementation of steps S210 to S220 has been described in detail in the above embodiment and will not be repeated here.

[0081] In step S230 , feature extraction is performed on the image to be evaluated based on the feature extraction module to obtain a feature extraction result.

[0082] In the disclosed embodiments, a feature extraction module extracts features from an image to be evaluated, generating a feature extraction result. The feature extraction result may include multiple feature maps, including low-level feature maps and high-level feature maps. The low-level feature maps are used to capture local, low-level features (such as edges and textures), while the high-level feature maps are used to capture more abstract, high-level features (such as shapes and object parts).

[0083] In addition, each feature map can capture different features, and the channels of the feature map can have different focuses, which can further increase the diversity of features.

[0084] For example, ResNet-18 is primarily designed for images with a resolution of 224*224. However, over-compressing an image can affect image quality. Therefore, the disclosed embodiment can increase the image resolution to 600*600. However, increasing the image resolution can cause errors in the ResNet-18 model, primarily due to the encoder's fully connected module. Therefore, the disclosed embodiment can introduce a global pooling layer before the fully connected layer. See step S240 for details.

[0085] In step S240, each channel corresponding to the feature extraction result is pooled based on the global pooling layer to obtain a target feature map.

[0086] In some embodiments, the fully connected module may include a global average pooling (GAP) layer, which is used to pool each channel. Specifically, the global pooling layer can pool each channel corresponding to the feature extraction result to obtain a target feature map. During this process, embodiments of the present disclosure may take the average or maximum value of the pooled results.

[0087] Here, the global pooling layer can also be called the global average pooling layer, which can be a special pooling layer. The structure of the global pooling layer can be as follows Figure 8 As shown in Figure 2 , the disclosed embodiment uses a global pooling layer to globally pool each feature map, pooling each feature map into a single element, thereby resizing the image to be evaluated. This can reduce model parameters to a certain extent, lowering the risk of overfitting while preserving the global information of each feature map.

[0088] For example, the image size input to the quality assessment model is 224*224. After passing through the feature extraction module, the resulting feature map is 16*16*4. Based on this, the global pooling layer is used to resize the image. That is, after passing through the global pooling layer, the feature map size can be 1*1*4, and its fully connected parameters are 4*6.

[0089] As another example, the size of the image to be evaluated input to the quality assessment model is 448*448. After passing through the feature extraction module, the feature map obtained is 32*32*4. On this basis, the global pooling layer is used to change the size of the image. That is, after passing through the global pooling layer, the size of the feature map obtained can be 1*1*4, and its full connection parameters are 4*6. It can be seen that the parameters input to the full connection are consistent, so the size of the input image can be changed at will. As another example, the output of the last convolutional layer of the feature extraction module can be a 10*10*64 tensor, that is, the feature extraction result can include 64 feature maps, and the size of each feature map can be 10*10. Based on the global pooling layer GAP, the 10*10*64 tensor can be pooled into a 1*1*64 tensor, that is, each 10*10 feature map can be pooled into a value. In this way, the embodiment of the present disclosure can compress a tensor with 6400 elements into a tensor with only 64 elements.

[0090] On this basis, the 1*1*64 tensor is directly connected to the fully connected layer for classification. For example, if a 10-category classification is performed, the number of nodes in the fully connected layer is 10. Each node in the fully connected layer can perform an inner product operation with these 64 values to obtain 10 values, which are then converted into 10 probability values through the softmax function. These 10 probability values can be the probabilities that the input image belongs to the 10 categories.

[0091] In other embodiments, the fully connected module may further include an Atrous Spatial Pyramid Pooling (ASPP) module. Since the Atrous Spatial Pyramid Pooling module combines the concepts of Atrous Convolution and Spatial Pyramid Pooling, it can capture image features at different scales.

[0092] Specifically, the disclosed embodiment can use the dilated spatial pyramid pooling module to perform dilated convolution on the feature extraction results to obtain candidate feature maps of different scales, and perform depth-separable convolution on the candidate feature maps of different scales to obtain a target feature map of the same size as the image.

[0093] Directly performing pooling on each channel and selecting only one value is too simple. Therefore, the embodiment of the present disclosure can replace the fully connected module with the dilated spatial pyramid pooling ASPP module to sample the given input in parallel with dilated convolutions of different sampling rates. Here, the dilated spatial pyramid pooling module can be called a dilated spatial convolution pooling layer, and its structure can be as follows: Figure 9 shown. Figure 9 The w*h*c in the formula can be the feature map output by the feature extraction module. After the feature map is subjected to the dilated spatial pyramid pooling, the target feature map of the same size can be obtained.

[0094] In other embodiments, in order to support images of certain specific shooting techniques, such as blurring and focusing scenes, the disclosed embodiment may add an attention module to the quality assessment model so that the quality assessment model knows where the focus and attention points of the image are.

[0095] Specifically, the quality assessment model may include Figure 10 After obtaining the feature extraction results, the channel attention module shown in the figure can perform channel weighting on the feature extraction results based on the channel attention module (channel attention module) to obtain channel-weighted results. Based on this, the dilated spatial pyramid pooling module is used to perform dilated convolution on the channel-weighted results to obtain candidate feature maps of different scales.

[0096] Here, the channel attention module is used to enhance important feature channels. Specifically, the channel attention mechanism assigns different weights to each channel, allowing the quality assessment model to focus more on important feature channels. Furthermore, the channel attention module can also achieve selective feature enhancement, that is, by enhancing channels containing more useful information, reducing redundancy and noise, and improving the quality of feature representation.

[0097] Optionally, the quality assessment model may include Figure 11 After obtaining the feature extraction results, the present embodiment can perform spatial weighting on the feature extraction results based on the spatial attention module (spatial attention module) to obtain spatially weighted results. Based on this, the spatially weighted results are then subjected to dilated convolution using the dilated spatial pyramid pooling module to obtain candidate feature maps of different scales.

[0098] Here, the spatial attention module is used to enhance important spatial locations. Specifically, the spatial attention mechanism assigns different weights to each location, allowing the quality assessment model to focus on important spatial regions. Furthermore, the spatial attention module can also focus on salient feature regions, improving the discriminative power of feature maps by enhancing important regions in the image.

[0099] Optionally, the quality assessment model may include Figure 10 The channel attention module shown in Figure 11 The spatial attention module shown in the figure can more accurately evaluate image quality by adding these two modules.

[0100] In summary, to support specific scenarios, the disclosed embodiments can add an Attention module before the Dilated Spatial Pyramid Pooling module. This Attention module can be a channel Attention module, a spatial Attention module, or both. In other words, the disclosed embodiments can add a learnable weight between channels and a learnable weight to the feature map.

[0101] As a specific implementation, the quality assessment model in the embodiment of the present disclosure may include a feature extraction module, an attention module, and a dilated space pyramid pooling module. In this case, the structure of the quality assessment model may be as follows: Figure 12 Based on Figure 12 It is known that after obtaining the image to be evaluated 241, the embodiment of the present disclosure can first use the feature extraction module 242 to extract the features of the image to obtain a feature extraction result, and then pass it through a channel / spatial attention module 243 to achieve weighting, and then pass it through a void spatial pyramid pooling module 244 to process the size of the feature map. Finally, through a full connection 245, the score corresponding to the image to be evaluated can be obtained.

[0102] Optionally, the embodiment of the present disclosure may also perform corresponding transformations on the backbone and input resolution of the algorithm to correspond to different tasks. The indicator table after the transformation is shown in Table 2.

[0103] Table 2

[0104] Table 2 shows the impact of different networks and resolutions on the results. By comparison, we can see that when the image resolution is 224*224, the disclosed embodiment can use ShuffleNet 0.5 to evaluate image quality. Furthermore, when the image resolution is 600*600, the disclosed embodiment can use MobileNet 1.0 to evaluate image quality.

[0105] In step S250, the target feature map is classified to obtain a quality score.

[0106] When evaluating the quality of an image or video, the size of the playback window is an important consideration. For example, an old movie may look okay on a mobile phone, but if played on a large-screen TV, it may appear blurry due to the difference in resolution. Therefore, the resolution of the image needs to be considered. That is, the quality score of the same image at 480P and 1080P resolutions will be different. This phenomenon emphasizes that when evaluating the quality of an image or video, the size of the playback window and the resolution of the image are both important considerations. If the scores of images of different resolutions are to be compared, they need to be fixed at a certain resolution. However, the fixed image resolution usually requires judgment or padding. However, judgment or padding will modify the image to a certain extent, which will directly affect the image quality score.

[0107] To solve the above problems, the embodiments of the present disclosure introduce a fully connected module (global pooling layer or dilated spatial pyramid pooling module). The quality assessment model including the fully connected module can adopt a fixed-length training and variable-length inference strategy.

[0108] For example, in the training phase, the embodiment of the present disclosure may use an image with a resolution of 600*600. However, in the inference phase, the embodiment of the present disclosure may not resize the image, but directly input it to obtain a quality score. Therefore, the output score actually corresponds to the scoring situation at that resolution. For example, the same image may receive a rating of 4 points at a 480P resolution, but when it is resized to a 1080P resolution, the score may become 2.5 points. The quality assessment model based on this strategy can flexibly cope with images of various resolutions and provide a more accurate assessment of image quality at different resolutions.

[0109] It should be noted that, in the process of annotating the image to be evaluated, the embodiment of the present disclosure can give an overall quality score for the image to obtain an annotated data set. After training the quality assessment model using the annotated data set, the quality assessment model can be enabled to have the ability to output the overall quality score of the image.

[0110] Optionally, during the labeling of the image to be evaluated, embodiments of the present disclosure may assign different scores to different regions of the image to obtain a labeled dataset. For example, when labeling sample image A, it may correspond to three regions, where the first region may be scored as 5, the second region may be scored as 4, and the third region may be scored as 3. After training a quality assessment model using this labeled dataset, the quality assessment model can be enabled to output quality scores for different regions of the image.

[0111] Furthermore, after obtaining quality scores for different regions of the same image, embodiments of the present disclosure can display the quality scores for the different regions. Furthermore, embodiments of the present disclosure can enhance the image of a first region with a relatively low quality score, while leaving the second region with a relatively high quality score unchanged.

[0112] Optionally, to ensure balanced image display, the disclosed embodiment may also process the second region with a relatively high quality score to lower the quality score of the second region. Exemplarily, the disclosed embodiment may reduce the resolution of the second region, increase the noise in the second region, or blur the second region.

[0113] On this basis, the embodiment of the present disclosure can compare and display the image before and after processing, so that the user can clearly know which areas have lower quality scores, and which areas have lower quality scores, as well as what the image looks like after adjusting the area quality scores. This can improve the user experience to a certain extent.

[0114] Optionally, after obtaining quality scores for different regions of the same image, embodiments of the present disclosure may display the image of the first region on a first display and the image of the second region on a second display, wherein the image quality of the first display matches that of the first region, and the image quality of the second display matches that of the second region. In other words, regions of the same image with different quality scores may correspond to different displays. Higher quality scores may correspond to higher resolution displays.

[0115] In summary, obtaining image quality scores by utilizing a quality assessment model can enhance user experience, that is, the quality assessment model can accurately assess the quality of images uploaded by users. Based on this, the disclosed embodiments can provide real-time feedback and guidance for user operations. This interactive method can reduce problems caused by low quality of uploaded images, thereby greatly improving the user experience.

[0116] Furthermore, the disclosed embodiments can enhance product service accuracy. For services that require users to upload images or videos, the quality assessment model can evaluate the quality of the uploaded content in real time. If the uploaded image quality is found to be too low, the user can be promptly notified, avoiding potential service errors. This can substantially improve product service accuracy.

[0117] Furthermore, the disclosed embodiments can also improve business flexibility and ensure accuracy. Specifically, the quality assessment model is capable of processing images of any size, which makes the solution more flexible in adapting to images of various sizes. Thus, the disclosed embodiments can provide accurate evaluations for images of various scenarios.

[0118] Figure 13 An image quality assessment device is shown according to an exemplary embodiment. Figure 13 The image quality assessment apparatus 300 shown may include an acquisition module 310 , an input module 320 and an extraction module 330 .

[0119] The acquisition module 310 is configured to acquire an image to be evaluated; The input module 320 is configured to input the image to be evaluated into a quality assessment model, wherein the quality assessment model includes a feature extraction module and a fully connected module, wherein the fully connected module is used to change the size of the image to be evaluated; The processing module 330 is configured to perform feature extraction on the image to be evaluated based on the feature extraction module to obtain a feature extraction result, and process the feature extraction result using the fully connected module to obtain a quality score of the image to be evaluated.

[0120] In some embodiments, the fully connected module includes a global pooling layer, and the processing module 330 can be configured to pool each channel corresponding to the feature extraction result based on the global pooling layer to obtain a target feature map; and classify the target feature map to obtain the quality score.

[0121] In some embodiments, the fully connected module also includes a dilated spatial pyramid pooling module, and the processing module 330 can also be configured to use the dilated spatial pyramid pooling module to perform dilated convolution on the feature extraction result to obtain candidate feature maps of different scales, and perform depthwise separable convolution on the candidate feature maps of different scales to obtain a target feature map with the same size as the image; and classify the target feature map to obtain the quality score.

[0122] In some embodiments, the quality assessment model also includes a channel attention module, and the processing module 330 can also be configured to perform channel weighting on the feature extraction result based on the channel attention module to obtain a channel weighted result; and use the void space pyramid pooling module to perform void convolution on the channel weighted result to obtain candidate feature maps of different scales.

[0123] In some embodiments, the quality assessment model also includes a spatial attention module, and the processing module 330 can also be configured to perform spatial weighting on the feature extraction results based on the spatial attention module to obtain spatial weighted results; and use the void spatial pyramid pooling module to perform void convolution on the spatial weighted results to obtain candidate feature maps of different scales.

[0124] In some embodiments, the image quality assessment apparatus 300 may further include: A sample acquisition module is configured to acquire a sample data set, wherein the sample data set includes a plurality of sample images; a scoring module, configured to receive scoring information corresponding to each of the sample images and generate a label data set based on the scoring information; The training module is configured to train the quality assessment model using a training dataset consisting of the label dataset and the sample dataset.

[0125] In some embodiments, each of the sample images corresponds to multiple scoring information, and the scoring module can be configured to average the multiple scoring information and use the average result as the label information of the corresponding sample image; and form the label data set based on the label information of each sample image.

[0126] After acquiring the image to be evaluated, the embodiment of the present disclosure inputs the image to be evaluated into a quality assessment model. The quality assessment model may include a feature extraction module and a fully connected module, wherein the fully connected module can be used to change the size of the image to be evaluated. On this basis, feature extraction is performed on the image to be evaluated based on the feature extraction module to obtain a feature extraction result, and the feature extraction module is processed by the fully connected module to obtain a quality score of the image to be evaluated. By utilizing the quality assessment model, the embodiment of the present disclosure can not only change the size of the image to be evaluated, but also flexibly and effectively implement image evaluation, thereby improving the accuracy of image evaluation.

[0127] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0128] Figure 14 FIG. 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. Figure 14 As shown, the electronic device 700 may include: a processor 701 , a memory 702 , and may further include one or more of a multimedia component 703 , an input / output (I / O) interface 704 , and a communication component 705 .

[0129] The processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned image quality assessment method. The memory 702 is used to store various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, as well as application-related data such as contact information, sent and received messages, images, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0130] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned image quality assessment method.

[0131] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided. When executed by a processor, the program instructions implement the steps of the above-described image quality assessment method. For example, the computer-readable storage medium may be the aforementioned memory 702 including the program instructions. The program instructions may be executed by the processor 701 of the electronic device 700 to implement the above-described image quality assessment method.

[0132] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a processor, and when the computer program is executed by the processor, the steps of the above-mentioned image quality assessment method are implemented.

[0133] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above-mentioned image quality assessment method are implemented.

[0134] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a processor, and when the computer program is executed by the processor, the steps of the above-mentioned image quality assessment method are implemented.

[0135] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0136] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0137] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A method for evaluating image quality, characterized in that: The method comprises: Obtain the image to be evaluated; Inputting the image to be evaluated into a quality assessment model, wherein the quality assessment model includes a feature extraction module and a fully connected module, and the fully connected module is used to change the size of the image to be evaluated; Based on the feature extraction module, feature extraction is performed on the image to be evaluated to obtain a feature extraction result, and the feature extraction result is processed by the fully connected module to obtain a quality score of the image to be evaluated.

2. The method according to claim 1, characterized in that The fully connected module includes a global pooling layer, and the processing of the feature extraction result by using the fully connected module to obtain the quality score of the image to be evaluated includes: Pooling each channel corresponding to the feature extraction result based on the global pooling layer to obtain a target feature map; The target feature map is classified to obtain the quality score.

3. The method according to claim 1, characterized in that The fully connected module further includes a dilated spatial pyramid pooling module. Obtaining a quality score of the image to be evaluated by utilizing the fully connected module and the feature extraction result includes: Performing dilated convolution on the feature extraction results using the dilated spatial pyramid pooling module to obtain candidate feature maps of different scales, and performing depthwise separable convolution on the candidate feature maps of different scales to obtain a target feature map of the same size as the image; The target feature map is classified to obtain the quality score.

4. The method according to claim 3, characterized in that The quality assessment model further includes a channel attention module, which extracts features from the image to be assessed based on the feature extraction module, and after obtaining the feature extraction results, includes: Performing channel weighting on the feature extraction result based on the channel attention module to obtain a channel weighted result; The method of performing dilated convolution on the feature extraction result using the dilated spatial pyramid pooling module to obtain candidate feature maps of different scales includes: The dilated spatial pyramid pooling module is used to perform dilated convolution on the channel weighted results to obtain candidate feature maps of different scales.

5. The method according to claim 3, characterized in that The quality assessment model further includes a spatial attention module; after extracting features from the image to be assessed based on the feature extraction module and obtaining a feature extraction result, the method includes: Performing spatial weighting on the feature extraction result based on the spatial attention module to obtain a spatial weighted result; The method of performing dilated convolution on the feature extraction result using the dilated spatial pyramid pooling module to obtain candidate feature maps of different scales includes: The dilated spatial pyramid pooling module is used to perform dilated convolution on the spatial weighted result to obtain candidate feature maps of different scales.

6. The method according to claim 1, characterized in that The method further comprises: Acquire a sample data set, where the sample data set includes a plurality of sample images; receiving scoring information corresponding to each of the sample images, and generating a label data set based on the scoring information; The quality assessment model is trained using a training dataset consisting of the label dataset and the sample dataset.

7. The method according to claim 6, characterized in that Each of the sample images corresponds to a plurality of scoring information, and generating a label data set from the scoring information includes: Averaging the multiple scoring information and using the average result as the label information of the corresponding sample image; The label data set is composed according to the label information of each sample image.

8. An image quality assessment device, characterized in that: The device comprises: an acquisition module, configured to acquire an image to be evaluated; an input module configured to input the image to be evaluated into a quality assessment model, wherein the quality assessment model includes a feature extraction module and a fully connected module, and the fully connected module is used to change the size of the image to be evaluated; The processing module is configured to perform feature extraction on the image to be evaluated based on the feature extraction module to obtain a feature extraction result, and process the feature extraction result using the fully connected module to obtain a quality score of the image to be evaluated.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images