Image quality evaluation method, device, storage medium and electronic device

Abstract

The present disclosure provides an image quality assessment method, device, computer-readable storage medium and electronic device, which relate to the field of image and video processing technology. The image quality assessment method comprises: obtaining basic feature images of a target image at multiple scales; performing pooling processing on the basic feature images at each scale, and extracting local quality features at each scale according to the abstract feature images after pooling processing; performing convolution processing on the basic feature images at each scale to extract semantic features at each scale, and determining adaptive parameters at each scale according to the semantic features; using the adaptive parameters at each scale to fuse the local quality features at the same scale to obtain fused quality features at each scale; and determining the evaluation value of the target image based on the fused quality features at each scale. The present disclosure can improve the accuracy of image quality assessment.

Description

Background Art

[0002] Image Quality Assessment (IQA) refers to the evaluation of image quality by computer to simulate the human perception of image quality. Image quality assessment usually includes three types of tasks: full reference image quality assessment, reduced reference image quality assessment and no reference image quality assessment (or blind quality assessment). Among them, no reference image quality assessment refers to image quality assessment without the help of any reference image, which is the most difficult but very critical task in image quality assessment.

[0003] When performing no-reference image quality evaluation, related technologies have certain differences from human visual perception, resulting in low accuracy of the evaluation results.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those skilled in the art. Summary of the invention

[0005] The present disclosure provides an image quality evaluation method, an image quality evaluation device, a computer-readable storage medium and an electronic device, thereby at least to a certain extent solving the problem in the related art that images cannot be accurately evaluated.

[0006] According to a first aspect of the present disclosure, there is provided an image quality assessment method, comprising: obtaining basic feature images of a target image at multiple scales; performing pooling processing on the basic feature images at each scale, and extracting local quality features at each scale according to the abstract feature images after the pooling processing; performing convolution processing on the basic feature images at each scale to extract semantic features at each scale, and determining adaptive parameters at each scale according to the semantic features; fusing the local quality features at the same scale using the adaptive parameters at each scale respectively to obtain fused quality features at each scale; and determining an assessment value of the target image based on the fused quality features at each scale.

[0007] According to a second aspect of the present disclosure, an image quality evaluation device is provided, comprising: a basic feature image acquisition module, configured to acquire basic feature images of a target image at multiple scales; a local quality feature extraction module, configured to perform pooling processing on the basic feature images at each scale, and extract the local quality features at each scale according to the abstract feature images after the pooling processing; an adaptive parameter determination module, configured to perform convolution processing on the basic feature images at each scale to extract semantic features at each scale, and determine the adaptive parameters at each scale according to the semantic features; a local quality feature fusion module, configured to fuse the local quality features at the same scale using the adaptive parameters at each scale respectively, to obtain fused quality features at each scale; and a multi-scale fusion module, configured to determine the evaluation value of the target image based on the fused quality features at each scale.

[0008] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image quality assessment method of the first aspect and possible implementation methods thereof are implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image quality assessment method of the above-mentioned first aspect and its possible implementation method by executing the executable instructions.

[0010] The technical solution disclosed in this disclosure has the following beneficial effects:

[0011] First, the basic feature image is obtained first, which belongs to the recognition of the image, and then the local quality features are extracted and the adaptive parameters are determined according to the semantic features, which belongs to the perception of the image. Therefore, this scheme adopts the image quality evaluation method of first recognition and then perception, which is in line with the process of human visual perception. Second, by performing pooling processing on the basic feature image and further extracting local quality features based on the obtained abstract feature image, the local distortion in the target image can be more keenly discovered, and it is conducive to processing the uneven distortion in the target image. Third, by performing convolution processing on the basic feature image and further determining the adaptive parameters based on the obtained semantic features, the parameters can be adjusted to adapt to the changes in the image content, that is, the method or standard of image quality evaluation can be adaptively adjusted to imitate the characteristics of human perception of image quality changing with the image content. Fourth, local quality features are extracted and adaptive parameters are determined at multiple scales to ensure that distortions of different sizes in the target image are fully evaluated. Combining the above aspects, this scheme can improve the accuracy of image quality evaluation.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0014] Figure 1 A schematic diagram showing a system architecture in this exemplary embodiment;

[0015] Figure 2 A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown;

[0016] Figure 3 A flowchart showing an image quality evaluation method in this exemplary embodiment is shown;

[0017] Figure 4 A schematic diagram showing processing of a target image by a basic feature extraction network in this exemplary embodiment;

[0018] Figure 5 A flowchart of extracting local quality features in this exemplary embodiment is shown;

[0019] Figure 6 A schematic diagram showing processing of a basic feature image by a local quality feature extraction network in this exemplary embodiment;

[0020] Figure 7 A flow chart of determining adaptive weight parameters and adaptive bias parameters in this exemplary embodiment is shown;

[0021] Figure 8 A schematic diagram showing processing of a first basic feature image by a first adaptive weight determination network in this exemplary embodiment is shown;

[0022] Fig. 9 A flowchart of determining a global adaptive weight parameter and a global adaptive bias parameter in this exemplary embodiment is shown;

[0023] Fig.10 A schematic diagram showing processing of the nth basic feature image by a global adaptive weight determination network in this exemplary embodiment is shown;

[0024] Fig.11 A schematic diagram showing processing of a target image by an image quality assessment model in this exemplary embodiment;

[0025] Fig.12 A flowchart of training an image quality assessment model in this exemplary embodiment is shown;

[0026] Fig.13 A schematic diagram showing the structure of an image quality evaluation device in this exemplary embodiment is shown;

[0027] Fig.14 A schematic diagram showing verification of the results of image quality evaluation in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0029] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0030] Most related technologies use deep learning to evaluate image quality without reference, and have made some progress. However, there are still the following shortcomings:

[0031] Most deep learning models built for no-reference image quality assessment are based on the structure of the target detection network and focus on learning the global information of the image. However, the distortion of real images mostly exists in local areas. Especially when most areas of the image show good quality, the human visual system is very sensitive to local distortion. Therefore, the deep learning model ignores local information, resulting in a large difference between the image quality assessment results it outputs and human perception.

[0032] After training, the structure and parameters of a deep learning model are fixed, which means that the way to evaluate image quality is also fixed. However, the way humans perceive image quality changes with the content of the image. For example, people think that a blue sky image has high quality, but most deep learning models mistake it for a blurry image because the image contains a large open area. Therefore, the image quality evaluation method fixed by the deep learning model will affect the accuracy of the results.

[0033] In view of one or more of the above problems, an exemplary embodiment of the present disclosure first provides an image quality evaluation method. Figure 1 The system architecture and application scenarios of the operating environment of this exemplary embodiment are exemplarily described.

[0034] Figure 1 1 shows a schematic diagram of a system architecture, which may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, etc. The server 120 generally refers to a background system that provides image quality assessment related services in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to perform data exchange.

[0035] In one implementation, the image quality evaluation method may be performed by the terminal 110. For example, after a user uses the terminal 110 to shoot an image or the user selects an image from an album of the terminal 110, the terminal 110 evaluates the quality of the image and outputs an evaluation value.

[0036] In one implementation, the image quality evaluation method may be performed by the server 120. For example, after a user uses the terminal 110 to take a picture or the user selects an image from the photo album of the terminal 110, the terminal 110 uploads the image to the server 120, and the server 120 evaluates the quality of the image and returns the evaluation value to the terminal 110.

[0037] As can be seen from the above, the execution subject of the image quality evaluation method in this exemplary embodiment may be the terminal 110 or the server 120, and the present disclosure does not limit this.

[0038] The exemplary embodiment of the present disclosure also provides an electronic device for executing the above-mentioned image quality assessment method, and the electronic device may be the above-mentioned terminal 110 or the server 120. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned image quality assessment method by executing the executable instructions.

[0039] Below Figure 2 The mobile terminal 200 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The construction in can also be applied to fixed type equipment.

[0040] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 201, a memory 202, a bus 203, a mobile communication module 204, an antenna 1, a wireless communication module 205, an antenna 2, a display screen 206, a camera module 207, an audio module 208, a power module 209 and a sensor module 210.

[0041] The processor 201 may include one or more processing units, for example, the processor 210 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The image quality assessment method in this exemplary embodiment may be executed by an AP, a GPU or a DSP, and when the method involves processing related to a neural network, it may be executed by an NPU.

[0042] The encoder can encode (i.e. compress) an image or video, for example, the target image can be encoded into a specific format to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data, such as reading the encoded data of the target image, decoding it through the decoder to restore the data of the target image, and then perform relevant processing on the data for image quality evaluation. The mobile terminal 200 can support one or more encoders and decoders. In this way, the mobile terminal 200 can process images or videos in multiple encoding formats, such as: JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap) and other image formats, MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding) and other video formats.

[0043] The processor 201 may be connected to the memory 202 or other components via a bus 203 .

[0044] The memory 202 may be used to store computer executable program codes, which include instructions. The processor 201 executes various functional applications and data processing of the mobile terminal 200 by running the instructions stored in the memory 202. The memory 202 may also store application data, such as images, videos and other files.

[0045] The communication function of the mobile terminal 200 can be realized by the mobile communication module 204, the antenna 1, the wireless communication module 205, the antenna 2, the modulation and demodulation processor, and the baseband processor. The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 204 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 200. The wireless communication module 205 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 200.

[0046] The display screen 206 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 207 is used to implement shooting functions, such as shooting images, videos, etc. The audio module 208 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 209 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 210 may include a depth sensor 2101, a pressure sensor 2102, a gyroscope sensor 2103, an air pressure sensor 2104, etc., to implement corresponding sensing detection functions.

[0047] Combine the following Figure 3 The image quality evaluation method in this exemplary embodiment is described. Figure 3 An exemplary process of the image quality assessment method is shown, which may include:

[0048] Step S310, obtaining basic feature images of the target image at multiple scales;

[0049] Step S320, performing pooling processing on the basic feature image at each scale, and extracting the local quality features at each scale according to the abstract feature image after the pooling processing;

[0050] Step S330, performing convolution processing on the basic feature image at each scale to extract the semantic features at each scale, and determining the adaptive parameters at each scale according to the semantic features;

[0051] Step S340, using the adaptive parameters at each scale to fuse the local quality features at the same scale, to obtain a fused quality feature at each scale;

[0052] Step S350: determining an evaluation value of the target image based on the fusion quality features at each scale.

[0053] Based on the above method, on the one hand, the basic feature image is first obtained, which belongs to the recognition of the image, and then the local quality features are extracted and the adaptive parameters are determined according to the semantic features, which belongs to the perception of the image, so that the scheme adopts the image quality evaluation method of first recognition and then perception, which is in line with the process of human visual perception. On the other hand, by performing pooling processing on the basic feature image and further extracting local quality features based on the obtained abstract feature image, the local distortion in the target image can be more keenly discovered, and it is conducive to processing the uneven distortion in the target image. On the third hand, by performing convolution processing on the basic feature image and further determining the adaptive parameters based on the obtained semantic features, the parameters can be adjusted to adapt to the changes in the image content, that is, the method or standard of image quality evaluation can be adaptively adjusted to imitate the characteristics of human perception of image quality changing with the image content. On the fourth hand, local quality features are extracted and adaptive parameters are determined at multiple scales to ensure that the distortions of different sizes in the target image are fully evaluated. Combining the above aspects, the scheme can improve the accuracy of image quality evaluation.

[0054] Below Figure 3 Each step in the procedure is described in detail.

[0055] refer to Figure 3 , in step S310, basic feature images of the target image at multiple scales are obtained.

[0056] The target image is an image that needs to be evaluated for quality. The present disclosure does not limit the source of the target image. For example, the target image can be a currently captured image or any image selected by the user.

[0057] In step S310, preliminary feature extraction may be performed on the target image at different scales. The features extracted at this stage are generally relatively basic and low-level features, so the obtained feature image is called a basic feature image.

[0058] The present disclosure does not limit the specific method for extracting the basic feature image. For example, different types of feature descriptors can be used to extract the basic feature image from the target image. In one embodiment, the processing of step S310 can be implemented through a specific network. Specifically, the above multiple scales are represented as the first scale to the nth scale, and n is a positive integer not less than 2. The image quality evaluation method can also include the following steps:

[0059] A basic feature extraction network is obtained. The basic feature extraction network includes n convolutional layer combinations, wherein the first convolutional layer combination to the nth convolutional layer combination correspond to the first scale to the nth scale respectively.

[0060] Accordingly, the above-mentioned acquisition of basic feature images of the target image at multiple scales may include the following steps:

[0061] The target image is input into the basic feature extraction network, and the first basic feature image to the nth basic feature image are output respectively through the first convolution layer combination to the nth convolution layer combination.

[0062] Among them, the first basic feature image to the nth basic feature image are respectively the basic feature image at the first scale to the basic feature image at the nth scale. The basic feature extraction network is used to extract the basic feature image, and the main structure is a combination of n convolutional layers. A convolutional layer combination refers to a multi-layer structure including multiple convolutional layers. The convolutional layer combination may also include other types of intermediate layers, such as pooling layers, upsampling layers, etc. The n convolutional layer combinations may be serially connected or parallelly connected. When the n convolutional layer combinations are serially connected, each convolutional layer combination can be used to process the intermediate data output by the previous convolutional layer combination. When the n convolutional layer combinations are parallelly connected, each convolutional layer combination can be used to process the target image or the same intermediate data. In addition to the convolutional layer combination, the basic feature extraction network may also include other intermediate layers, such as a single convolutional layer, a pooling layer, an upsampling layer, etc.

[0063] Figure 4 The schematic structure of the basic feature extraction network is shown, including a convolutional layer and four convolutional layer combinations. Each convolutional layer combination includes multiple convolutional layers. The convolution parameters are shown in each convolutional layer, including the size of the convolution kernel, the number of channels and the step size. Figure 4 In the figure, some convolutional layers are folded to ×2, ×3, and ×5. For example, there are two structures in the first convolutional layer combination (convolution 1*1*64 + convolution 3*3*64 + convolution 1*1*512). In addition, in any convolutional layer, corresponding operations such as BN (Batch Normalization) and ReLU activation (Rectified Linear Unit, linear correction unit, in addition to ReLU, other activation functions can also be used) can be performed after convolution.

[0064] For example, assuming that the size of the target image is 224*224, it is input into the basic feature extraction network, and the size and dimension (dimension is the number of channels of the image) of the basic feature image obtained through each part are shown in Table 1. In Table 1, the convolution parameter S1 indicates that the step size is 1, and S2 indicates that the step size is 2. After the target image passes through the first convolution layer, the size of the feature image (Featuremap) obtained is 112*112*64; the feature image continues to be input into the first convolution layer combination, and the first basic feature image is output, and its size is 56*56*256; the first basic feature image continues to be input into the second convolution layer combination, and the second basic feature image is output, and its size is 28*28*512; the second basic feature image is input into the third convolution layer combination, and the third basic feature image is output, and its size is 14*14*1024; the third basic feature image continues to be input into the fourth convolution layer combination, and the fourth basic feature image is output, and its size is 7*7*2048. From the first basic feature image to the fourth basic feature image, the basic features of the target image at a microscopic scale, as well as the basic features at increasingly macroscopic scales, are reflected.

[0065] Table 1

[0066]

[0067]

[0068] In one implementation, a residual block structure may be added to the basic feature extraction network. For example, a residual block direct connection structure may be added between different convolutional layers of each convolutional layer combination to enrich the information of the basic feature image.

[0069] Continue to refer Figure 3 In step S320, the basic feature image at each scale is pooled, and the local quality features at each scale are extracted based on the abstract feature image after the pooling process.

[0070] The local quality feature is a feature that characterizes the quality information within the local range of the image. Pooling processing generally uses a pooling window of a certain size to traverse the image and abstractly represent the image information within the pooling window. In this exemplary embodiment, the basic feature image is pooled to obtain an abstract feature image, and the abstract feature image is further feature extracted to obtain the local quality feature.

[0071] Before the basic feature image is pooled, certain preprocessing may be performed, including but not limited to dimensionality reduction, cropping, reconstruction, etc. For the abstract feature image after pooling, local quality features may be further extracted through global pooling, flattening, full connection, and other operations.

[0072] In one implementation, the processing of step S320 may be implemented through a specific network. Specifically, the image quality evaluation method may further include the following steps:

[0073] Obtain a first local quality feature extraction network to an nth local quality feature extraction network, wherein each local quality feature extraction network includes a pooling layer and a fully connected layer.

[0074] Accordingly, reference Figure 5 As shown, the above-mentioned pooling process is performed on the basic feature image at each scale, and the local quality features at each scale are extracted according to the abstract feature image after the pooling process, which may include the following steps S510 to S530:

[0075] Step S510, inputting the i-th basic feature image into the i-th local quality feature extraction network;

[0076] Step S520, performing pooling processing on the i-th basic feature image through the pooling layer of the i-th local quality feature extraction network to obtain the i-th abstract feature image, where the i-th abstract feature image is an abstract feature image at the i-th scale;

[0077] Step S530, processing the i-th abstract feature image through the fully connected layer of the i-th local quality feature extraction network to obtain the i-th local quality feature, where the i-th local quality feature is a local quality feature at the i-th scale.

[0078] Wherein, i is any positive integer in [1, n], and the i-th basic feature image represents each basic feature image from the first basic feature image to the n-th basic feature image, that is, for each basic feature image, it can be used Figure 5 The process is processed to obtain the local quality characteristics at the corresponding scale.

[0079] The structures of the first local quality feature extraction network to the nth local quality feature extraction network can be the same. Since the basic feature images processed by them are different, the network parameters (herein, the network parameters are hyperparameters) can be different. The first local quality feature extraction network to the nth local quality feature extraction network can be respectively connected to the back end of the first convolutional layer combination to the nth convolutional layer combination in the above-mentioned basic feature extraction network, that is, the first basic feature image output by the first convolutional layer combination is input into the first local quality extraction network, and the second basic feature image output by the second convolutional layer combination is input into the second local quality extraction network, and so on.

[0080] Each local quality feature extraction network includes a pooling layer and a fully connected layer for performing pooling and fully connected processing. Among them, the pooling process can extract abstract features, and the fully connected process can fuse and further learn the abstract features to obtain local quality features. In addition, other types of intermediate layers can be set in the local quality feature extraction network, such as a dimensionality reduction layer, a cropping layer, a reconstruction layer, etc. for preprocessing the basic feature image. Exemplarily, each local quality feature extraction network can also include a dimensionality reduction layer; the above-mentioned pooling process of the basic feature image at each scale, and extracting the local quality features at each scale according to the abstract feature image after the pooling process, can also include the following steps:

[0081] The dimensionality reduction process of the i-th basic feature image is performed through the dimensionality reduction layer of the i-th local quality feature extraction network.

[0082] Accordingly, the above-mentioned pooling process of the i-th basic feature image through the pooling layer of the i-th local quality feature extraction network to obtain the i-th abstract feature image may include the following steps:

[0083] The i-th basic feature image after dimensionality reduction is pooled through the pooling layer of the i-th local quality feature extraction network to obtain the i-th abstract feature image.

[0084] Among them, the dimension reduction layer can be 1*1*m i (i=1,2,…,n) convolution to achieve, m i The number of convolution channels of the dimension reduction layer of the i-th local quality feature extraction network corresponds to the dimension of the input basic feature image. As shown in Table 1, considering that a deeper network structure is conducive to extracting abstract features, m i The value of increases gradually as the dimension of the basic feature image deepens, such as: m1=28, m2=56, m3=112, m4=224.

[0085] When performing dimensionality reduction, the features of the basic feature image at different dimensions are actually fused, which is conducive to extracting abstract features and reducing the computational complexity of subsequent processing.

[0086] The abstract feature image after pooling usually needs to be transformed in dimension before full connection processing. Generally, the abstract feature image can be stretched into a one-dimensional array through a stretching layer, or the abstract feature image of each channel can be converted into one-dimensional data through a global pooling layer, thereby obtaining a one-dimensional array composed of each channel. Then, the one-dimensional array is fully connected.

[0087] Figure 6The schematic diagram of processing the basic feature image through the local quality feature extraction network is shown. Each local quality feature extraction network includes a dimensionality reduction layer, a pooling layer, a stretching layer and a fully connected layer. The data dimension information output by each part can be referred to as shown in Table 2. For example, the first basic feature image of 56*56*256 is input into the first local quality feature extraction network, and the dimensionality is reduced by a 1*1*28 convolution kernel in the dimensionality reduction layer to obtain a 56*56*28 first basic feature image after dimensionality reduction; then enter the pooling layer, perform 7*7 (the step size is also 7) average pooling processing, and obtain the abstract features of each local area, that is, the first abstract feature image, whose dimension is 8*8*28; then stretch it into a one-dimensional array through the stretching layer, and its dimension is 1792 (=1*8*8*28); then output the first local quality feature of 28 dimensions through the fully connected layer, usually in the form of a vector. The processing flow for the second, third, and fourth basic feature images is the same, and the relevant data dimension information can be referred to as shown in Table 2, so it will not be repeated.

[0088] Table 2

[0089]

[0090]

[0091] Continue to refer Figure 3 In step S330, convolution processing is performed on the basic feature image at each scale to extract the semantic features at each scale, and the adaptive parameters at each scale are determined according to the semantic features.

[0092] Adaptive parameters refer to parameters for fusing local quality features. In this exemplary embodiment, the parameters need to adapt to the basic feature image (which can also be understood as adapting to the target image or the above-mentioned local quality features), that is, they vary with different basic feature images, so they are called adaptive parameters. By performing convolution processing on the basic feature image and extracting semantic features, the computer can adapt to different evaluation methods or evaluation standards of different image contents by humans on the basis of understanding the semantics of the image, and determine the adaptive parameters. The adaptive parameters reflect the changes in the evaluation methods or evaluation standards.

[0093] The adaptive parameter may include an adaptive weight parameter (weight). In one embodiment, the processing of step S330 may be implemented through a specific network. Specifically, the image quality evaluation method may further include the following steps:

[0094] Obtain the first adaptive parameter determination network to the nth adaptive parameter determination network, wherein each adaptive parameter determination network includes a semantic feature extraction subnetwork and a weight parameter determination subnetwork, the semantic feature extraction subnetwork includes a convolutional layer, and the weight parameter determination subnetwork includes a reconstruction layer.

[0095] Accordingly, reference Figure 7 As shown, the above-mentioned convolution processing is performed on the basic feature image at each scale to extract the semantic features at each scale, and the adaptive parameters at each scale are determined according to the semantic features, which may include the following steps S710 to S730:

[0096] Step S710, inputting the i-th basic feature image into the i-th adaptive parameter determination network;

[0097] Step S720, performing convolution processing on the i-th basic feature image through the i-th semantic feature extraction subnetwork to extract the i-th semantic feature, the i-th semantic feature extraction subnetwork is the semantic feature extraction subnetwork of the i-th adaptive parameter determination network, and the i-th semantic feature is the semantic feature at the i-th scale;

[0098] Step S730, reconstructing the data of the i-th semantic feature through the i-th weight parameter determination subnetwork to obtain the i-th adaptive weight parameter, the i-th weight parameter determination subnetwork is the weight parameter determination subnetwork of the i-th adaptive parameter determination network, and the i-th adaptive weight parameter is the adaptive weight parameter at the i-th scale.

[0099] Wherein, i is any positive integer in [1, n], and the i-th basic feature image represents each basic feature image from the first basic feature image to the n-th basic feature image, that is, for each basic feature image, it can be used Figure 7 The process is processed to obtain the adaptive weight parameters under the corresponding scale.

[0100] The structures of the first adaptive parameter determination network to the nth adaptive parameter determination network can be the same, and the parameters of the network can be different because the basic feature images they process are different. The first adaptive parameter determination network to the nth adaptive parameter determination network can be respectively connected to the back end of the first convolution layer combination to the nth convolution layer combination in the above-mentioned basic feature extraction network, that is, the first basic feature image output by the first convolution layer combination is input into the first adaptive parameter determination network, the second basic feature image output by the second convolution layer combination is input into the second adaptive parameter determination network, and so on.

[0101] Each adaptive parameter determination network includes a semantic feature extraction subnetwork and a weight parameter determination subnetwork. The semantic feature extraction subnetwork includes a convolution layer for extracting semantic features by convolution processing the basic feature image; the weight parameter determination subnetwork includes a reconstruction layer for reshaping the semantic features. Generally, the specific form of the semantic feature can be a semantic feature image. When reconstructing the semantic feature image, the data in the semantic feature image can be rearranged according to the dimensional information of the local quality feature to obtain the adaptive weight parameter.

[0102] In addition, other types of intermediate layers may be set in the semantic feature extraction subnetwork or the weight parameter determination subnetwork. In one embodiment, the semantic feature extraction subnetwork may further include a pooling layer. Thus, the above-mentioned convolution processing of the i-th basic feature image by the i-th semantic feature extraction subnetwork to extract the i-th semantic feature may include the following steps:

[0103] The i-th basic feature image is convolved and pooled through the i-th semantic feature extraction subnetwork to obtain the i-th semantic feature.

[0104] The pooling layer in the semantic feature extraction subnetwork is usually located after the convolution layer. After the basic feature image is input into the semantic feature extraction subnetwork, it is first processed by convolution to extract semantic features, and then processed by pooling to further abstract the extracted semantic features to improve the generalization of the semantic features.

[0105] In one embodiment, the weight parameter determination subnetwork may further include a convolution layer. Thus, the above-mentioned data reconstruction of the i-th semantic feature through the i-th weight parameter determination subnetwork to obtain the i-th adaptive weight parameter may include the following steps:

[0106] The i-th weight parameter is used to determine the sub-network to perform convolution processing and data reconstruction on the i-th semantic feature to obtain the i-th adaptive weight parameter.

[0107] That is to say, before reconstructing the semantic features, convolution processing can be performed again to fuse the information of the semantic features in different dimensions, and the dimensions of the semantic features can be adjusted, such as increasing or decreasing the dimensions of the semantic features according to the dimensional information of the local quality features. Then, data reconstruction is performed again, which is conducive to further improving the accuracy of the obtained adaptive weight parameters.

[0108] In one embodiment, the adaptive parameter may further include an adaptive bias parameter (bias). In order to obtain the adaptive bias parameter, a bias parameter determination subnetwork may be set in each adaptive parameter determination network, and the bias parameter determination subnetwork may include a fully connected layer. Generally, the weight parameter determination subnetwork and the bias parameter determination subnetwork are two branches connected to the semantic feature extraction subnetwork. Figure 7 As shown, the above-mentioned convolution processing is performed on the basic feature image at each scale to extract the semantic features at each scale, and the adaptive parameters at each scale are determined according to the semantic features, and the following steps S740 may also be included:

[0109] Step S740, the i-th semantic feature is fully connected through the i-th bias parameter determination subnetwork to obtain the i-th adaptive bias parameter, the i-th bias parameter determination subnetwork is the bias parameter determination subnetwork of the i-th adaptive parameter determination network, and the i-th adaptive bias parameter is the adaptive bias parameter at the i-th scale.

[0110] Among them, the fully connected layer in the bias parameter determination subnetwork is used to perform fully connected processing on the semantic features to obtain adaptive bias parameters.

[0111] Generally, the specific form of the semantic feature can be a semantic feature image, and before the full connection processing is performed, a dimensional transformation is usually required. For example, the bias parameter determination subnetwork may also include a global pooling layer for converting the semantic feature image of each channel into one-dimensional data, thereby obtaining a one-dimensional array composed of each channel. Alternatively, the bias parameter determination subnetwork may also include a stretching layer for stretching the semantic feature image into a one-dimensional array. After obtaining the one-dimensional array, the one-dimensional array is fully connected through the fully connected layer.

[0112] Figure 8A schematic diagram of processing the first basic feature image by the first adaptive parameter determination network is shown. The first adaptive parameter determination network includes a first semantic feature extraction subnetwork, a first weight parameter determination subnetwork, and a first bias parameter determination subnetwork. The first weight parameter determination subnetwork and the first bias parameter determination subnetwork are two branches connected to the first semantic feature extraction subnetwork. The data dimension information output by each part in the first adaptive parameter determination network can be referred to as shown in Table 3. The first basic feature image of 56*56*256 is input into the first adaptive parameter determination network, and first passes through two convolution layers in the first semantic feature extraction subnetwork. For example, both convolution layers can perform convolution and ReLU activation processing through a 3*3 convolution kernel; then passes through a pooling layer, for example, the pooling layer can use an 8*8 pooling window and a step size of 8 for pooling processing to extract the first semantic feature, whose dimension is 7*7*14; then the first semantic feature is respectively input into the first weight parameter determination subnetwork and the first bias parameter determination subnetwork, and convolution and data reconstruction processing are performed in the first weight parameter determination subnetwork to output the first adaptive weight parameter, whose dimension is 14*28*1*1; global pooling and full connection processing are performed in the first bias parameter determination subnetwork, and the first adaptive bias parameter is output, whose dimension is 14*1.

[0113] It should be understood that in other adaptive parameter determination networks, network parameters can be set according to actual conditions (the network parameters here are hyperparameters). For example, the pooling layer in the second adaptive parameter determination network can be set to a 4*4 pooling window and a step size of 4, the pooling layer in the third adaptive parameter determination network can be set to a 2*2 pooling window and a step size of 2, and the pooling layer in the fourth adaptive parameter determination network can be set to a 1*1 pooling window and a step size of 1 (equivalent to no pooling); the number of channels of the convolution kernel used in the convolution layer in the second, third, and fourth adaptive parameter determination networks may also be different from that in the first adaptive parameter determination network. The present disclosure does not limit this. The processing flow for the second, third, and fourth basic feature images is the same, and the relevant data dimension information can be referred to as shown in Table 3, so it will not be repeated.

[0114] Table 3

[0115]

[0116] Continue to refer Figure 3 In step S340, the local quality features at the same scale are fused using the adaptive parameters at each scale to obtain the fused quality features at each scale.

[0117] Exemplarily, the first local quality feature can be fully connected using the first adaptive weight parameter, which is equivalent to fusing the first local quality feature in different dimensions to obtain the fused quality feature at the first scale. The first local quality feature can also be fully connected using the first adaptive weight parameter and the first adaptive bias parameter to obtain the fused quality feature at the first scale.

[0118] In one embodiment, the image quality evaluation method may further include the following steps:

[0119] Get a fused network, which includes a fully connected layer.

[0120] Correspondingly, the above-mentioned method of fusing local quality features at the same scale by using adaptive parameters at each scale to obtain fused quality features at each scale may include the following steps:

[0121] The adaptive parameter at the i-th scale and the i-th local quality feature are input into the fusion network, and the i-th local quality feature is fully connected in the fusion network using the adaptive parameter to output the fusion quality feature at the i-th scale.

[0122] Among them, i is any positive integer in [1, n], that is, for the adaptive parameters and local quality features at each scale, the above steps can be used to process to obtain the fusion quality features at the corresponding scale.

[0123] It should be noted that the fusion network can be expressed as follows:

[0124] HM(L x ,W x )=q (1)

[0125] Where HM (Hyper Module) represents the fusion network, x represents the target image, and L x Represents the local quality characteristics, W x It represents the parameters of the fusion network, that is, the adaptive parameters, and q represents the fusion quality feature. It can be seen that Wx is related to x, which means that the parameters of the fusion network are not fixed and will change with different images, so that the output fusion quality features also adapt to the changes in image content, so as to adaptively evaluate the image quality according to the image content.

[0126] Continue to refer Figure 3 , in step S350, the evaluation value of the target image is determined based on the fusion quality features at each scale.

[0127] Generally, the fusion quality features at different scales can be further fused, so that the information of local distortions of different sizes in the target image can be integrated, and finally the evaluation value of the target image can be output.

[0128] In one embodiment, reference Fig. 9 As shown, the image quality evaluation method may further include the following steps S910 to S940:

[0129] Step S910, obtaining a global adaptive parameter determination network, the global adaptive parameter determination network includes a global semantic feature extraction subnetwork and a global weight parameter determination subnetwork, the global semantic feature extraction subnetwork includes a convolution layer and a global pooling layer, and the global weight parameter determination subnetwork includes a dimension adjustment layer;

[0130] Step S920, inputting the nth basic feature image into the global adaptive parameter determination network;

[0131] Step S930, performing convolution and global pooling processing on the nth basic feature image through the global semantic feature extraction sub-network to extract global semantic features;

[0132] Step S940, determining the sub-network through the global weight parameter to adjust the dimension of the global semantic feature to obtain the global adaptive weight parameter.

[0133] Accordingly, the above-mentioned determination of the evaluation value of the target image based on the fusion quality features at each scale may include the following steps:

[0134] The fusion quality features at each scale are fully connected using global adaptive weight parameters to obtain an evaluation value of the target image.

[0135] The global adaptive parameter is a parameter used to further fuse the fusion quality features at different scales, which may include a global adaptive weight parameter, which is a weight used to weight the fusion quality features at different scales.

[0136] The global adaptive parameter determination network is used to determine the global adaptive parameters, and its structure may be different from the above-mentioned first adaptive parameter determination network to the nth adaptive parameter determination network. Specifically, the global semantic feature extraction subnetwork needs to extract global semantic features, which are semantic features of a relatively macroscopic scale. A global pooling layer can be set therein to abstract the global semantic features, which can be a one-dimensional array. The dimensions of the global weight parameter determination subnetwork can be processed without convolution and data reconstruction. The global adaptive weight parameters can be obtained by adjusting the dimensions of the global semantic features, so that the dimensions of the global adaptive weight parameters correspond to the dimensions of the fusion quality features of each scale. Therefore, the global weight parameter determination subnetwork may include a dimensionality adjustment layer, which may be a dimensionality increase layer or a dimensionality reduction layer, and the present disclosure does not limit this.

[0137] By using the global adaptive weight parameters, the fusion quality features at each scale can be fully connected, which is equivalent to further fusing the fusion quality features at different scales. For example, the fusion quality features at the same scale can be fused into a value, which is the evaluation value of the target image.

[0138] In one implementation, the global adaptive parameter may also include a global adaptive bias parameter, which is a bias used when fusing fusion quality features at different scales. Accordingly, the global adaptive parameter determination network may also include a global bias parameter determination subnetwork, and the global bias parameter determination subnetwork includes a fully connected layer. Fig. 9 As shown, after extracting the global semantic features, the image quality evaluation method may further include the following steps S950:

[0139] Step S950, the sub-network is determined to fully connect the global semantic features through the global bias parameter to obtain a global adaptive bias parameter.

[0140] Correspondingly, the above-mentioned fully connected processing of the fusion quality features at each scale using the global adaptive weight parameter to obtain the evaluation value of the target image may include the following steps:

[0141] The fusion quality features at each scale are fully connected using global adaptive weight parameters and global adaptive bias parameters to obtain the evaluation value of the target image.

[0142] Among them, the global semantic feature can be a one-dimensional array. After the global bias parameter determines the fully connected layer in the sub-network, the global semantic features are fused and the global adaptive bias parameter is output, which can be one-dimensional data.

[0143] Fig.10 The schematic diagram of processing the nth basic feature image by the global adaptive parameter determination network is shown. The global adaptive parameter determination network includes a global semantic feature extraction subnetwork, a global weight parameter determination subnetwork, and a global bias parameter determination subnetwork. The global weight parameter determination subnetwork and the global bias parameter determination subnetwork are two branches connected to the global semantic feature extraction subnetwork. It can be seen that the global adaptive parameter determination network and Figure 8 The structure of the first adaptive parameter determination network in is very similar, except that there is an additional global pooling layer in the global semantic feature extraction subnetwork, the global weight parameter determination subnetwork is a 1*1 convolution layer (i.e., dimension adjustment layer), and the global bias parameter determination subnetwork does not have a global pooling layer. The data dimension information output by each part of the global adaptive parameter determination network can be referred to as shown in Table 4.

[0144] Table 4

[0145]

[0146]

[0147] In one implementation, the above-mentioned partial networks may form an image quality assessment model. Fig.11 The figure shows a schematic diagram of the image quality assessment model processing the target image. Among them, Backbone represents the basic feature extraction network, including the first convolution layer (Conv1) and four convolution layer combinations (Stage1~Stage4). LQE (Local Quality Extrac) represents the local quality extraction network. The image quality assessment model includes 4 local quality extraction networks, namely LQE1~LQE4. AWM (Adaptive Weight Module) represents the adaptive parameter determination network. The image quality assessment model includes 5 adaptive parameter determination networks, namely AWM1~AWM4 and AWMG (global adaptive parameter determination network). HM represents the fusion network. The fusion network is an algorithm module and does not include parameters.

[0148] Input the target image Fig.11 The image quality assessment model shown is first processed by the first convolution layer, and then passes through Stage 1 to Stage 4 in sequence to output the first basic feature image to the fourth basic feature image.

[0149] The first basic feature image is input into LQE1 and AWM1, LQE1 outputs the first local quality feature L1, AWM1 outputs the first adaptive weight parameter W1 and the first adaptive bias parameter B1; the second basic feature image is input into LQE2 and AWM2, LQE2 outputs the second local quality feature L2, AWM2 outputs the second adaptive weight parameter W2 and the second adaptive bias parameter B2; the third basic feature image is input into LQE3 and AWM3, LQE3 outputs the third local quality feature L3, AWM3 outputs the third adaptive weight parameter W3 and the third adaptive bias parameter B3; the fourth basic feature image is input into LQE4 and AWM4, LQE4 outputs the fourth local quality feature L4, AWM4 outputs the fourth adaptive weight parameter W4 and the fourth adaptive bias parameter B4; the fourth basic feature image is input into AWMG, AWMG outputs the global adaptive weight parameter WG and the global adaptive bias parameter BG.

[0150] L1, W1, and B1 are input into HM together, and HM uses W1 and B1 to fully connect L1 and output the fusion quality feature H1 at the first scale; L2, W2, and B2 are input into HM together, and HM uses W2 and B2 to fully connect L2 and output the fusion quality feature H2 at the second scale; L3, W3, and B3 are input into HM together, and HM uses W3 and B3 to fully connect L3 and output the fusion quality feature H3 at the third scale; L4, W4, and B4 are input into HM together, and HM uses W4 and B4 to fully connect L4 and output the fusion quality feature H4 at the fourth scale. When fusing local quality features at each scale, the algorithm of HM is as follows:

[0151]

[0152] After H1, H2, H3, and H4 are concatenated, they are input into HM together with WG and BG. HM uses W4 and B4 to fully connect L4 and outputs the evaluation value (Score) of the target image. When fusing local quality features at all scales, the algorithm of HM is as follows:

[0153] Score=WG*H′+BG (3)

[0154]

[0155] in, Can represent splicing.

[0156] In one embodiment, the image quality assessment method may further include a training process for an image quality assessment model, referring to Fig.12 As shown, the training process may include the following steps S1210 to S1230:

[0157] Step S1210, establishing an image quality assessment model, the image quality assessment model includes a basic feature extraction network to be trained, a first local quality feature extraction network to an nth local quality feature extraction network, a first adaptive parameter determination network to an nth adaptive parameter determination network and a global adaptive parameter determination network.

[0158] The image quality assessment model established at this time is the initial model, and the parameters therein can be initialized in any way, such as random initialization, or the basic feature extraction network adopts the ImageNet pre-training model, and the other parts use the Xavier initialization method.

[0159] Step S1220: Acquire a data set, where the data set includes sample images and evaluation value labels of the sample images.

[0160] The sample image may be a real distorted image. For example, the KonIQ-10k dataset may be used, which includes 10,073 real distorted sample images, and the image content and distortion have a high diversity. The evaluation value label of the sample image may be an evaluation value obtained by manual evaluation. In the KonIQ-10k dataset, each sample image is evaluated by at least 120 people, and the evaluation value is between [1,100]. The evaluation value label of the sample image is obtained by taking the average value.

[0161] After obtaining the data set, it can be divided into a training set and a test set. For example, it can be randomly divided according to a ratio of 8:2. The obtained training set includes 8058 sample images and the test set includes 2015 sample images.

[0162] In one embodiment, any one or more of the following preprocessing may be performed on the sample image:

[0163] ① Set a certain probability (such as 0.5), and use this probability to perform affine transformation on the sample image, including but not limited to horizontal flipping, vertical flipping, translation, rotation, scaling, etc., to obtain a new sample image after affine transformation.

[0164] ② Unify the sample images to a predetermined resolution, such as 512*384.

[0165] ③ Randomly capture a certain number (e.g., 25) of local images from the sample image. The size of the local images can be the same as the input image size of the image quality assessment model, such as 224*224. The captured local images are used as the real input for the subsequent training of the image quality assessment model.

[0166] ④ Normalize the pixel values ​​of the above local image to [0,1.0].

[0167] Step S1230, setting a first learning rate for the basic feature extraction network, setting a second learning rate for the first local quality feature extraction network to the nth local quality feature extraction network, the first adaptive parameter determination network to the nth adaptive parameter determination network and the global adaptive parameter determination network, and using the data set to train the image quality assessment model.

[0168] Among them, both the first learning rate and the second learning rate can be variable learning rates, such as gradually decreasing as the epoch increases. The first learning rate can be regarded as the basic learning rate of the image quality assessment model, and the second learning rate is a special learning rate set to accelerate the convergence of the local quality feature extraction network and the adaptive parameter determination network. In the first z epochs of training, the second learning rate is greater than the first learning rate, and after the first z epochs of training, the second learning rate is equal to the first learning rate, and z can be any positive integer. Exemplarily, taking z=8 as an example, the relationship between the first learning rate and the second learning rate can be as follows:

[0169]

[0170]

[0171] Among them, Lr1 represents the first learning rate, Lr2 represents the second learning rate, and epoch represents the number of "epochs" of the current training. In the first 5 epochs, Lr1 is the initial learning rate, which is set to 2×10 -5 , starting from the 6th epoch, the learning rate decreases by 10 times with each additional epoch. In the first 8 epochs, Lr2 is 10 times that of Lr1, and starting from the 9th epoch, Lr2 is the same as Lr1.

[0172] In model training, any form of loss function can be used, such as L1 loss, as shown below:

[0173]

[0174] Score j is the evaluation value of sample image j output by the image quality evaluation model, Q j is the evaluation value label of sample image j. E represents the number of sample images used in one training cycle.

[0175] For example, the weight decay is 5×10 -4 The Adam optimizer was used to train the image quality assessment model for 15 epochs with a training batch size of 96. The first and second learning rates were set as above, and the model parameters were updated through back propagation of the loss function value.

[0176] In one embodiment, during the testing phase, 5 local images with a resolution of 224×224 pixels can be randomly captured from the sample images in the test set and input into the image quality evaluation model to obtain an evaluation value for each local image. The evaluation values ​​of the 5 local images are averaged to obtain an evaluation value for the sample image.

[0177] In one implementation, before step S310, the following steps may be performed:

[0178] Obtain the image to be evaluated;

[0179] A plurality of local images of preset sizes are captured from the image to be evaluated as target images.

[0180] After determining the evaluation value of the target image, the image quality evaluation method may further include the following steps:

[0181] The evaluation value of each target image is combined to obtain the evaluation value of the image to be evaluated.

[0182] The image to be evaluated may be the original image to be evaluated, and the same preprocessing method as the sample image may be used to process it. For example, the image to be evaluated may be adjusted to a predetermined resolution, such as 512*384, and the pixel value may be normalized to within [0,1.0], and then t partial images may be randomly intercepted according to actual needs to serve as t target images.

[0183] For each target image, Figure 3 The method flow is processed to obtain an evaluation value. Thus, t evaluation values ​​of t target images are obtained, and then the t evaluation values ​​are integrated, such as calculating an average value or a weighted average value, to obtain the evaluation value of the image to be evaluated. Exemplarily, the average value of the t evaluation values ​​can be calculated as follows

[0184]

[0185] Among them, S0 is the evaluation value of the image to be evaluated, S k is the evaluation value of the target image (partial image) k.

[0186] By intercepting multiple local images to process the image to be evaluated, the input image size of the image quality evaluation model can be adapted, and more attention can be paid to discovering local distortion in the image, thereby further improving the accuracy of image quality evaluation.

[0187] The exemplary embodiment of the present disclosure also provides an image quality evaluation device. Fig.13 As shown, the image quality assessment device 1300 may include:

[0188] A basic feature image acquisition module 1310 is configured to acquire basic feature images of a target image at multiple scales;

[0189] The local quality feature extraction module 1320 is configured to perform pooling processing on the basic feature image at each scale, and extract the local quality feature at each scale according to the abstract feature image after the pooling processing;

[0190] The adaptive parameter determination module 1330 is configured to perform convolution processing on the basic feature image at each scale to extract the semantic features at each scale, and determine the adaptive parameters at each scale according to the semantic features;

[0191] The local quality feature fusion module 1340 is configured to fuse the local quality features at the same scale using the adaptive parameters at each scale to obtain a fused quality feature at each scale;

[0192] The multi-scale fusion module 1350 is configured to determine an evaluation value of the target image based on the fusion quality features at each scale.

[0193] In one implementation, the basic feature image acquisition module 1310 is further configured to:

[0194] Obtain the image to be evaluated;

[0195] Cutting out a plurality of local images of preset sizes from the image to be evaluated as target images;

[0196] The multi-scale fusion module 1350 is further configured to:

[0197] After determining the evaluation value of the target image, the evaluation value of each target image is integrated to obtain the evaluation value of the image to be evaluated.

[0198] In one implementation, the multiple scales include a first scale to an nth scale, where n is a positive integer not less than 2; the image quality assessment apparatus 1300 may further include a model acquisition module configured to:

[0199] Obtain a basic feature extraction network, the basic feature extraction network comprising n convolutional layer combinations, wherein the first convolutional layer combination to the nth convolutional layer combination correspond to the first scale to the nth scale respectively;

[0200] The basic feature image acquisition module 1310 is configured to:

[0201] The target image is input into the basic feature extraction network, and the first convolution layer combination to the nth convolution layer combination respectively output the first basic feature image to the nth basic feature image, and the first basic feature image to the nth basic feature image are respectively the basic feature images at the first scale to the basic feature images at the nth scale.

[0202] In one implementation, the model acquisition module is further configured to:

[0203] Obtaining a first local quality feature extraction network to an nth local quality feature extraction network, wherein each local quality feature extraction network includes a pooling layer and a fully connected layer;

[0204] The local quality feature extraction module 1320 is configured to:

[0205] Input the i-th basic feature image into the i-th local quality feature extraction network;

[0206] The i-th basic feature image is pooled by the pooling layer of the i-th local quality feature extraction network to obtain the i-th abstract feature image, where the i-th abstract feature image is an abstract feature image at the i-th scale;

[0207] The i-th abstract feature image is processed by the fully connected layer of the i-th local quality feature extraction network to obtain the i-th local quality feature, where the i-th local quality feature is a local quality feature at the i-th scale;

[0208] Wherein, i is any positive integer in [1, n], and the i-th basic feature image represents each basic feature image from the first basic feature image to the n-th basic feature image.

[0209] In one embodiment, the adaptive parameters include adaptive weight parameters.

[0210] The model acquisition module is also configured as:

[0211] Obtain the first adaptive parameter determination network to the nth adaptive parameter determination network, wherein each adaptive parameter determination network includes a semantic feature extraction subnetwork and a weight parameter determination subnetwork, the semantic feature extraction subnetwork includes a convolutional layer, and the weight parameter determination subnetwork includes a reconstruction layer.

[0212] The adaptive parameter determination module 1330 is configured to:

[0213] Inputting the i-th basic feature image into the i-th adaptive parameter determination network;

[0214] The i-th basic feature image is convolved by the i-th semantic feature extraction subnetwork to extract the i-th semantic feature, the i-th semantic feature extraction subnetwork is the semantic feature extraction subnetwork of the i-th adaptive parameter determination network, and the i-th semantic feature is the semantic feature at the i-th scale;

[0215] The i-th semantic feature is reconstructed through the i-th weight parameter determination subnetwork to obtain the i-th adaptive weight parameter. The i-th weight parameter determination subnetwork is the weight parameter determination subnetwork of the i-th adaptive parameter determination network. The i-th adaptive weight parameter is the adaptive weight parameter at the i-th scale.

[0216] In one embodiment, the adaptive parameter further includes an adaptive bias parameter. Each adaptive parameter determination network further includes a bias parameter determination subnetwork, and the bias parameter determination subnetwork includes a fully connected layer.

[0217] The adaptive parameter determination module 1330 is further configured to:

[0218] The i-th semantic feature is fully connected through the i-th bias parameter determination subnetwork to obtain the i-th adaptive bias parameter, the i-th bias parameter determination subnetwork is the bias parameter determination subnetwork of the i-th adaptive parameter determination network, and the i-th adaptive bias parameter is the adaptive bias parameter at the i-th scale.

[0219] In one implementation, the model acquisition module is further configured to:

[0220] Obtain a global adaptive parameter determination network, the global adaptive parameter determination network includes a global semantic feature extraction subnetwork and a global weight parameter determination subnetwork, the global semantic feature extraction subnetwork includes a convolution layer and a global pooling layer, and the global weight parameter determination subnetwork includes a dimension adjustment layer;

[0221] Input the nth basic feature image into the global adaptive parameter determination network;

[0222] The nth basic feature image is convolved and globally pooled through the global semantic feature extraction sub-network to extract global semantic features;

[0223] The sub-network is determined by the global weight parameter to adjust the dimension of the global semantic feature and obtain the global adaptive weight parameter;

[0224] The multi-scale fusion module 1350 is configured as follows:

[0225] The fusion quality features at each scale are fully connected using the global adaptive weight parameters to obtain the evaluation value of the target image.

[0226] In one embodiment, the global adaptive parameter determination network further includes a global bias parameter determination subnetwork, and the global bias parameter determination subnetwork includes a fully connected layer. The model acquisition module is further configured to:

[0227] After the nth basic feature image is input into the global adaptive parameter determination network, the global semantic features are fully connected through the global bias parameter determination subnetwork to obtain the global adaptive bias parameter;

[0228] The multi-scale fusion module 1350 is configured as follows:

[0229] The fusion quality features at each scale are fully connected using global adaptive weight parameters and global adaptive bias parameters to obtain the evaluation value of the target image.

[0230] In one implementation, the model acquisition module is further configured to:

[0231] Establishing an image quality assessment model, the image quality assessment model includes a basic feature extraction network to be trained, a first local quality feature extraction network to an nth local quality feature extraction network, a first adaptive parameter determination network to an nth adaptive parameter determination network and a global adaptive parameter determination network;

[0232] Obtain a data set, which includes sample images and evaluation value labels of the sample images;

[0233] A first learning rate is set for the basic feature extraction network, a second learning rate is set for the first local quality feature extraction network to the nth local quality feature extraction network, the first adaptive parameter determination network to the nth adaptive parameter determination network and the global adaptive parameter determination network, and an image quality assessment model is trained using the data set;

[0234] Among them, the second learning rate is greater than the first learning rate in the first z epochs of training, and the second learning rate is equal to the first learning rate after the first z epochs of training, and k is a positive integer.

[0235] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0236] In order to verify the effectiveness of the above method or device, four sample images are selected from the KonIQ-10k data set. Five 224*224 resolution partial images are randomly intercepted from each sample image. The image quality evaluation method in this exemplary embodiment is used to output its evaluation value, and the average evaluation value of the five partial images is calculated as the evaluation value of the sample image. The verification results are shown in Figure 2. Fig.14 As shown, MOS represents the evaluation value label, and S represents the evaluation value obtained in this exemplary embodiment. The verification results of the 4 sample images are: (a) MOS = 77.384, S = 77.585. (b) MOS = 62.558, S = 62.101. (c) MOS = 53.073, S = 53.901. (d) MOS = 31.226, S = 32.939. The 4 sample images can represent images in different evaluation value intervals. It can be seen that for images in different evaluation value intervals, the evaluation values ​​obtained in this exemplary embodiment are relatively close to the evaluation value labels, and the image quality can be evaluated more accurately.

[0237] In addition, the image quality assessment method in this exemplary embodiment is compared with nine existing algorithms including BRISQUE (no-reference image spatial quality assessor), IL-NIQE (integrated local natural image quality assessment), HOSA (high-order statistical aggregation), BIECON (blind image assessor based on convolutional neural network), WaDIQaM (weighted average deep image quality metric), SFA (semantic feature aggregation), PQR (probability mass representation), DBCNN (deep linear convolutional neural network) and HyperNet (hypernetwork) on the KonIQ-10k dataset for no-reference image quality assessment effect. PLCC (PearsonCorrelation Coefficient) and SROCC (Spearman Rank OrderCorrelation Coefficient) are used to measure the performance of image quality assessment. PLCC is used to quantitatively measure the consistency between the quality score prediction result and the true result, and SROCC is used to quantitatively measure the ranking correlation between the quality score prediction result and the true result. The larger the value of PLCC or SROCC, the better the prediction performance of the image quality assessment method. The comparison between this exemplary embodiment and the other nine algorithms is shown in Table 5, and it can be seen that this exemplary embodiment has the best image quality evaluation performance.

[0238] Table 5

[0239] Algorithm Name PLCC SROCC BRISQUE 0.681 0.665 IL-NIQE 0.523 0.507 HOSA 0.694 0.671 BIECON 0.651 0.618 QUR 0.805 0.797 SFA 0.872 0.856 PQR 0.884 0.880 DBCNN 0.884 0.875 HyperNet 0.892 0.874 This exemplary embodiment 0.897 0.881

[0240] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In an optional embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0241] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0242] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0243] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0244] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0245] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0246] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0247] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for evaluating image quality, characterized in that: include: Obtain basic feature images of the target image at multiple scales; Performing pooling processing on the basic feature image at each scale, and extracting the local quality features at each scale according to the abstract feature image after the pooling processing; Performing convolution processing on the basic feature image at each scale to extract semantic features at each scale, and determining adaptive parameters at each scale according to the semantic features; The local quality features at the same scale are respectively fused using the adaptive parameters at each scale to obtain fused quality features at each scale; Determining an evaluation value of the target image based on the fusion quality features at each scale; The multiple scales include a first scale to an nth scale, where n is a positive integer not less than 2; and the method further includes: Acquire a basic feature extraction network, wherein the basic feature extraction network includes n convolutional layer combinations, wherein the first convolutional layer combination to the nth convolutional layer combination correspond to the first scale to the nth scale respectively; The step of obtaining basic feature images of the target image at multiple scales includes: Input the target image into the basic feature extraction network, and output a first basic feature image to an nth basic feature image through the first convolution layer combination to the nth convolution layer combination, respectively, where the first basic feature image to the nth basic feature image are respectively a basic feature image at the first scale to a basic feature image at the nth scale; The adaptive parameters include adaptive weight parameters; the method further includes: Acquire a first adaptive parameter determination network to an nth adaptive parameter determination network, wherein each adaptive parameter determination network includes a semantic feature extraction subnetwork and a weight parameter determination subnetwork, the semantic feature extraction subnetwork includes a convolution layer, and the weight parameter determination subnetwork includes a reconstruction layer; The convolution processing is performed on the basic feature image at each scale to extract the semantic features at each scale, and the adaptive parameters at each scale are determined according to the semantic features, including: Inputting the i-th basic feature image into the i-th adaptive parameter determination network; Performing convolution processing on the i-th basic feature image through an i-th semantic feature extraction subnetwork to extract an i-th semantic feature, wherein the i-th semantic feature extraction subnetwork is a semantic feature extraction subnetwork of the i-th adaptive parameter determination network, and the i-th semantic feature is a semantic feature at an i-th scale; The i-th semantic feature is reconstructed through the i-th weight parameter determination subnetwork to obtain the i-th adaptive weight parameter, the i-th weight parameter determination subnetwork is the weight parameter determination subnetwork of the i-th adaptive parameter determination network, and the i-th adaptive weight parameter is the adaptive weight parameter at the i-th scale.

2. The method according to claim 1, characterized in that The method further comprises: Obtain the image to be evaluated; Cutting out a plurality of local images of preset sizes from the image to be evaluated as the target image; After determining the evaluation value of the target image, the method further includes: The evaluation value of each target image is combined to obtain the evaluation value of the image to be evaluated.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining a first local quality feature extraction network to an nth local quality feature extraction network, wherein each local quality feature extraction network includes a pooling layer and a fully connected layer; The pooling process is performed on the basic feature image at each scale, and the local quality feature at each scale is extracted according to the abstract feature image after the pooling process, including: Input the i-th basic feature image into the i-th local quality feature extraction network; Performing pooling processing on the i-th basic feature image through the pooling layer of the i-th local quality feature extraction network to obtain an i-th abstract feature image, wherein the i-th abstract feature image is an abstract feature image at an i-th scale; The i-th abstract feature image is processed by a fully connected layer of the i-th local quality feature extraction network to obtain an i-th local quality feature, where the i-th local quality feature is a local quality feature at an i-th scale; Wherein, i is any positive integer in [1, n], and the i-th basic feature image represents each basic feature image from the first basic feature image to the n-th basic feature image.

4. The method according to claim 1, characterized in that: The adaptive parameters also include adaptive bias parameters; each adaptive parameter determination network also includes a bias parameter determination subnetwork, and the bias parameter determination subnetwork includes a fully connected layer; The method further comprises: performing convolution processing on the basic feature image at each scale to extract the semantic features at each scale, and determining the adaptive parameters at each scale according to the semantic features. The i-th semantic feature is fully connected through the i-th bias parameter determination subnetwork to obtain the i-th adaptive bias parameter, the i-th bias parameter determination subnetwork is the bias parameter determination subnetwork of the i-th adaptive parameter determination network, and the i-th adaptive bias parameter is the adaptive bias parameter at the i-th scale.

5. The method according to claim 3, characterized in that: The method further comprises: Acquire a global adaptive parameter determination network, wherein the global adaptive parameter determination network includes a global semantic feature extraction subnetwork and a global weight parameter determination subnetwork, wherein the global semantic feature extraction subnetwork includes a convolution layer and a global pooling layer, and the global weight parameter determination subnetwork includes a dimension adjustment layer; Inputting the nth basic feature image into the global adaptive parameter determination network; Performing convolution and global pooling processing on the nth basic feature image through the global semantic feature extraction subnetwork to extract global semantic features; The sub-network is determined by the global weight parameter to adjust the dimension of the global semantic feature to obtain a global adaptive weight parameter; The step of determining the evaluation value of the target image based on the fusion quality feature at each scale includes: The global adaptive weight parameter is used to perform full connection processing on the fusion quality features at each scale to obtain an evaluation value of the target image.

6. The method according to claim 5, characterized in that The global adaptive parameter determination network further includes a global bias parameter determination subnetwork, and the global bias parameter determination subnetwork includes a fully connected layer; after the nth basic feature image is input into the global adaptive parameter determination network, the method further includes: The global semantic feature is fully connected by the sub-network determined by the global bias parameter to obtain a global adaptive bias parameter; The step of using the global adaptive weight parameter to perform full connection processing on the fusion quality features at each scale to obtain an evaluation value of the target image includes: The global adaptive weight parameter and the global adaptive bias parameter are used to perform full connection processing on the fusion quality features at each scale to obtain an evaluation value of the target image.

7. The method according to claim 5, characterized in that The method further comprises: Establishing an image quality assessment model, the image quality assessment model comprising the basic feature extraction network to be trained, the first local quality feature extraction network to the nth local quality feature extraction network, the first adaptive parameter determination network to the nth adaptive parameter determination network and the global adaptive parameter determination network; Acquire a data set, the data set including sample images and evaluation value labels of the sample images; Setting a first learning rate for the basic feature extraction network, setting a second learning rate for the first local quality feature extraction network to the nth local quality feature extraction network, the first adaptive parameter determination network to the nth adaptive parameter determination network and the global adaptive parameter determination network, and training the image quality assessment model using the data set; In which, in the first z epochs of training, the second learning rate is greater than the first learning rate, and after the first z epochs of training, the second learning rate is equal to the first learning rate, and z is a positive integer.

8. An image quality evaluation device, characterized in that: include: A basic feature image acquisition module is configured to acquire basic feature images of a target image at multiple scales; A local quality feature extraction module is configured to perform pooling processing on the basic feature image at each scale, and extract the local quality feature at each scale according to the abstract feature image after the pooling processing; An adaptive parameter determination module is configured to perform convolution processing on the basic feature image at each scale to extract semantic features at each scale, and determine the adaptive parameters at each scale according to the semantic features; A local quality feature fusion module is configured to fuse the local quality features at the same scale using the adaptive parameters at each scale to obtain a fused quality feature at each scale; A multi-scale fusion module, configured to determine an evaluation value of the target image based on the fusion quality feature at each scale; The multiple scales include a first scale to an nth scale, where n is a positive integer not less than 2; the device is further configured to: Acquire a basic feature extraction network, wherein the basic feature extraction network includes n convolutional layer combinations, wherein the first convolutional layer combination to the nth convolutional layer combination correspond to the first scale to the nth scale respectively; The method of obtaining basic feature images of a target image at multiple scales is configured as follows: Input the target image into the basic feature extraction network, and output a first basic feature image to an nth basic feature image through the first convolution layer combination to the nth convolution layer combination, respectively, where the first basic feature image to the nth basic feature image are respectively a basic feature image at the first scale to a basic feature image at the nth scale; The adaptive parameters include adaptive weight parameters; the device is also configured to: Acquire a first adaptive parameter determination network to an nth adaptive parameter determination network, wherein each adaptive parameter determination network includes a semantic feature extraction subnetwork and a weight parameter determination subnetwork, the semantic feature extraction subnetwork includes a convolution layer, and the weight parameter determination subnetwork includes a reconstruction layer; The convolution process is performed on the basic feature image at each scale to extract the semantic features at each scale, and the adaptive parameters at each scale are determined according to the semantic features, and are configured as follows: Inputting the i-th basic feature image into the i-th adaptive parameter determination network; Performing convolution processing on the i-th basic feature image through an i-th semantic feature extraction subnetwork to extract an i-th semantic feature, wherein the i-th semantic feature extraction subnetwork is a semantic feature extraction subnetwork of the i-th adaptive parameter determination network, and the i-th semantic feature is a semantic feature at an i-th scale; The i-th semantic feature is reconstructed through the i-th weight parameter determination subnetwork to obtain the i-th adaptive weight parameter, the i-th weight parameter determination subnetwork is the weight parameter determination subnetwork of the i-th adaptive parameter determination network, and the i-th adaptive weight parameter is the adaptive weight parameter at the i-th scale.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images