Method for generating an image quality evaluation model, image quality evaluation method and device

Abstract

This disclosure discloses a method, an image quality assessment method, and an apparatus for generating an image quality assessment model. The method for generating the image quality assessment model includes: constructing a training model for generating the image quality assessment model, the training model including a feature extraction component, a pooling component, and a score prediction component; inputting a training image into the training model for processing, including: extracting features from the training image using the feature extraction component to obtain a feature map; fusing the feature map using the pooling component to obtain a feature fusion map; processing the feature fusion map using the score prediction component to obtain a predicted score indicating the quality of the training image; adjusting the model parameters of the training model by calculating the loss value between the predicted score and the subjective score of the training image until training is complete, obtaining a trained training model; and performing an equivalent transformation on the trained training model to generate the image quality assessment model.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a method for generating image quality evaluation models, an image quality evaluation method, and an apparatus. Background Technology

[0002] As business demands for image quality and user experience continue to increase, research into image quality assessment methods is becoming increasingly important. Image quality assessment is mainly divided into objective assessment and subjective assessment. Objective assessment primarily uses algorithms to evaluate image quality, while subjective assessment relies on the human eye to directly provide a visual score, which is closest to the user's experience. However, in real-world scenarios, subjective assessment is extremely time-consuming and easily influenced by external factors such as display devices and the evaluator's subjective emotions. Therefore, the goal of image quality assessment methods is to design an objective assessment algorithm that can closely approximate the results of subjective assessments.

[0003] Image quality assessment methods are categorized into full-reference image quality assessment, half-reference image quality assessment, and no-reference image quality assessment based on the extent to which original image information is used. In real-world business scenarios, it is generally impossible to obtain an undistorted reference image, thus no-reference image quality assessment has attracted attention. However, due to the lack of available reference images and the richness of the image content, no-reference image quality assessment presents significant challenges.

[0004] Existing technologies often rely on traditional image processing methods for no-reference image quality assessment, primarily involving manually designing image quality features and then using a classifier to evaluate the image quality. However, using manually designed features as input to a no-reference image classification and regression model results in insufficient model generalization and inability to guarantee stability in scenarios with large data volumes; furthermore, there is the problem of error accumulation, leading to insufficient accuracy.

[0005] Therefore, a new image quality evaluation scheme is needed. Summary of the Invention

[0006] This disclosure provides an image quality assessment scheme in an attempt to solve or at least alleviate at least one of the problems mentioned above.

[0007] According to one aspect of this disclosure, a method for generating an image quality assessment model is provided, comprising: constructing a training model for generating the image quality assessment model, the training model including: a feature extraction component, a pooling component, and a score prediction component; inputting a training image into the training model for processing, including: extracting features from the training image using the feature extraction component to obtain a feature map; fusing the feature map using the pooling component to obtain a feature fusion map; processing the feature fusion map using the score prediction component to obtain a predicted score indicating the quality of the training image; adjusting the model parameters of the training model by calculating a loss value between the predicted score and the subjective score of the training image until training is completed, thereby obtaining a trained training model; and performing an equivalent transformation on the trained training model to generate the image quality assessment model.

[0008] Optionally, the method according to this disclosure further includes: performing an equivalent transformation on the feature extraction components in the trained model to obtain an equivalent feature extraction component.

[0009] Optionally, in the method according to this disclosure, the feature extraction component includes at least one feature extraction stage, and the feature extraction stage includes a different number of convolutional processing blocks, wherein the convolutional processing blocks include coupled multi-branch modules and attention modules, the multi-branch modules being adapted to extract features of the image, and the attention modules being adapted to enhance the features through the generated attention weights.

[0010] Optionally, in the method according to this disclosure, the pooling component includes three pooling layers and a feature fusion layer, wherein the pooling layers pool the input feature map to obtain a corresponding pooled feature map, and the feature fusion layer fuses each pooled feature map with the feature map to obtain a feature fusion map of a fixed size.

[0011] Optionally, in the method according to this disclosure, the score prediction component comprises four fully connected layers.

[0012] Optionally, in the method according to this disclosure, the multi-branch module includes a 3x3 convolution branch, a 1x1 convolution branch, and an identity mapping branch.

[0013] Optionally, in the method according to this disclosure, the step of performing an equivalent transformation on the feature extraction component in the trained model to obtain an equivalent feature extraction component includes: transforming a multi-branch module into a single-branch structure; and replacing the multi-branch module in the feature extraction component with the single-branch structure to obtain an equivalent feature extraction component.

[0014] Optionally, in the method according to this disclosure, the step of obtaining a single-branch structure based on the multi-branch module conversion includes: for each branch, combining the learned parameters, fusing convolutional layers and batch normalization layers to obtain fused convolutional layers; and stacking the fused convolutional layers of each branch to obtain a single-branch structure.

[0015] Optionally, in the method according to this disclosure, the attention module is also adapted to generate attention weights by calculating the reciprocal of the minimum energy.

[0016] Alternatively, in the method according to this disclosure, the loss value is calculated using the following formula:

[0017]

[0018] In the formula, q represents the predicted score output by the trained model. This indicates the subjective score.

[0019] According to another aspect of this disclosure, an image quality assessment method is provided, comprising: inputting an image to be evaluated into an image quality assessment model for processing to obtain an image quality assessment score, wherein the image quality assessment model is generated by performing the method described above.

[0020] According to another aspect of this disclosure, an image quality evaluation apparatus is provided, comprising: a model training unit adapted to construct a training model and generate an image quality evaluation model by training the training model; and an image quality evaluation unit adapted to process an image to be evaluated using the image quality evaluation model to obtain an image quality evaluation score.

[0021] According to another aspect of this disclosure, a computing device is provided, comprising: one or more processor memories; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described above.

[0022] According to another aspect of this disclosure, a computer-readable storage medium is provided for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0023] In summary, according to the scheme disclosed herein, when constructing the training model, the feature extraction component (especially the multi-branch module) uses a multi-branch network to learn image features during the training phase; during the prediction phase, the multi-branch network structure is converted into a single-branch network structure through structural reparameterization technology, thereby improving the model's inference speed, reducing the model's peak memory usage, and facilitating model deployment.

[0024] Furthermore, in the feature extraction component, the SimAttention module is used as the attention module, which improves the model's ability to extract effective features of image quality without increasing parameters or changing the module structure.

[0025] In addition, the use of pooled components solves the problem that the fully connected layer of the subsequent score prediction component requires a fixed input, making it more suitable for use in real business scenarios.

[0026] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description

[0027] To achieve the foregoing and related objectives, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings. These aspects indicate various ways in which the principles disclosed herein may be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The foregoing and other objectives, features, and advantages of this disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings. Throughout this disclosure, the same reference numerals generally refer to the same parts or elements.

[0028] Figure 1 A schematic diagram of an image quality evaluation apparatus 100 according to some embodiments of the present disclosure is shown;

[0029] Figure 2 A schematic diagram of a computing device 200 according to some embodiments of the present disclosure is shown;

[0030] Figure 3 A flowchart illustrating a method 300 for generating an image quality assessment model according to some embodiments of the present disclosure is shown;

[0031] Figure 4 A schematic block diagram of a training model 400 according to some embodiments of the present disclosure is shown;

[0032] Figure 5 A schematic diagram of the structure of a convolution processing block according to some embodiments of the present disclosure is shown;

[0033] Figure 6 A schematic diagram of the structure of an attention module according to some embodiments of the present disclosure is shown;

[0034] Figure 7 A schematic diagram of the structure of a pooling component according to some embodiments of the present disclosure is shown;

[0035] Figure 8 A schematic flowchart of an image quality evaluation method 800 according to some embodiments of the present disclosure is shown. Detailed Implementation

[0036] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0037] With the continuous advancement of computer vision methods, deep learning-based image quality assessment methods have emerged in existing technologies. For example, Chinese patent application CN113066065A proposes a no-reference image quality assessment model. This model first uses a feature extraction network to extract image quality features from the input image, and then uses a dual-channel convolutional computation network to output the corresponding image quality features. The dual-channel convolutional computation network includes dense Hadamard convolutions and multiple convolutional channels. The dense Hadamard convolutions output a first feature vector, and the multiple convolutional channels output a second feature vector. Finally, the first and second feature vectors are combined, and this fused feature vector is used to map the image quality score. This method is completely end-to-end and can improve the accuracy of image quality assessment compared to traditional methods. However, because this method introduces two feature extraction networks, the model complexity is high, the computational resource requirements are high, and it is not conducive to the deployment of the model.

[0038] To address the issues of insufficient real-time performance and low accuracy of existing no-reference image quality assessment models, this disclosure proposes an image quality assessment scheme.

[0039] Figure 1 A schematic diagram of an image quality assessment apparatus 100 according to some embodiments of the present disclosure is shown. The image quality assessment apparatus 100 includes a coupled model training unit 110 and an image quality assessment unit 120.

[0040] The model training unit 110 is used to construct a training model and generate an image quality assessment model by training the training model. According to some embodiments, the training model includes coupled feature extraction components, pooling components, and score prediction components. The structure of the training model will be described in detail below.

[0041] The image quality evaluation unit 120 is used to process the image to be evaluated using an image quality evaluation model to obtain an image quality evaluation score.

[0042] Furthermore, the model training unit 110 executes the method 300 for generating an image quality assessment model. First, it trains the training model to obtain a trained model; then, it performs an equivalent transformation on the trained model to generate the image quality assessment model. In some embodiments, the network parameters of each component in the trained model remain unchanged, and only a portion of the network structure in the feature extraction component is transformed to improve the model inference speed, reduce the model's peak memory usage, and facilitate model deployment.

[0043] Image quality evaluation unit 120 executes image quality evaluation method 800, which uses image quality evaluation model to process the image to be evaluated in order to obtain image quality evaluation score.

[0044] Regarding the image quality assessment device 100, please refer to the detailed descriptions below of the method 300 for generating the image quality assessment model and the image quality assessment method 800.

[0045] According to this disclosure, the image quality evaluation device 100 can be displayed by one or more computing devices.

[0046] Figure 2 A structural block diagram of a computing device 200 according to an embodiment of the present disclosure is shown.

[0047] like Figure 2 As shown, in basic configuration 202, computing device 200 typically includes system memory 206 and one or more processors 204. Memory bus 208 can be used for communication between processor 204 and system memory 206.

[0048] Depending on the desired configuration, processor 204 can be any type of processor, including but not limited to: microprocessors (μP), microcontrollers (μC), digital information processors (DSPs), or any combination thereof. Processor 204 may include one or more levels of cache such as L1 cache 210 and L2 cache 212, processor core 214, and registers 216. Example processor core 214 may include an arithmetic logic unit (ALU), a floating-point unit (FPU), a digital signal processing (DSP) core, or any combination thereof. Example memory controller 218 may be used with processor 204, or in some implementations, memory controller 218 may be an internal part of processor 204.

[0049] Depending on the desired configuration, system memory 206 can be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. Physical memory in a computing device typically refers to volatile RAM, and data on a disk needs to be loaded into physical memory before it can be read by processor 204. System memory 206 may include operating system 220, one or more applications 222, and program data 224. In some embodiments, applications 222 may be arranged to execute instructions on the operating system using program data 224 by one or more processors 204. Operating system 220 may be, for example, Linux, Windows, etc., and includes program instructions for handling basic system services and performing hardware-dependent tasks. Applications 222 include program instructions for implementing various user-desired functions, and applications 222 may be, for example, browsers, instant messaging software, software development tools (such as integrated development environments (IDEs), compilers, etc.), but are not limited to these.

[0050] When the computing device 200 starts up, the processor 204 reads the program instructions of the operating system 220 from the memory 206 and executes them. The application 222 runs on the operating system 220 and uses the interfaces provided by the operating system 220 and the underlying hardware to implement various functions expected by the user. When the user starts the application 222, the application 222 is loaded into the memory 206, and the processor 204 reads and executes the program instructions of the application 222 from the memory 206.

[0051] The computing device 200 also includes a storage device 232, which includes a removable storage device 236 (e.g., CD, DVD, USB flash drive, portable hard drive, etc.) and a non-removable storage device 238 (e.g., hard disk drive, HDD, etc.). Both the removable storage device 236 and the non-removable storage device 238 are connected to the storage interface bus 234.

[0052] The computing device 200 may also include a storage interface bus 234. The storage interface bus 234 enables communication from storage devices 232 (e.g., removable storage 236 and non-removable storage 238) to the basic configuration 202 via the bus / interface controller 230. At least a portion of the operating system 220, applications 222, and program data 224 may be stored on the removable storage 236 and / or the non-removable storage 238, and loaded into system memory 206 via the storage interface bus 234 when the computing device 200 is powered on or when the application 222 is to be executed, and executed by one or more processors 204.

[0053] The computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via a bus / interface controller 230. Example output devices 242 include a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices such as displays or speakers via one or more A / V ports 252. Example peripheral interfaces 244 may include a serial interface controller 254 and a parallel interface controller 256, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 258. Example communication devices 246 may include a network controller 260, which may be arranged to facilitate communication with one or more other computing devices 262 via a network communication link through one or more communication ports 264.

[0054] A network communication link can be an example of a communication medium. A communication medium can typically be embodied in a modulated data signal, such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A “modulated data signal” can be a signal whose data set, or its modifications, can be encoded as information within the signal. As a non-limiting example, a communication medium can include wired media such as wired networks or leased lines, and various wireless media including sound, radio frequency (RF), microwave, infrared (IR), or other wireless media. The term “computer-readable medium” as used herein can include both storage media and communication media.

[0055] The computing device 200 can be implemented as a personal computer, including desktop and laptop computer configurations. Of course, the computing device 200 can also be implemented as part of a small-sized portable (or mobile) electronic device, such as a cellular phone, digital camera, personal digital assistant (PDA), personal media player device, wireless network browsing device, personal head-mounted device, application-specific device, or a hybrid device that may include any of the above functions. It can even be implemented as a server, such as a file server, database server, application server, and web server. The embodiments of the present invention do not limit this.

[0056] In embodiments according to this disclosure, computing device 200 is configured to execute method 300 for generating an image quality assessment model according to this disclosure, and / or image quality assessment method 800. Application 222 arranged on an operating system includes multiple program instructions for executing one or more of the above methods, which can instruct processor 204 to execute the methods of this disclosure to perform text detection.

[0057] Figure 3 A flowchart illustrating a method 300 for generating an image quality assessment model according to an embodiment of the present disclosure is shown. In one embodiment, method 300 is performed using the aforementioned model training unit 110.

[0058] like Figure 3 As shown, method 300 begins with 310. In 310, a training model is constructed to generate the image quality assessment model.

[0059] According to embodiments of this disclosure, the training model is based on a convolutional neural network. In some embodiments, the training model includes a coupled feature extraction component, a pooling component, and a score prediction component. The feature extraction component extracts image features from the input image; the pooling component fuses the extracted features across different receptive fields; and the score prediction component typically employs a fully connected layer to predict image quality scores. In one embodiment, the pooling component can also adapt to inputs of various sizes and fuse them into a fixed-size feature fusion map to ensure that the image size input to the score prediction component remains consistent.

[0060] The training model will be described in more detail below.

[0061] Figure 4 A schematic block diagram of a training model 400 according to some embodiments of the present disclosure is shown. Figure 4 As shown, the training model 400 includes a feature extraction component 410, a pooling component 420, and a score prediction component 430 that are coupled in sequence.

[0062] According to some embodiments of this disclosure, the feature extraction component 410 includes at least one feature extraction stage, and each feature extraction stage includes a different number of convolutional processing blocks. In one embodiment, the feature extraction component 410 includes five feature extraction stages, denoted sequentially as stage1, stage2, stage3, stage4, and stage5, such as... Figure 4 As shown. In one embodiment, stage 1 includes one convolutional processing block, stage 2 includes two convolutional processing blocks, stage 3 includes four convolutional processing blocks, stage 4 includes fourteen convolutional processing blocks, and stage 5 includes one convolutional processing block. However, this is not the only possible implementation.

[0063] Figure 5 A schematic diagram of the structure of a convolution processing block (C) according to some embodiments of the present disclosure is shown.

[0064] When the feature extraction stage contains more than one convolutional processing block (C), the multiple convolutional processing blocks (C) are coupled sequentially, such as... Figure 5 Two convolutional processing blocks (C) are shown. Each convolutional processing block (C) includes a coupled multi-branch module (R) and an attention module (A). The multi-branch module (R) is mainly used to extract image features, while the attention module (A) enhances the features using the generated attention weights.

[0065] like Figure 5 As shown in (a), the multi-branch module (R) includes one 3x3 convolutional branch (g(x)), one 1x1 convolutional branch (f(x)), and one identity mapping branch (x), denoted as: y = x + f(x) + g(x). It should be understood that each convolutional branch contains not only convolutional layers but also BN (batch normalization) layers. Convolutional layers may also contain activation operations in addition to convolution operations; this disclosure does not impose excessive restrictions on these aspects. By employing a multi-branch network structure, the multi-branch module (R) can improve the accuracy of feature extraction, which is beneficial for improving the performance of the trained model in extracting image quality features.

[0066] Furthermore, to further improve the performance of the trained model, an attention module (A) is introduced in the convolution processing block (C). For example... Figure 5 As shown in (a), after fusing the features of the three branches, an attention module (A) is introduced, and the features are enhanced by the attention weights generated by the attention module (A).

[0067] like Figure 6 A schematic diagram of the structure of an attention module (A) according to some embodiments of the present disclosure is shown. The attention module (A) calculates 3D attention weights from the input feature map. In some embodiments, the attention module (A) employs a SimAttention module to generate attention weights by calculating the reciprocal of the minimum energy. The processing of the input by the attention module (A) can be referred to the following formula:

[0068]

[0069] Where X represents the input, This represents the output, and E represents the minimum energy.

[0070] The following combination Figure 6 The process of deriving the minimum energy for the SimAttention module will be further explained.

[0071] The SimAttention module identifies more important neurons by defining the linear separability between them. First, it defines the neuron energy function, as shown in the following equation, where C channels contain M = W * H energy functions.

[0072]

[0073]

[0074]

[0075] In the formula, Therefore, the minimum energy e can be obtained. t * As shown in the following formula,

[0076]

[0077] Lower energy indicates a greater difference between neuron t and surrounding neurons, suggesting higher importance. Therefore, attention can be represented by calculating the reciprocal of the minimum energy, thereby generating attention weights. Figure 6 The attention weights generated by the SimAttention module are three-dimensional (3D weights). These attention weights are then expanded in terms of channel and spatial location, and finally fused with the input features (C×W×H) to obtain the enhanced features, which are then output.

[0078] Since only the minimum energy needs to be calculated and no other parameters are introduced, changes to the neural network structure are avoided, making the development of training models more convenient, while at the same time improving the performance of training models.

[0079] like Figure 4 As shown, the pooling component 420 is coupled to the feature extraction component 410, and includes three parallel pooling layers and a feature fusion layer. The feature map is input into the pooling component 420 for fusion processing to obtain a feature fusion map. By introducing multiple parallel pooling layers, the model can adapt to various input images. At the same time, the output feature fusion map is of a fixed size, which solves the problem that fully connected layers require input images of the same size.

[0080] Figure 7 A schematic diagram of the structure of a pooling component 420 according to some embodiments of the present disclosure is shown. For example... Figure 7As shown, pooling layers 422, 424, and 426 are connected in parallel, pooling the input feature maps respectively to obtain corresponding pooled feature maps. Then, feature fusion layer 428 fuses each pooled feature map with the original feature map to obtain a fixed-size feature fusion map. In one embodiment, pooling layers 422, 424, and 426 all use max pooling, with pooling window sizes of 5*5, 7*7, and 13*13 respectively. By fusing features under different receptive fields, the model accuracy can be further improved.

[0081] Continue as Figure 4 The score prediction component 430 is coupled to the pooling component 420 and contains four fully connected layers for processing the feature fusion map, such as convolution, to ultimately output a predicted score indicating the quality of the training image. According to one embodiment, the output dimensions of the four fully connected layers (denoted as FC1, FC2, FC3, and FC4, respectively) are 2048, 1024, 256, and 1. It should be understood that, in addition to convolution, the fully connected layers also include activation processing; optionally, the activation function is ReLU, but this is not a limitation.

[0082] It should be understood that when constructing the training model, step 310 also includes acquiring training images and obtaining the subjective evaluation score of the image quality of the training images in advance (hereinafter referred to as "subjective score").

[0083] In some embodiments, the KonIQ-10k dataset is used as the training image dataset. This dataset is a current image quality assessment dataset containing 10,073 image samples. For each image, 1,459 volunteers gave five categories of ratings for image quality, and these five categories of ratings were mapped to [0, 100].

[0084] Furthermore, for subjective scores, multiple professional photographers can conduct subjective scoring under the same experimental conditions to obtain an average subjective score. The distribution of the average subjective scores from volunteers and professional photographers is then fitted to obtain the final subjective score, thus balancing the reliability of the participants in the subjective evaluation.

[0085] Then, in step 320, the training images are input into the training model for processing.

[0086] Specifically, the training image is input to the feature extraction component 410, which performs feature extraction to obtain a feature map. Then, the feature map is input to the pooling component 420, which performs feature fusion to obtain a fused feature map. Finally, the fused feature map is processed by the score prediction component 430 to obtain a predicted score indicating the quality of the training image.

[0087] For details on the process of training model 400 in processing training images, please refer to the previous description of the network structure of training model 400, which will not be repeated here.

[0088] Subsequently, in step 330, the model parameters of training model 400 are adjusted by calculating the loss value between the predicted score and the subjective score of the training image, until training ends and a well-trained model is obtained.

[0089] Experiments have shown that the first-order loss function provides more stable training compared to the second-order loss function. Therefore, in some embodiments, the loss value L... MAE Calculated using the following formula:

[0090]

[0091] In the formula, q represents the predicted score output by the trained model. This indicates the subjective score.

[0092] Furthermore, to prevent overfitting, according to some embodiments of this disclosure, Dropout is applied to the fully connected layers in the score prediction component 430 during training. Specifically, Dropout = 0.25 is set in FC1, FC2, and FC3.

[0093] The training continues until the end, resulting in a trained model, which includes: a trained feature extraction component, a trained pooling component, and a trained score prediction component.

[0094] Subsequently, in step 340, the trained model is subjected to an equivalent transformation to generate an image quality evaluation model.

[0095] According to some embodiments of this disclosure, the trained feature extraction component in the trained model is transformed into an equivalent feature extraction component. Then, the equivalent feature extraction component, the trained pooling component, and the trained score prediction component are coupled together to form the image quality assessment model.

[0096] In some embodiments, the equivalent feature extraction component is obtained through the following steps.

[0097] As mentioned earlier, the feature extraction component 410 utilizes its multi-branch module (R) to improve accuracy and facilitate model training. However, during the prediction phase, the low parallelism of the multi-branch network structure leads to a high MAC (memory access cost), and the multi-branch structure has low memory utilization. This is because the model output requires summing the branches, and the parameters of each branch need to be saved before this operation, resulting in high peak memory usage. However, memory usage decreases significantly when performing other operations. Therefore, the multi-branch module (R) is first processed to be equivalently converted into a single-branch structure. Then, all the multi-branch modules (R) in the feature extraction component 410 are replaced with single-branch structures to obtain the equivalent feature extraction component.

[0098] In one embodiment, the multi-branch module (R) is equivalently converted into a single-branch network structure, including the following process: For each branch, combining the parameters learned by the Batch Normalization (BN) layer, the convolutional layer and the BN layer are fused to obtain a fused convolutional layer. The fused convolutional layers from the three branches are then stacked together, equivalent to convolutional kernels of the same size, serving as a single-branch structure. Figure 5 In (b), the three branches are equivalently transformed into single-branch convolutional layers. The kernel size of the single-branch convolutional layer is 3*3. The features output by the single-branch convolutional layer are fed into the attention module (A).

[0099] According to embodiments of this disclosure, the parameters of the BN layer include: a scaling factor γ, a bias factor β, a cumulative mean μ, and a standard deviation σ.

[0100] As shown in the following equation, the input M is defined. (1) ∈R NxC1xH1xW1 Output M (2) ∈R NxC2xH2xW2 In the formula W (3) ∈R C2xC1x3x3 W represents a 3x3 convolution (i.e., a 3x3 convolution branch) with input C1 and output C2. (1) W (0) represents the 1x1 convolution branch and the identity mapping branch, respectively, and μ, σ, γ, β represent the parameters of the BN layer during inference, as detailed below:

[0101] M (2) =BN(M (1) *W (3) ,μ (3) ,σ (3) ,γ (3) ,β (3) )+BN(M (1) *W (1) ,μ (1) ,σ (1) ,γ (1),β (1) )+BN(M (1) *W (0) ,μ (0) ,σ (0) ,γ (0) ,β (0) )

[0102] Furthermore, according to the following formula, combining the learned parameters, the BN layer and the convolutional layer are equivalent to a single biased convolutional layer. Specifically, the identity mapping branch is equivalent to a 1x1 convolution with an identity matrix kernel, which is then expanded; the parameters of the 1x1 convolution kernel are added to the center of a 3x3 convolution kernel. Finally, the biases of the three branches are summed, thus making the three branches equivalent to a single 3x3 convolution kernel.

[0103]

[0104]

[0105] In the formula, W i ′ represents the weight of the convolution after the i-th branch is equivalent, b i ′ represents the bias after the i-th branch is equivalent, W i,:,:,: This represents the parameters of the convolutional layer before the equivalent transformation.

[0106] This concludes the introduction to the method 300 for training and generating image quality assessment models.

[0107] According to some embodiments of this disclosure, the feature extraction component (especially the multi-branch module) uses a RepVGG network structure when constructing the training model. During the training phase, the multi-branch network learns image features; during the prediction phase, structural reparameterization technology is used to convert the multi-branch network structure into a single-branch network structure, improving model inference speed, reducing peak memory usage, and facilitating model deployment. It should be noted that no-reference image quality assessment can be achieved by replacing or improving some network structures in this disclosure. This disclosure does not impose excessive restrictions on this, and a balance can be struck between model accuracy and real-time performance according to actual needs.

[0108] Furthermore, in the feature extraction component, the SimAttention module is used as the attention module, which improves the model's ability to extract effective features of image quality without increasing parameters or changing the module structure.

[0109] In addition, the use of pooled components solves the problem that the fully connected layer of the subsequent score prediction component requires a fixed input, making it more suitable for use in real business scenarios.

[0110] Accordingly, Figure 8A schematic flowchart of an image quality evaluation method 800 according to some embodiments of the present disclosure is shown. In one embodiment, the method 800 is performed using the aforementioned image quality evaluation unit 120.

[0111] like Figure 8 As shown, method 800 includes 810, which involves inputting the image to be evaluated into an image quality evaluation model for processing to obtain an image quality evaluation score.

[0112] According to embodiments of this disclosure, the image quality assessment model is generated by performing the aforementioned method 300. The image quality assessment model includes a coupled equivalent feature extraction component, a trained pooling component, and a trained score prediction component.

[0113] Combination Figure 3 and Figure 4 The relevant description involves inputting the image to be evaluated into the image quality assessment model. First, a feature extraction component, after being converted to an equivalent model, extracts features to generate a feature map. Then, the feature map is input into a trained pooling component to obtain a feature fusion map. Finally, the feature fusion map is input into a trained score prediction component for further processing, ultimately yielding a predicted score indicating the quality of the image to be evaluated.

[0114] For details on the specific structure of the image quality assessment model and the processing procedure for the input image, please refer to the descriptions of Method 300 and Training Model 400 in the previous text, which will not be repeated here.

[0115] The various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the methods and apparatus of this disclosure, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, USB flash drive, floppy disk, CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into and executed by a machine such as a computer, the machine becomes an apparatus for practicing this disclosure.

[0116] When the program code is executed on a programmable computer, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute, according to instructions in the program code stored in the memory, the method for generating an image quality assessment model and the image quality assessment method of this disclosure.

[0117] By way of example, and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer-readable instructions, data structures, program modules, or other data. Communication media generally embodies computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery medium. Any combination of the above is also included within the scope of readable media.

[0118] In the specification provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with the examples of this disclosure. Based on the above description, the required structure for constructing such systems is apparent. Furthermore, this disclosure is not directed to any particular programming language. It should be understood that the contents of this disclosure described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing preferred embodiments of this disclosure.

[0119] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0120] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various aspects of the disclosure, in the foregoing description of exemplary embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed disclosure requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, the aspects of the disclosure consist of fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the disclosure.

[0121] Those skilled in the art will understand that modules, units, or components of the devices disclosed in the examples herein can be arranged in the devices described in this embodiment, or alternatively, can be located in one or more devices different from the devices in this example. The modules in the foregoing examples can be combined into a single module or, in addition, can be divided into multiple sub-modules.

[0122] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0123] This disclosure also includes:

[0124] A7. The method described in A6, wherein the step of performing an equivalent transformation on the feature extraction components in the trained model to obtain an equivalent feature extraction component includes: transforming the multi-branch module into a single-branch structure; replacing the multi-branch module in the feature extraction component with the single-branch structure to obtain an equivalent feature extraction component.

[0125] A8. The method described in A7, wherein the step of obtaining a single-branch structure based on multi-branch module conversion includes: for each branch, combining the learned parameters, fusing convolutional layers and batch normalization layers to obtain fused convolutional layers; and superimposing the fused convolutional layers of each branch to obtain a single-branch structure.

[0126] A9. The method as described in A3, wherein the attention module is further adapted to generate attention weights by calculating the reciprocal of the minimum energy.

[0127] A10. The method as described in any one of A1-9, wherein the loss value is calculated using the following formula:

[0128] In the formula, q represents the predicted score output by the trained model. This indicates the subjective score.

[0129] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this disclosure and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0130] Furthermore, some of the embodiments described herein are methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing the functions. Therefore, a processor having the necessary instructions for implementing the method or method elements forms means for implementing the method or method elements. Furthermore, the elements described herein in the apparatus embodiments are examples of means for implementing the functions performed by the elements for the purposes of this disclosure.

[0131] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.

[0132] Although this disclosure has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of this disclosure. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and edibility purposes, and not for interpreting or limiting the subject matter of this disclosure. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. Regarding the scope of this disclosure, the disclosure is illustrative and not restrictive, and the scope of this disclosure is defined by the appended claims.

Claims

1. A method for generating an image quality assessment model, comprising: A training model for generating an image quality assessment model is constructed, the training model comprising: a feature extraction component, a pooling component, and a score prediction component; The training image is input into the training model for processing, including: the training image is subjected to feature extraction by the feature extraction component to obtain a feature map; the feature map is fused by the pooling component to obtain a feature fusion map; the feature fusion map is processed by the score prediction component to obtain a predicted score indicating the quality of the training image; The model parameters of the training model are adjusted by calculating the loss value between the predicted score and the subjective score of the training image until the training is completed, and a well-trained training model is obtained. The trained model is then subjected to an equivalent transformation to generate an image quality evaluation model. The step of performing an equivalent transformation on the trained model to generate an image quality assessment model includes: The feature extraction components in the trained model are transformed into equivalent components to obtain equivalent feature extraction components. The feature extraction component includes at least one feature extraction stage, and the feature extraction stage includes a different number of convolutional processing blocks. The convolutional processing blocks include a coupled multi-branch module and an attention module. The multi-branch module includes a 3x3 convolutional branch, a 1x1 convolutional branch, and an identity mapping branch. The multi-branch module is adapted to extract features of the image, and the attention module is adapted to enhance the features through the generated attention weights. The step of performing an equivalent transformation on the feature extraction components in the trained model to obtain an equivalent feature extraction component includes: Based on the multi-branch module, a single-branch structure is obtained; By replacing the multi-branch module in the feature extraction component with the single-branch structure, an equivalent feature extraction component is obtained.

2. The method as described in claim 1, wherein, The pooling component includes three pooling layers and a feature fusion layer. The pooling layers pool the input feature map to obtain a corresponding pooled feature map. The feature fusion layer fuses each pooled feature map with the feature map to obtain a feature fusion map of a fixed size.

3. The method as described in claim 1, wherein, The score prediction component contains four fully connected layers.

4. The method of claim 1, wherein, The steps for obtaining a single-branch structure based on multi-branch module conversion include: For each branch, the learned parameters are combined to fuse the convolutional layer and the batch normalization layer to obtain the fused convolutional layer; The convolutional layers after merging the branches are stacked to obtain a single-branch structure.

5. The method of claim 1, wherein, The attention module is also adapted to generate attention weights by calculating the reciprocal of the minimum energy.

6. The method according to any one of claims 1-5, wherein, The loss value is calculated using the following formula: In the formula, q represents the predicted score output by the trained model. This indicates the subjective score.

7. An image quality assessment method, comprising: The image to be evaluated is input into the image quality assessment model for processing, and an image quality assessment score is obtained. The image quality assessment model is generated by performing the method described in any one of claims 1-6.

8. A computing device, comprising: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method as described in any one of claims 1-7.

9. A computer-readable storage medium storing one or more programs, said one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-7.

Images