A dual-channel no-reference image quality assessment method combining global and local features
By using a dual-channel network based on the Swin-T module to extract global and local features from images, the problems of information loss and low efficiency in quality assessment without reference images are solved, achieving efficient quality assessment of images of arbitrary size and improving the consistency of assessment results with human vision.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ELECTRIC POWER RES INST OF GUANGXI POWER GRID CO LTD
- Filing Date
- 2022-12-22
- Publication Date
- 2026-07-03
AI Technical Summary
Existing no-reference image quality assessment algorithms suffer from information loss or low efficiency when combining global and local features, especially in deep learning-based networks. Image resizing leads to the loss of local distortion information and lacks semantic object information of the whole image.
A dual-channel network based on the Swin-T module is used to extract global and local features from the input image. The image is preprocessed by resizing and local reconstruction operations, and the fully connected layer is used to map the features to the quality score.
It enables quality assessment of images of any size, enhances the ability to adapt to image distortion, improves consistency with human visual characteristics, and enhances the effectiveness of image quality assessment.
Smart Images
Figure CN116091422B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image quality assessment, specifically relating to a dual-channel no-reference image quality assessment method that combines global and local features. Background Technology
[0002] In recent years, with the rapid development of computer and internet technologies, multimedia technology has also made significant progress. However, in multimedia systems, distortions in digital images during acquisition, processing, compression, storage, and transmission can lead to a decline in image quality. Therefore, research on algorithms for scoring the quality of an image has become an important research direction in the field of image processing.
[0003] Image Quality Assessment (IQA) can be used in many image processing applications, such as image denoising, image reconstruction, image compositing, and image and video coding. Based on the use of reference image information, IQA is divided into three types: Full-Reference, Reduced-Reference, and No-Reference. Among these, the No-Reference Image Quality Assessment (NR-IQA) algorithm is the most widely applicable because in practical applications, there are often no distortion-free images available for full-reference or reduced-reference algorithms.
[0004] To make IQA algorithm results more consistent with human subjective evaluations of images, various algorithms are striving to integrate global and local image features for quality assessment. While extracting and integrating global and local image features is relatively simple for traditional, manually crafted feature-based IQA algorithms—which only require extracting the two types of features separately using image processing methods and then combining them using a single Feature Variation Detector (SVR) to map the features to a quality score—simultaneously extracting both types of features is a significant challenge for deep learning-based IQA algorithms. Deep learning networks typically require input images to have a fixed size. To meet this requirement, one approach is to resize the input image. However, this approach suffers from the drawback that resizing the image can lead to the loss of local distortion information. Another approach is to use image patches, where multiple non-repeating local image patches are cropped from the original image and directly input into the deep network. The quality score of the entire image is then averaged from the scores of all its image patches. This approach can effectively capture local distortion information of the image, but when evaluating the entire image, it lacks global contextual information about semantic objects in the image. Furthermore, this approach is inefficient in calculating image quality scores, as predicting the score of an image generally requires the quality scores of a large number of image patches. Summary of the Invention
[0005] To address or improve the aforementioned problems, this invention proposes a dual-channel no-reference image quality assessment method that combines global and local features. This method utilizes a dual-channel network based on the Swin-T module to extract global and local features from the input image, thereby combining the two features to map to a quality score.
[0006] The specific technical solution of the present invention is as follows:
[0007] This invention provides a dual-channel no-reference image quality assessment method combining global and local features, comprising the following steps:
[0008] Step S1, image preprocessing, including processing an input image of arbitrary size to obtain an image containing global information and a set size, and an image containing local distortion information and a set size;
[0009] Step S2: Use a dual-channel network based on the Swin-T module to extract global and local features from the two preprocessed images respectively.
[0010] Step S3: Design multiple fully connected layers after the dual-channel network, use the fully connected layers to form a quality regression network, and then use the quality regression network to map image quality features to image quality scores.
[0011] Step S4: Train the dual-channel network using massive amounts of publicly available image data according to the methods in steps S1 to S3, and finally save the trained network model.
[0012] Step S5: Input the image to be evaluated into the network model obtained in step S4 to perform no-reference image quality evaluation and obtain the image quality score of the image to be evaluated.
[0013] Optionally, an image containing global information and of a set size can be obtained by performing a resize operation on an input image of arbitrary size, and used as the input image for the global channel network.
[0014] Optionally, a local reconstruction operation can be performed on an input image of arbitrary size to obtain an image containing local distortion information and of a set size, which can then be used as the input image for the local channel network.
[0015] Optionally, the local reconstruction operation includes: first, dividing the original image into n×n image blocks evenly; then, taking out an image patch of size (224 / n)×(224 / n) from the center of each image block; and then reconstructing the image patch according to its original position in the image to finally obtain a reconstructed image of size 224×224.
[0016] Optionally, the size can be set to 224×224.
[0017] Optionally, the dual-channel network based on the Swin-T module includes two Swin-T modules with the same structure. One channel based on the Swin-T module is used to extract global features from an image containing global information, and the other channel based on the Swin-T module is used to extract local features from an image containing local distortion information.
[0018] Optionally, the quality regression network is a network that connects the global and local features extracted by the dual-channel network and then uses three fully connected layers to map the features to quality scores.
[0019] According to another aspect of the present invention, a dual-channel no-reference image quality assessment system combining global-local features is also provided, comprising:
[0020] The image preprocessing module is used to preprocess images, including processing input images of arbitrary size to obtain an image containing global information of a set size and an image containing local distortion information of a set size.
[0021] The dual-channel network module is used to extract global and local features from two preprocessed images using a dual-channel network based on the Swin-T module.
[0022] The fully connected layer module is used to design multiple fully connected layers after the dual-channel network. The fully connected layers are used to form a quality regression network, and then the quality regression network is used to map the image quality features to the image quality score, thus obtaining the dual-channel network.
[0023] The training module is used to train a dual-channel network obtained by fully connected layer modules from massive amounts of publicly available image data, and finally saves the trained network model; the image to be evaluated is input into the trained network model to perform no-reference image quality evaluation and obtain the image quality score of the image to be evaluated.
[0024] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the dual-channel no-reference image quality assessment method combining global-local features as described above.
[0025] According to another aspect of the present invention, a processor is also provided, the processor being used to run a program, wherein the program, when running, executes the dual-channel no-reference image quality assessment method combining global-local features as described in any of the preceding embodiments. The beneficial effects of the present invention are as follows: The present invention first performs image preprocessing operations of resizing and local reconstruction on the image, enabling the present invention to perform quality assessment on input images of any size; then, it utilizes a dual-channel network based on the Swin-T module to extract features from the two preprocessed input images respectively, obtaining global and local features of the image, which can be combined to compensate for the deficiency of single-branch networks in extracting image quality features to a certain extent; finally, a quality regression network is composed of fully connected layers to complete the mapping from global-local features to quality scores. The present invention has a strong adaptability to the non-uniformity of image distortion and can effectively perform quality assessment on truly distorted images. The proposed algorithm's performance in image quality assessment has a high consistency with human visual characteristics. Attached Figure Description
[0026] Figure 1 This is a flowchart of a dual-channel no-reference image quality assessment method combining global and local features according to the present invention;
[0027] Figure 2 This is a schematic diagram of the Resize operation according to the present invention;
[0028] Figure 3 This is a schematic diagram of partial reconstruction according to the present invention;
[0029] Figure 4 This is a dual-channel network diagram according to the present invention. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0032] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0033] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0034] To address or improve the poor performance of referenceless image quality assessment, the following proposals are put forward: Figure 1 The method shown is a dual-channel no-reference image quality assessment method that combines global and local features, including:
[0035] Step S1 involves preprocessing the image, including processing an input image of arbitrary size to obtain an image containing global information and a set size, and an image containing local distortion information and a set size.
[0036] As an optional embodiment, step S1 includes:
[0037] Step S11, as follows Figure 2 As shown, in the image preprocessing stage, a Resize operation is performed on the input image of any size to obtain an image containing global information with a size of 224×224, which is used as the input image of the global channel network.
[0038] Step S12: In the image preprocessing stage, a local reconstruction operation is performed on the input image of any size to obtain an image containing local distortion information with a size of 224×224 as the input image of the local channel network.
[0039] like Figure 3 As shown, the local reconstruction operation includes: first, dividing the original image into n×n image blocks evenly; then, taking out an image patch of size (224 / n)×(224 / n) from the center of each image block; and then reconstructing the image patch according to its original position in the image, finally obtaining a reconstructed image of size 224×224.
[0040] Step S2: Use a dual-channel network based on the Swin-T module to extract global and local features from the two preprocessed images respectively.
[0041] As an optional embodiment, the dual-channel network based on the Swin-T module includes two Swin-T modules with the same structure. One channel based on the Swin-T module is used to extract global features from an image containing global information, and the other channel based on the Swin-T module is used to extract local features from an image containing local distortion information.
[0042] Specifically, the Swin-T module belongs to the structure of the Swin-Transformer model, and the specific network parameter information is shown in Table 1.
[0043] Table 1. Swin-T Feature Extraction Network Structure Information
[0044]
[0045] Table 1 lists the parameters, including the number of Swing Transformer Blocks (×N), the depth of the feature map (dim), the number of heads in the multi-head attention, and the window size (win.sz.).
[0046] Step S3: Design multiple fully connected layers after the dual-channel network, use the fully connected layers to form a quality regression network, and then use the quality regression network to map image quality features to image quality scores.
[0047] As an optional embodiment, the quality regression network connects the global and local features extracted by the dual-channel network and then uses three fully connected layers to complete the mapping from features to quality scores.
[0048] Step S4: Train the dual-channel network using massive amounts of publicly available image data following the methods in steps S1 to S3. Finally, save the trained network model, as shown in the image. Figure 4 As shown;
[0049] Step S5: Input the image to be evaluated into the network model obtained in step S4 to perform no-reference image quality evaluation and obtain the image quality score of the image to be evaluated.
[0050] Example 2
[0051] According to another aspect of the present invention, a dual-channel no-reference image quality assessment system combining global-local features is also provided, the system comprising:
[0052] The image preprocessing module is used to preprocess images, including processing input images of arbitrary size to obtain an image containing global information of a set size and an image containing local distortion information of a set size.
[0053] The dual-channel network module is used to extract global and local features from two preprocessed images using a dual-channel network based on the Swin-T module.
[0054] The fully connected layer module is used to design multiple fully connected layers after the dual-channel network. The fully connected layers are used to form a quality regression network, and then the quality regression network is used to map the image quality features to the image quality score, thus obtaining the dual-channel network.
[0055] The training module is used to train a dual-channel network obtained by fully connected layer modules from massive amounts of publicly available image data, and finally saves the trained network model; the image to be evaluated is input into the trained network model to perform no-reference image quality evaluation and obtain the image quality score of the image to be evaluated.
[0056] This invention is not limited to the specific embodiments described above. The above are merely preferred embodiments of this invention and are not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
[0057] Example 3
[0058] According to another aspect of the present invention, a dual-channel no-reference image quality assessment system combining global and local features is also provided, with the following operating environment: Windows 10 operating system, Intel Core i7-8700K central processing unit, 32GB memory, NVIDIA GeForce Titan XP image processor, CUDA 11.1 image processor acceleration library, cuDNN 8.0 image processor acceleration library, and PyTorch deep learning framework.
[0059] Table 2 Performance comparison on the LIVEC and SPAQ datasets
[0060]
[0061] Table 2 shows LIVEC and SPAQ as publicly available real-distortion image datasets. Images from these datasets are used as input to a dual-channel network, and the labeled scores of these images are used as the network's output. This is used to train the network and obtain the actual values of its parameters. The horizontal axis of the table compares the performance of this invention and some other superior algorithms in the field of image quality assessment on the LIVEC and SPAQ datasets. "Proposed" indicates the algorithm of this patent. The SROCC and PLCC metrics are used to represent the algorithm's performance; the closer the metric value is to 1, the better the algorithm's performance. Values closer to 1 are more difficult to improve. Based on the comparison, the algorithm of this patent can be considered to have high accuracy in the field of image quality assessment.
[0062] The method of the present invention is then compared with an algorithm for extracting image quality features using a single-branch network. The comparison results are shown in Table 3.
[0063] Table 3. Ablation comparison experiments on the LIVEC and SPAQ datasets.
[0064]
[0065] In Table 3, G_IQA represents a single-channel network model preprocessed only with Resize operations; L_IQA represents a single-channel network model preprocessed only with local recombination operations; and GL_IQA, proposed in this invention, represents a dual-channel network model preprocessed with both Resize and local recombination. Experimental results show that GL_IQA, using the method proposed in this invention, achieved the best ablation results on both datasets. This demonstrates that the combination of global and local features in the proposed algorithm improves the consistency between the model performance and human subjective vision.
[0066] In summary, this invention first performs image preprocessing operations such as resizing and local reconstruction on the image, enabling it to assess the quality of input images of any size. Then, it utilizes two channels based on the Swin-T module to extract features from the two preprocessed input images, obtaining global and local features. These two features can be combined, thus mitigating the limitations of single-branch networks in extracting image quality features. Finally, a quality regression network composed of fully connected layers completes the mapping from global to local features to quality scores. This invention exhibits strong adaptability to the non-uniformity of image distortion and can effectively assess the quality of truly distorted images. Experimental results also demonstrate that the proposed algorithm's performance in image quality assessment is highly consistent with human visual characteristics.
[0067] Example 4, according to another aspect of the embodiments of the present invention, also provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to execute any of the above-described dual-channel no-reference image quality assessment methods combining global-local features.
[0068] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals, and the computer-readable storage medium includes a stored program.
[0069] Optionally, during program execution, the device containing the computer-readable storage medium performs the following functions: Step S1, image preprocessing, including processing an input image of arbitrary size to obtain an image containing global information of a set size and an image containing local distortion information of a set size; Step S2, using a dual-channel network based on the Swin-T module to extract global and local features from the two preprocessed images respectively; Step S3, designing multiple fully connected layers after the dual-channel network, using the fully connected layers to form a quality regression network, and then using the quality regression network to map image quality features to image quality scores; Step S4, training the dual-channel network using massive amounts of publicly available image data according to the methods in steps S1 to S3, and finally saving the trained network model; Step S5, inputting the image to be evaluated into the network model obtained in step S4 to perform no-reference image quality evaluation, and obtaining the image quality score of the image to be evaluated.
[0070] Example 5
[0071] According to another aspect of the present invention, a processor is also provided for running a program, wherein the program executes any of the above-described dual-channel no-reference image quality assessment methods combining global-local features.
[0072] This invention provides an apparatus including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of a dual-channel no-reference image quality assessment method that combines global and local features.
[0073] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0074] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0075] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the invention.
[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.
Claims
1. A dual-channel no-reference image quality assessment method combining global and local features, characterized in that, Includes the following steps: Step S1, image preprocessing, including processing an input image of arbitrary size to obtain an image containing global information and a set size, and an image containing local distortion information and a set size; Step S2: Use a dual-channel network based on the Swin-T module to extract global and local features from the two preprocessed images respectively. The dual-channel network based on the Swin-T module includes two Swin-T modules with the same structure. One channel based on the Swin-T module is used to extract global features from the image containing global information, and the other channel based on the Swin-T module is used to extract local features from the image containing local distortion information. Step S3: Design multiple fully connected layers after the dual-channel network, use the fully connected layers to form a quality regression network, and then use the quality regression network to map image quality features to image quality scores; the quality regression network is formed by connecting the global and local features extracted by the dual-channel network, and then using three fully connected layers to complete the mapping of features to quality scores. Step S4: Train the dual-channel network using massive amounts of publicly available image data according to the methods in steps S1 to S3, and finally save the trained network model. Step S5: Input the image to be evaluated into the network model obtained in step S4 to perform no-reference image quality evaluation and obtain the image quality score of the image to be evaluated.
2. The dual-channel no-reference image quality assessment method combining global and local features according to claim 1, characterized in that, By performing a resize operation on an input image of arbitrary size, an image containing global information and of a set size is obtained and used as the input image for the global channel network.
3. The dual-channel no-reference image quality assessment method combining global and local features according to claim 1, characterized in that, A local reconstruction operation is performed on an input image of arbitrary size to obtain an image of a set size containing local distortion information, which is then used as the input image for the local channel network.
4. The dual-channel no-reference image quality assessment method combining global and local features according to claim 3, characterized in that, The local reconstruction operation includes: first, dividing the original image into n×n image blocks evenly; then, taking out an image patch of size (224 / n)×(224 / n) from the center of each image block; and then reconstructing the image patch according to its original position in the image, finally obtaining a reconstructed image of size 224×224 pixels.
5. The dual-channel no-reference image quality assessment method combining global and local features according to claim 1, characterized in that, The size is set to 224×224 pixels.
6. A dual-channel no-reference image quality assessment system combining global and local features, characterized in that, The dual-channel no-reference image quality assessment method combining global and local features as described in any one of claims 1 to 5 includes: The image preprocessing module is used to preprocess images, including processing input images of arbitrary size to obtain an image containing global information of a set size and an image containing local distortion information of a set size. The dual-channel network module is used to extract global and local features from two preprocessed images respectively through a dual-channel network based on the Swin-T module. The fully connected layer module is used to design multiple fully connected layers after the dual-channel network. The fully connected layers are used to form a quality regression network, and then the quality regression network is used to map the image quality features to the image quality score, thus obtaining the dual-channel network. The training module is used to train a dual-channel network obtained by fully connected layer modules from massive amounts of publicly available image data, and finally saves the trained network model; the image to be evaluated is input into the trained network model to perform no-reference image quality evaluation and obtain the image quality score of the image to be evaluated.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the dual-channel no-reference image quality assessment method combining global-local features as described in any one of claims 1 to 5.
8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the dual-channel no-reference image quality assessment method combining global-local features as described in any one of claims 1 to 5.