Underwater image quality evaluation method based on sparse representation and Transform

Abstract

The invention belongs to the technical field of image processing, and discloses an underwater image quality evaluation method based on sparse representation and Transform, comprising the following steps: S1, selecting N underwater images with subjective quality labels, and extracting visual perception features of each underwater image by using sparse representation; s2, extracting a semantic feature of each underwater image by using a Transform; s3, splicing the visual perception features and the semantic features, inputting the spliced features into a multi-layer perceptron, and constructing an underwater image quality evaluation model; s4, performing end-to-end training on the underwater image quality evaluation model; and S5, predicting the quality of an underwater image to be evaluated by using the trained image quality model. The method does not need to refer to an original image, can obtain high prediction performance, and meets the application requirements of underwater imaging.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an underwater image quality evaluation method based on sparse representation and Transformer. Background Art

[0002] Underwater imaging plays a key role in underwater operations such as ocean observation, scientific research, engineering construction, etc. However, due to the refraction and scattering of light by water, the quality of underwater imaging is often low and cannot meet the needs of underwater applications. Therefore, it is necessary to study reliable underwater image quality evaluation technology to accurately judge the quality of underwater images, thereby improving the quality of underwater images.

[0003] In the underwater image quality evaluation, Yang et al. proposed the underwater color image quality evaluation (UCIQE) in the paper "An underwater color image quality evaluation metric" published in IEEE Trans. Image Process. This method evaluates the image quality by linearly weighting the three indicators of color, contrast and saturation. However, this method uses linear weighting to combine the color, contrast and saturation indicators, which may not accurately reflect the complex nonlinear relationship between image quality and these indicators; it is not universal for different types of underwater images, and the evaluation results may be biased.

[0004] Panetta et al. proposed the underwater image quality measure (UIQM) in the paper “Human-visual-system-inspired underwater image quality measures” published in IEEE Journal of Oceanic Engineering. This measure measures the degradation of underwater images from three aspects: color, blur and contrast. However, this method relies on manually designed features, which cannot fully capture all aspects of image quality, and the evaluation criteria are somewhat subjective.

[0005] Guo et al. published a paper titled "Quality Assessment Method for Underwater Images" in the Journal of Image and Graphics. This method uses a deep neural network to automatically extract image features, avoiding manual feature design, and then uses a random forest to learn a regression model from image features to image quality scores, improving the efficiency of image evaluation. However, deep learning models usually require a large amount of annotated data for training, and the dataset of underwater images may not be large enough or diverse enough, which may lead to limited generalization of the model.

[0006] The defects of these existing methods mainly include the versatility, accuracy, computational efficiency, data dependence, subjectivity and interpretability of the evaluation model. In order to improve the accuracy and practicality of underwater image quality evaluation, the present invention proposes an underwater image quality evaluation method based on sparse representation and Transformer. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention provides an underwater image quality evaluation method based on sparse representation and Transformer, extracts the visual perception features of underwater images through sparse representation, extracts the semantic features of underwater images through Transformer, constructs an underwater image quality evaluation model using a multi-layer perceptron, and trains the image quality evaluation model in an end-to-end manner to achieve accurate judgment of underwater image quality. The present invention does not need to refer to the original image, effectively guides underwater image enhancement, and improves the efficiency of underwater operations.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] The underwater image quality evaluation method based on sparse representation and Transformer includes the following steps:

[0010] S1. Select N underwater images with subjective quality annotations, and use sparse representation to extract visual perception features of each underwater image.

[0011] S2, use Transformer to extract the semantic features of each underwater image;

[0012] S3, splicing visual perception features and semantic features, inputting them into a multi-layer perceptron, and constructing an underwater image quality evaluation model;

[0013] S4. Perform end-to-end training on the underwater image quality assessment model, wherein the image features include the visual perception features and semantic features of the underwater image obtained in steps S1-S2.

[0014] S5. Use the trained image quality model to predict the underwater image quality to be evaluated.

[0015] Furthermore, step S1 described in selecting N underwater images with subjective quality annotations, for each underwater image, using sparse representation to extract its visual perception features, the specific method is as follows: collect N underwater images with subjective quality annotations, convert each underwater image into a grayscale image, perform sparse representation on the grayscale image, calculate the difference image between the grayscale image and its sparse representation, and extract the statistical features of the difference image as the visual perception features of each underwater image.

[0016] Furthermore, the step S1 specifically includes: decomposing each underwater image into three channels of red, green and blue, represented as R, G and B respectively, and then converting it into a grayscale image, represented as:

[0017]

[0018] in, Represents the grayscale image corresponding to the underwater image, and performs sparse representation on it. The specific process is as follows: first extract An image block, assuming Its size is The process is expressed as:

[0019]

[0020] Here, R k (·) is the image block extraction operator, which extracts the image block at position k, k = 1, 2, 3, ..., n, and n represents the total number of image blocks;

[0021] For x k , it is in the dictionary The sparse representation on is to find a sparse vector satisfy:

[0022]

[0023] Use the orthogonal matching pursuit algorithm to solve the above equation and get the image block x k The sparse representation coefficient of Then x k The sparse representation of Whole image The sparse representation of is:

[0024]

[0025] in, represents a vector whose values ​​are all 1, and its dimension is 64. ". / " represents the division of the corresponding elements of two matrices. I' is The sparse representation of Indicates that the image block x k Put it back to the k position of the image;

[0026] Compute the residual between the grayscale image and its sparse representation

[0027]

[0028] Extract residuals The mean, variance, kurtosis, skewness and information entropy of are used as visual perception features of underwater images.

[0029] Furthermore, the semantic features of each underwater image are extracted using Transformer as described in step S2 to characterize the semantic information of the underwater image. The specific method is as follows: each underwater image is input into a visual Transformer model, the visual Transformer model includes an encoder and a decoder, and the vector output by the visual Transformer model is used as the semantic feature of the underwater image.

[0030] Furthermore, the step S2 specifically includes: dividing each underwater image into image blocks of a certain size, then position encoding the image blocks, and then inputting the image blocks and their position encodings into the encoder of the Transformer model to obtain the output of the encoder, which is expressed as:

[0031] f i =encoder(P,PE)

[0032] Among them, f i represents the output of the encoder, P and PE represent all image blocks and their position encodings, and encoder(·) represents the encoder of the Transformer model; f i Input the decoder of the Transformer model, and the output of the decoder is used as the semantic feature of the underwater image, expressed as:

[0033] f2=decoder(f i )

[0034] Among them, f2 represents the semantic features of underwater images, and decoder(·) is the decoder of Transformer.

[0035] Furthermore, step S3 specifically includes: concatenating the visual perception features and the semantic features, expressed as:

[0036]

[0037] Among them, f1 and f2 represent the visual perception features and semantic features of underwater images respectively. represents the splicing operation, and then f is input into the multi-layer perceptron to build an image quality assessment model.

[0038] Furthermore, the underwater image quality assessment model described in step S4 is end-to-end trained, and the specific method is as follows: the image quality assessment model is trained using the collected N underwater images and their corresponding subjective quality scores.

[0039] Furthermore, step S4 specifically includes: inputting each underwater image into the constructed underwater image quality assessment model to obtain the quality score predicted by the model, and training the image quality assessment model using the mean square error, which is expressed as:

[0040]

[0041] Among them, q i is the quality score of the i-th image predicted by the model, s i It represents the subjective quality score of the i-th image, and the model is trained end-to-end using the Adam algorithm.

[0042] Furthermore, step S5 specifically includes: for an underwater image I to be evaluated, input it into the trained model, and the model outputs its quality score, which is expressed as:

[0043] q=M(I)

[0044] Where M(·) represents the trained underwater image quality assessment model, and q represents the quality score of I.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] The present invention extracts the visual perception features of underwater images through sparse representation, extracts the semantic features of underwater images through Transformer, constructs an underwater image quality evaluation model using a multi-layer perceptron, and trains the image quality evaluation model in an end-to-end manner to achieve accurate judgment of the underwater image quality, achieving performance superior to the existing technology.

[0047] In addition, the method proposed by the present invention runs fast and meets the application requirements of underwater operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0049] Figure 1 Schematic diagram of the underwater image quality evaluation model of the present invention;

[0050] Figure 2 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0052] The underwater imaging environment is complex. The present invention designs an underwater image quality evaluation system based on sparse representation and Transformer, including an image feature extraction module and a model learning module. The image feature extraction module extracts the image features of the underwater image to obtain the image feature values ​​of the underwater image. The model learning module uses a group of underwater images as a training set to train the image quality evaluation model, and uses the trained model to predict the quality of the underwater image.

[0053] The image features of the underwater image include visual perception features and semantic features of the image. The present invention extracts the visual perception features of the underwater image through sparse representation, uses the Transformer model to extract the semantic features of the underwater image, inputs the two parts of features into a multi-layer perceptron, thereby constructing an underwater image quality evaluation model, and trains the quality evaluation model using an end-to-end training method. After the training is completed, the model is used to make an accurate judgment on the quality of the underwater image.

[0054] Combination Figure 1 and Figure 2 The underwater image quality assessment method based on sparse representation and Transformer in this embodiment includes the following steps:

[0055] S1. Select 1,000 underwater images with subjective quality annotations, and use sparse representation to extract visual perception features of each underwater image.

[0056] S2, use Transformer to extract the semantic features of each underwater image;

[0057] S3, splicing visual perception features and semantic features, inputting them into a multi-layer perceptron, and constructing an underwater image quality evaluation model;

[0058] S4. Perform end-to-end training on the underwater image quality assessment model, wherein the image features include the visual perception features and semantic features of the underwater image obtained in steps S1-S2.

[0059] S5. Use the trained image quality model to predict the underwater image quality to be evaluated.

[0060] This method does not require reference to the original image and can achieve higher prediction performance while running fast, meeting the application requirements of underwater imaging.

[0061] As an embodiment, the specific method of step S1 is as follows: collect 1,000 underwater images with subjective quality annotations, convert each underwater image into a grayscale image, perform sparse representation on the grayscale image, calculate the difference image between the grayscale image and its sparse representation, and extract the statistical features of the difference image as the visual perception features of each underwater image.

[0062] Specifically, step S1 decomposes each underwater image into three channels: red, green, and blue, represented as R, G, and B, respectively. Then, the underwater image is converted into a grayscale image, represented as:

[0063]

[0064] in, Represents the grayscale image corresponding to the underwater image, and performs sparse representation on it. The specific process is as follows: first extract An image block, assuming Its size is The process is expressed as:

[0065]

[0066] Here, R k (·) is an image block extraction operator, which extracts the image block at position k, k = 1, 2, 3, ..., n, and n represents the total number of image blocks.

[0067] For x k , it is in the dictionary The sparse representation on is to find a sparse vector satisfy:

[0068]

[0069] Use the orthogonal matching pursuit algorithm to solve the above equation and get the image block x k The sparse representation coefficient of Then x k The sparse representation of Whole image The sparse representation of is:

[0070]

[0071] in, represents a vector whose values ​​are all 1, and its dimension is 64. ". / " represents the division of the corresponding elements of two matrices. I' is The sparse representation of Indicates that the image block x k Put it back to position k in the image.

[0072] Compute the residual between the grayscale image and its sparse representation

[0073]

[0074] Extract residuals The mean, variance, kurtosis, skewness and information entropy of are used as visual perception features of underwater images.

[0075] As an embodiment, the method of using Transformer to extract semantic features of each underwater image and characterize the semantic information of the underwater image in step S2 is as follows: each underwater image is input into a visual Transformer model, the visual Transformer model includes an encoder and a decoder, and the vector output by the visual Transformer model is used as the semantic feature of the underwater image.

[0076] Specifically, each underwater image is divided into image blocks of a certain size (such as 16×16), and then the image blocks are positionally encoded. The image blocks and their positional encodings are then input into the encoder of the Transformer model to obtain the output of the encoder, which is expressed as:

[0077] f i =encoder(P,PE)

[0078] Among them, f i represents the output of the encoder, P and PE represent all image blocks and their position encodings, and encoder(·) represents the encoder of the Transformer model.

[0079] f i Input the decoder of the Transformer model, and the output of the decoder is used as the semantic feature of the underwater image, expressed as:

[0080] f2=decoder(f i )

[0081] Among them, f2 represents the semantic features of underwater images, and decoder(·) is the decoder of Transformer.

[0082] As an embodiment, step S3 specifically includes: concatenating the visual perception feature and the semantic feature, expressed as:

[0083]

[0084] Among them, f1 and f2 represent the visual perception features and semantic features of underwater images respectively. represents the splicing operation, and then f is input into the multi-layer perceptron to build an image quality assessment model.

[0085] As an embodiment, step S4, the specific method is as follows: the image quality evaluation model is trained using the collected 1000 underwater images and their corresponding subjective quality scores.

[0086] Specifically, each underwater image is input into the constructed underwater image quality assessment model to obtain the quality score predicted by the model, and the image quality assessment model is trained using the mean square error, which is expressed as:

[0087]

[0088] Among them, q i is the quality score of the i-th image predicted by the model, s i represents the subjective quality score of the i-th image. The model is trained end-to-end using the Adam algorithm, and the learning rate is set to 1×e -4 , the number of samples used for each iteration training is set to 32.

[0089] Step S5 specifically includes: after the training is completed, for an underwater image I to be evaluated, it is input into the trained model, and the model outputs its quality score, which is expressed as:

[0090] q=M(I)

[0091] Where M(·) represents the trained underwater image quality assessment model, and q represents the quality score of I.

[0092] In a specific embodiment of the present invention, the performance of the method of the present invention is tested on the UWIQA library, and the performance of the algorithm is measured using three indicators: Spearman rank correlation coefficient (SRCC), Pearson linear correlation coefficient (PLCC) and root mean square error (RMSE). The experimental results are shown in Table 1.

[0093] Performance Indicators UCIQE UIQM CCF FDUM URQ The present invention SRCC 0.6271 0.5960 0.4456 0.6780 0.3724 0.7535 PLCC 0.6261 0.5928 0.4634 0.6462 0.3527 0.7516 RMSE 0.1185 0.1225 0.1348 0.1160 0.1423 0.1008

[0094] Table 1 compares the latest underwater image quality evaluation methods, including UCIQE, UIQM, CCF, FDUM, and URQ. It can be seen that the present invention achieves higher prediction performance than similar methods, which proves the effectiveness and superiority of the present invention.

[0095] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the protection scope of the present invention.

Claims

1. An underwater image quality assessment method based on sparse representation and Transformer, characterized by: The steps include: S1. Select N underwater images with subjective quality annotations, and use sparse representation to extract visual perception features of each underwater image. S2, use Transformer to extract the semantic features of each underwater image; S3, splicing visual perception features and semantic features, inputting them into a multi-layer perceptron, and constructing an underwater image quality evaluation model; S4, end-to-end training of underwater image quality assessment model; S5. Use the trained image quality model to predict the underwater image quality to be evaluated.

2. The underwater image quality assessment method based on sparse representation and Transformer according to claim 1, characterized in that: In step S1, N underwater images with subjective quality annotations are selected, and for each underwater image, its visual perception features are extracted using sparse representation. The specific method is as follows: N underwater images with subjective quality annotations are collected, each underwater image is converted into a grayscale image, a sparse representation is performed on the grayscale image, a difference image between the grayscale image and its sparse representation is calculated, and the statistical features of the difference image are extracted as the visual perception features of each underwater image.

3. The underwater image quality assessment method based on sparse representation and Transformer according to claim 2, characterized in that: The step S1 specifically includes: decomposing each underwater image into three channels of red, green and blue, represented as R, G and B respectively, and then converting it into a grayscale image, represented as: in, Represents the grayscale image corresponding to the underwater image, and performs sparse representation on it. The specific process is as follows: first extract An image block, assuming Its size is The process is expressed as: Here, R k (·) is the image block extraction operator, which extracts the image block at position k, k = 1, 2, 3, ..., n, and n represents the total number of image blocks; For x k , it is in the dictionary The sparse representation on is to find a sparse vector satisfy: Use the orthogonal matching pursuit algorithm to solve the above equation and get the image block x k The sparse representation coefficient of Then x k The sparse representation of Whole image The sparse representation of is: in, represents a vector whose values ​​are all 1, and its dimension is 64. ". / " represents the division of the corresponding elements of two matrices. I' is The sparse representation of Indicates that the image block x k Put it back to the k position of the image; Compute the residual between the grayscale image and its sparse representation Extract residuals The mean, variance, kurtosis, skewness and information entropy of are used as visual perception features of underwater images.

4. The underwater image quality assessment method based on sparse representation and Transformer according to claim 1, characterized in that: The method described in step S2 is to use Transformer to extract the semantic features of each underwater image and characterize the semantic information of the underwater image. The specific method is: input each underwater image into a visual Transformer model, the visual Transformer model includes an encoder and a decoder, and the vector output by the visual Transformer model is used as the semantic feature of the underwater image.

5. The underwater image quality assessment method based on sparse representation and Transformer according to claim 4 is characterized in that: The step S2 specifically includes: dividing each underwater image into image blocks of a certain size, then position encoding the image blocks, and then inputting the image blocks and their position encodings into the encoder of the Transformer model to obtain the output of the encoder, which is expressed as: f i =encode(P,PE) Among them, f i represents the output of the encoder, P and PE represent all image blocks and their position encodings, and encoder(·) represents the encoder of the Transformer model; f i Input the decoder of the Transformer model, and the output of the decoder is used as the semantic feature of the underwater image, expressed as: f2=decoder(f i ) Among them, f2 represents the semantic features of underwater images, and decoder(·) is the decoder of Transformer.

6. The underwater image quality assessment method based on sparse representation and Transformer according to claim 4, characterized in that: Step S3 specifically includes: concatenating the visual perception features and the semantic features, expressed as: Among them, f1 and f2 represent the visual perception features and semantic features of underwater images respectively. represents the splicing operation, and then f is input into the multi-layer perceptron to build an image quality assessment model.

7. The underwater image quality assessment method based on sparse representation and Transformer according to claim 1, characterized in that: The underwater image quality assessment model described in step S4 is trained end-to-end, and the specific method is as follows: the image quality assessment model is trained using the collected N underwater images and their corresponding subjective quality scores.

8. The underwater image quality assessment method based on sparse representation and Transformer according to claim 7, characterized in that: Step S4 specifically includes: inputting each underwater image into the constructed underwater image quality evaluation model to obtain the quality score predicted by the model, and training the image quality evaluation model using the mean square error, which is expressed as: Among them, q i is the quality score of the i-th image predicted by the model, s i It represents the subjective quality score of the i-th image, and the model is trained end-to-end using the Adam algorithm.

9. The underwater image quality assessment method based on sparse representation and Transformer according to claim 1, characterized in that: Step S5 specifically includes: for an underwater image I to be evaluated, input it into the trained model, and the model outputs its quality score, which is expressed as: q=M(I) Where M(·) represents the trained underwater image quality assessment model, and q represents the quality score of I.

Images