No-reference underwater image quality assessment method based on subject and scene semantic cues

Abstract

A reference-free underwater image quality assessment method based on subject and scene semantic cues includes: constructing an underwater image dataset; building an underwater image quality assessment model; fusing subject and scene text cue features; calculating the semantic similarity between the underwater image and the subject and scene text to obtain a semantic similarity score; training a subject and scene semantic similarity assessment module based on the semantic similarity score; extracting subject and scene visual cue features; fusing the subject and scene visual cue features with shallow and deep image features to obtain a second visual feature matrix; fusing quality grade, distortion type, and grade text features to obtain a quality and distortion text semantic feature matrix; extracting a quality and distortion visual feature matrix based on the second visual feature matrix, and calculating a similarity score between the quality and distortion visual feature matrix and the quality and distortion text semantic feature matrix as the image quality score. The method of the present invention is comprehensive and accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image processing, and in particular to a reference-free underwater image quality evaluation method based on subject and scene semantic cues. Background Art

[0002] Underwater imaging is widely used in fields such as marine scientific research and engineering monitoring. However, the absorption and scattering of light in underwater environments can cause image quality degradation, including color deviation, blurring, and low contrast. Water's selective absorption of light of different wavelengths causes loss of color information, while forward scattering and backscattering blur details and reduce contrast, respectively. Suspended particles further exacerbate image quality degradation.

[0003] CN118918086A discloses a method for underwater image quality assessment guided by multiple priors. This method first constructs a dataset containing multiple sets of underwater images with the same scene but different qualities, and then builds a network model with serial feature extraction, parallel feature interaction modules, and linear mapping layers. Prior maps are generated using color, contrast, and detail prior modules to guide feature extraction, and a twin network is trained to ultimately achieve underwater image quality assessment. However, the multiple priors introduced by this method may increase model complexity, leading to increased computational costs, and there is a dependency on specific scenarios.

[0004] CN119295397A discloses a system and method for underwater image quality assessment using joint learning of luminance and chrominance. This method first converts underwater images from RGB space to CIELAB space, which is more relevant to human subjective visual perception. Features such as histograms, morphology, and moment statistics are then extracted and fused from the luminance and color components. Finally, support vector regression is used to learn the relationship between the fused features and the image quality score, thereby constructing a quality prediction model. However, this method relies on joint modeling of luminance and chrominance. Underwater images exhibit unique color distortion (such as a blue-green cast and color attenuation), and traditional color spaces may be unable to effectively separate the degradation features of luminance and chrominance.

[0005] CN119048515B discloses an underwater image quality assessment method based on weighted fusion analysis of image features. This method first obtains an optimized underwater image and preprocesses it to a fixed size. It then converts the image from RGB to Lab color space, calculates the variance and mean of the a and b channels, and uses logarithmic operations to derive a color feature evaluation value. Multi-scale and multi-directional Log-Gabor filtering is then performed to calculate the LPC intensity and combine it with weights to obtain a clarity feature evaluation value. Finally, these two values ​​are linearly weighted with weight coefficients fitted from subjective scores to obtain an image quality evaluation score. However, this method relies on weighted fusion of manually extracted multidimensional features. The degradation factors of underwater images are highly complex, and a single feature combination cannot fully cover all distortion types.

[0006] More importantly, existing methods suffer from a lack of semantic understanding, insufficient scene adaptation, and a limited quality perception. They fail to fully utilize the semantic information of both the subject (e.g., fish, equipment) and the scene (e.g., coral reefs, seabed topography) in underwater images to comprehensively assess image quality, limiting model performance. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned background technology and provide a reference-free underwater image quality evaluation method based on subject and scene semantic cues, which uses the semantic information of the subject and scene in the underwater image to evaluate the overall image quality, effectively improving the performance of the model and making the evaluation method more comprehensive and accurate.

[0008] The technical solution adopted by the present invention to solve the technical problem is a reference-free underwater image quality assessment method based on subject and scene semantic cues, comprising the following steps:

[0009] Step S1: constructing learnable text prompt information for underwater images and constructing an underwater image dataset; the learnable text prompt information includes subject text prompt information, scene text prompt information, quality level text prompt information, distortion type and level text prompt information corresponding to the underwater image;

[0010] Step S2: constructing an underwater image quality assessment model, which includes a subject and scene semantic similarity assessment module and an image quality assessment module, wherein the subject and scene semantic similarity assessment module is connected to the image quality assessment module;

[0011] Step S3: Input the underwater images in the underwater image dataset into the subject and scene semantic similarity assessment module to extract shallow features and deep features of the images; fuse the shallow features and the deep features to obtain a first visual feature matrix I1; input the subject text prompt information and the scene text prompt information corresponding to the underwater images in the underwater image dataset into the subject and scene semantic similarity assessment module to extract the subject text prompt features and the scene text prompt features; fuse the subject text prompt features and the scene text prompt features to obtain the subject and scene text semantic feature matrix T1;

[0012] Step S4: Calculate the semantic similarity between the underwater image and the subject and scene text based on the first visual feature matrix I1 and the subject and scene text semantic feature matrix T1 to obtain a semantic similarity score;

[0013] Step S5: training the subject and scene semantic similarity assessment module according to the semantic similarity score to obtain a trained subject and scene semantic similarity assessment module;

[0014] Step S6: Input the underwater images in the underwater image dataset into the trained subject and scene semantic similarity evaluation module to extract subject visual cue features and scene visual cue features;

[0015] Step S7: Input the underwater images in the underwater image dataset into the image quality assessment module to extract the shallow features and deep features of the images; fuse the subject visual cue features, the scene visual cue features, the shallow features of the images, and the deep features of the images to obtain a second visual feature matrix I2; input the quality grade text prompt information, the distortion type, and the grade text prompt information corresponding to the underwater images in the underwater image dataset into the image quality assessment module, extract the quality grade text features, the distortion type, and the grade text features, and fuse the quality grade text features, the distortion type, and the grade text features to obtain a quality and distortion text semantic feature matrix T2;

[0016] Step S8: Extract the quality and distortion visual feature matrix K1 based on the second visual feature matrix I2, calculate the similarity score between the quality and distortion visual feature matrix K1 and the quality and distortion text semantic feature matrix T2, and use the similarity score as the final image quality score .

[0017] Further, in step S1, collect MIn this paper, we propose a method for constructing underwater original images, adding 6 distortion types to each original underwater image, and adding 10 distortion levels to each distortion type. We construct multiple underwater images with different distortion conditions, score each underwater image with different distortion conditions, and obtain multiple underwater images with subjective quality score labels. We also construct learnable text prompt information for multiple underwater images with subjective quality score labels, and obtain an underwater image dataset.

[0018] Furthermore, the subjects corresponding to underwater images are divided into fish, shipwrecks, divers, and anchors; the scenes corresponding to underwater images are divided into reefs, corals, sunken stones, and sediments; the quality level is divided into 5 levels according to the quality score, namely very poor, poor, fair, good, and perfect; the distortion types are divided into 6 types, namely blur distortion, JPEG compression distortion, Gaussian noise distortion, fog distortion, low brightness distortion, and turbidity distortion; each distortion type is divided into 10 distortion levels, represented by 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10, which represent the degree of distortion from low to high, with level 1 indicating the lightest distortion and level 10 indicating the most severe distortion.

[0019] Furthermore, the subject and scene semantic similarity assessment module includes a first text semantic feature extraction module, a first semantic feature fusion module, a first image visual feature extraction module, a first visual feature fusion module, a sparse perception similarity calculation module, and a pre-training loss calculation module. The first text semantic feature extraction module is connected to the first semantic feature fusion module, and the first semantic feature fusion module is connected to the sparse perception similarity calculation module; the first image visual feature extraction module is connected to the first visual feature fusion module, the first visual feature fusion module is connected to the sparse perception similarity calculation module, the sparse perception similarity calculation module is connected to the pre-training loss calculation module, and the pre-training loss calculation module is connected to the first image visual feature extraction module;

[0020] The first image visual feature extraction module is used to extract the shallow image features, deep image features, subject visual cue features, and scene visual cue features of underwater images in the underwater image dataset;

[0021] The first visual feature fusion module is used to fuse shallow features and deep features to obtain a first visual feature matrix I1;

[0022] The first text semantic feature extraction module is used to extract the main text prompt features and scene text prompt features corresponding to the underwater image;

[0023] The first semantic feature fusion module is used to fuse the subject text prompt feature and the scene text prompt feature to obtain the subject and scene text semantic feature matrix T1;

[0024] The sparse-aware similarity calculation module is used to calculate the semantic similarity between the underwater image and the subject and scene text to obtain the semantic similarity score;

[0025] The pre-training loss calculation module is used to calculate the loss according to the semantic similarity score, and back-propagate the calculated loss to update the first image visual feature extraction module.

[0026] Furthermore, the image quality assessment module includes a second text semantic feature extraction module, a second semantic feature fusion module, a second image visual feature extraction module, a second visual feature fusion module, a quality and distortion visual feature extraction module, and a maximization similarity calculation module. The first image visual feature extraction module is connected to the second image visual feature extraction module, the second image visual feature extraction module is connected to the second visual feature fusion module, the second visual feature fusion module is connected to the quality and distortion visual feature extraction module, and the quality and distortion visual feature extraction module is connected to the maximization similarity calculation module; the second text semantic feature extraction module is connected to the second semantic feature fusion module, and the second semantic feature fusion module is connected to the maximization similarity calculation module.

[0027] The second text semantic feature extraction module is used to extract quality grade text features, distortion type and grade text features;

[0028] The second semantic feature fusion module is used to fuse the quality grade text features, distortion type and grade text features to obtain the quality and distortion text semantic feature matrix T2;

[0029] The second image visual feature extraction module is used to extract shallow image features and deep image features of underwater images in the underwater image dataset;

[0030] The second visual feature fusion module is used to fuse the subject visual cue features, the scene visual cue features, the image shallow features, and the image deep features to obtain the second visual feature matrix I2;

[0031] The quality and distortion visual feature extraction module is used to extract the quality and distortion visual feature matrix K1 according to the second visual feature matrix I2;

[0032] The maximization similarity calculation module is used to calculate the similarity score between the quality and distortion visual feature matrix K1 and the quality and distortion text semantic feature matrix T2.

[0033] Furthermore, in steps S3 and S7, the shallow image features include clarity, sharpness, contrast, noise, and color shift of the underwater image; and the deep image features include image content features, image structure features, and contextual association features.

[0034] Furthermore, the sparse-perception similarity calculation module includes a text multi-head self-attention module, a first multi-scale pooling module, a visual self-attention module, a second multi-scale pooling module, a cross-modal semantic interaction module, a feature fusion module, and a sparse processing module. The text multi-head self-attention module is connected to the first multi-scale pooling module, the visual self-attention module is connected to the second multi-scale pooling module, the first multi-scale pooling module and the second multi-scale pooling module are connected to the cross-modal semantic interaction module, the cross-modal semantic interaction module is connected to the feature fusion module, and the feature fusion module is connected to the sparse processing module.

[0035] Furthermore, the process of calculating semantic similarity by the sparse perception similarity calculation module is as follows: the subject and scene text semantic feature matrix T1 is enhanced by the text multi-head self-attention module to obtain the enhanced text semantic feature T1_att, and the text semantic feature T1_att is decomposed into the global text semantic feature T1_global and the local text semantic feature T1_local by the first multi-scale pooling module; the first visual feature matrix I1 is enhanced by the image visual self-attention module to obtain the enhanced image visual feature I1_att, and the image visual feature I1_att is decomposed into the global image visual feature I1_global and the local image visual feature I1_local by the second multi-scale pooling module; the global text semantic feature T1_global, the local text semantic feature T1_local, the global image visual feature I1_global, and the local image visual feature I1_local are passed through the cross-modal semantic interaction module to calculate the semantic similarity between the global text semantic feature T1_global and the global image visual feature I1_global by the dot product method to obtain the global semantic interaction feature T1_global I1_global, calculates the semantic similarity between the local text semantic feature T1_local and the local image visual feature I1_local by dot product, and obtains the local semantic interaction feature T1_local I1_local, global semantic interaction feature T1_global I1_global, local semantic interaction feature T1_local I1_local is spliced ​​and fused through the feature fusion module to obtain fused features. The fused features are then processed by the sparse processing module, which dynamically adjusts the threshold based on the text-image semantic complexity. The fused features are sparsely screened to obtain sparse features. Finally, the sparse features are globally pooled and the semantic similarity score is output to achieve deep semantic association.

[0036] Furthermore, in step S8, the method for calculating the similarity score between the quality and distortion visual feature matrix K1 and the quality and distortion text semantic feature matrix T2 is as follows:

[0037] Assume that the semantic feature matrix of quality and distortion text is , quality and distortion visual feature matrix ,in m is the quality and number of semantic features of distorted text, n is the quality and number of distorted visual features, d is the feature dimension, R represents the set of real numbers;

[0038] First, calculate the cosine similarity matrix C, the formula is as follows:

[0039] ;

[0040] in Represents matrix multiplication;

[0041] Second, all values ​​in the cosine similarity matrix C are averaged to get the similarity score:

[0042] ;

[0043] in, Represents the first cosine similarity matrix C j elements, i Represents the number of all values ​​in the cosine similarity matrix C.

[0044] Compared with the prior art, the advantages of the present invention are as follows:

[0045] (1) The present invention combines the characteristics of the underwater environment, fully explores the information of the subject and scene in the underwater image, and uses the semantic information of the subject and scene in the underwater image in combination with other image features to evaluate the comprehensive quality of the image, which effectively improves the performance of the model and makes the evaluation method more comprehensive, accurate and reliable; the model has low complexity and low computational cost, and does not need to rely on specific scenes; it can effectively extract the degradation features of brightness and chroma, and can cover all types of distortion.

[0046] (2) The present invention can introduce a sparse perception similarity calculation module, which accurately screens the subject and scene information and effectively integrates the two, thereby preventing the model from being overly distracted when perceiving image quality, making image perception more efficient and focused. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a principle diagram of the method of the embodiment of the present invention.

[0048] Figure 2 yes Figure 1 A schematic structural diagram of the underwater image quality evaluation model according to the embodiment shown.

[0049] Figure 3yes Figure 1 A schematic structural diagram of the sparse-aware similarity calculation module in the illustrated embodiment.

[0050] Figure 4 yes Figure 1 The diagram shows the calculation principle of the sparse-aware similarity calculation module in the embodiment shown. DETAILED DESCRIPTION

[0051] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0052] Reference Figure 1 :The method of the present invention comprises the following steps:

[0053] Step S1: constructing an underwater image dataset. The specific method is as follows: collecting 60 underwater original images, adding 6 distortion types to each underwater original image, adding 10 distortion levels to each distortion type, constructing multiple underwater images with different distortion conditions, scoring each underwater image with different distortion conditions, obtaining multiple underwater images with subjective quality score labels, constructing learnable text prompt information for the multiple underwater images with subjective quality score labels, and obtaining an underwater image dataset. The learnable text prompt information includes subject text prompt information, scene text prompt information, quality level text prompt information, distortion type and level text prompt information corresponding to the underwater image.

[0054] The subjects corresponding to underwater images are divided into fish, shipwrecks, divers, and anchors; the scenes corresponding to underwater images are divided into reefs, corals, sunken stones, and sediments; the quality level is divided into 5 levels based on the quality score, namely very poor, poor, fair, good, and perfect; there are 6 types of distortion, namely blur distortion, JPEG compression distortion, Gaussian noise distortion, fog distortion, low brightness distortion, and turbidity distortion; each distortion type is divided into 10 distortion levels, represented by 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10, which represent the degree of distortion from low to high, with level 1 indicating the lightest distortion and level 10 indicating the most severe distortion.

[0055] Step S2: Construct underwater image quality evaluation model:

[0056] Reference Figure 2The underwater image quality evaluation model includes a subject and scene semantic similarity evaluation module and an image quality evaluation module. The subject and scene semantic similarity evaluation module is connected to the image quality evaluation module. The subject and scene semantic similarity evaluation module includes a first text semantic feature extraction module, a first semantic feature fusion module, a first image visual feature extraction module, a first visual feature fusion module, a sparse perception similarity calculation module, and a pre-training loss calculation module. The image quality evaluation module includes a second text semantic feature extraction module, a second semantic feature fusion module, a second image visual feature extraction module, a second visual feature fusion module, a quality and distortion visual feature extraction module, and a maximization similarity calculation module. The first text semantic feature extraction module is connected to the first semantic feature fusion module, and the first semantic feature fusion module is connected to the sparse perception similarity calculation module. The sparse perception similarity calculation module is connected; the first image visual feature extraction module is connected to the first visual feature fusion module, the first visual feature fusion module is connected to the sparse perception similarity calculation module, the sparse perception similarity calculation module is connected to the pre-training loss calculation module, and the pre-training loss calculation module is connected to the first image visual feature extraction module; the first image visual feature extraction module is connected to the second image visual feature extraction module, the second image visual feature extraction module is connected to the second visual feature fusion module, the second visual feature fusion module is connected to the quality and distortion visual feature extraction module, and the quality and distortion visual feature extraction module is connected to the maximization similarity calculation module; the second text semantic feature extraction module is connected to the second semantic feature fusion module, and the second semantic feature fusion module is connected to the maximization similarity calculation module.

[0057] The first image visual feature extraction module is used to extract the shallow image features, deep image features, subject visual cue features, and scene visual cue features of underwater images in the underwater image dataset;

[0058] The first visual feature fusion module is used to fuse shallow features and deep features to obtain a first visual feature matrix I1;

[0059] The first text semantic feature extraction module is used to extract the main text prompt features and scene text prompt features corresponding to the underwater image;

[0060] The first semantic feature fusion module is used to fuse the subject text prompt feature and the scene text prompt feature to obtain the subject and scene text semantic feature matrix T1;

[0061] The sparse-aware similarity calculation module is used to calculate the semantic similarity between the underwater image and the subject and scene text to obtain the semantic similarity score;

[0062] The pre-training loss calculation module is used to calculate the loss according to the semantic similarity score, and back-propagate the calculated loss to update the first image visual feature extraction module.

[0063] The second text semantic feature extraction module is used to extract quality grade text features, distortion type and grade text features;

[0064] The second semantic feature fusion module is used to fuse the quality grade text features, distortion type and grade text features to obtain the quality and distortion text semantic feature matrix T2;

[0065] The second image visual feature extraction module is used to extract shallow image features and deep image features of underwater images in the underwater image dataset;

[0066] The second visual feature fusion module is used to fuse the subject visual cue features, the scene visual cue features, the image shallow features, and the image deep features to obtain the second visual feature matrix I2;

[0067] The quality and distortion visual feature extraction module is used to extract the quality and distortion visual feature matrix K1 according to the second visual feature matrix I2;

[0068] The maximization similarity calculation module is used to calculate the similarity score between the quality and distortion visual feature matrix K1 and the quality and distortion text semantic feature matrix T2.

[0069] Step S3: Input the underwater images in the underwater image dataset into the first image visual feature extraction module to extract the shallow features and deep features of the image; use the first visual feature fusion module to fuse the shallow features and deep features to obtain the first visual feature matrix I1;

[0070] The shallow features of the image include basic quality attributes such as clarity, sharpness, and contrast of underwater images, as well as local distortion features such as noise and color shift; the deep features of the image include image content features, image structure features, and contextual association features.

[0071] Input the subject text prompt information and scene text prompt information corresponding to the underwater image in the underwater image dataset into the first text semantic feature extraction module, extract the subject text prompt features and the scene text prompt features, and fuse the subject text prompt features and the scene text prompt features through the first semantic feature fusion module to obtain the subject and scene text semantic feature matrix T1;

[0072] Step S4: using the sparse perception similarity calculation module, the semantic similarity between the underwater image and the subject and scene text is calculated based on the first visual feature matrix I1 and the subject and scene text semantic feature matrix T1 to obtain a semantic similarity score;

[0073] Reference Figure 3The sparse-aware similarity calculation module includes a text multi-head self-attention module, a first multi-scale pooling module, a visual self-attention module, a second multi-scale pooling module, a cross-modal semantic interaction module, a feature fusion module, and a sparse processing module. The text multi-head self-attention module is connected to the first multi-scale pooling module, the visual self-attention module is connected to the second multi-scale pooling module, the first multi-scale pooling module and the second multi-scale pooling module are connected to the cross-modal semantic interaction module, the cross-modal semantic interaction module is connected to the feature fusion module, and the feature fusion module is connected to the sparse processing module.

[0074] Reference Figure 4 ,The sparse-aware similarity calculation module calculates the semantic similarity process as follows:

[0075] The process of calculating semantic similarity in the sparse perception similarity calculation module is as follows: the subject and scene text semantic feature matrix T1 is enhanced by the text multi-head self-attention module to obtain the enhanced text semantic feature T1_att, and the text semantic feature T1_att is decomposed into the global text semantic feature T1_global and the local text semantic feature T1_local by the first multi-scale pooling module; the first visual feature matrix I1 is enhanced by the image visual self-attention module to obtain the enhanced image visual feature I1_att, and the image visual feature I1_att is decomposed into the global image visual feature I1_global and the local image visual feature I1_local by the second multi-scale pooling module; the global text semantic feature T1_global, the local text semantic feature T1_local, the global image visual feature I1_global, and the local image visual feature I1_local are passed through the cross-modal semantic interaction module to calculate the semantic similarity between the global text semantic feature T1_global and the global image visual feature I1_global by the dot product method, and obtain the global semantic interaction feature T1_global I1_global, calculates the semantic similarity between the local text semantic feature T1_local and the local image visual feature I1_local by dot product, and obtains the local semantic interaction feature T1_local I1_local, global semantic interaction feature T1_global I1_global, local semantic interaction feature T1_local I1_local is spliced ​​and fused through the feature fusion module to obtain fused features. The fused features are then processed by the sparse processing module, which dynamically adjusts the threshold based on the text-image semantic complexity. The fused features are sparsely screened to obtain sparse features. Finally, the sparse features are globally pooled and the semantic similarity score is output to achieve deep semantic association.

[0076] Step S5: Train the first image visual feature extraction module according to the semantic similarity score to obtain a trained first image visual feature extraction module. Specifically, the pre-training loss calculation module calculates the loss according to the semantic similarity score, and back-propagates the calculated loss to update the first image visual feature extraction module.

[0077] A batch of underwater image samples contains X positive pairs and Y negative pairs. In this example, X = 3600 and Y = 10800. A positive pair is a combination of samples where the text prompt and the underwater image content match in semantic labeling, while a negative pair is a combination where the text prompt and the underwater image content do not match in semantic labeling. For example, if the subject in the text prompt is a school of fish and the scene is coral, then an image containing fish and coral in the actual image content is a positive pair with the text prompt. Conversely, an image containing neither fish nor coral in the actual image content is a negative pair with the text prompt.

[0078] Assume that the positive sample pair includes text prompt T and matching image , output semantic similarity score , negative sample pairs include text prompts T, mismatched images , output semantic similarity score , then the pre-training loss function is:

[0079] ;

[0080] The hyperparameters m =1.0, It is i The semantic similarity score of the positive sample pairs, It is i Before the formal evaluation by the image quality assessment module, the pre-trained loss calculation module in the subject and scene semantic similarity assessment module is used to guide the pre-training of the first image visual feature extraction module. This allows the first image visual feature extraction module to first learn to capture features that match the text semantics. The pre-trained first image visual feature extraction module is then used for image quality assessment tasks to improve the feature foundation.

[0081] The subject and scene semantic similarity evaluation module is continuously optimized through the pre-training loss function, and the first image visual feature extraction module is trained to obtain the subject visual cues and scene visual cues. The ultimate training goal is to make the subject visual cues and scene visual cues include and only include the subject and scene visual features of all underwater images, so that the image quality evaluation module focuses on the subject and scene information of the image, and maximizes the assistance to the image quality evaluation of the subsequent image quality evaluation module.

[0082] Step S6: Input the underwater images in the underwater image dataset into the trained first image visual feature extraction module to extract subject visual cue features and scene visual cue features.

[0083] Step S7: Input the underwater images in the underwater image dataset into the second image visual feature extraction module to extract the shallow features and deep features of the image; use the second visual feature fusion module to fuse the subject visual cue features, scene visual cue features, image shallow features, and image deep features to obtain the second visual feature matrix I2.

[0084] The quality grade text prompt information, distortion type and grade text prompt information corresponding to the underwater images in the underwater image dataset are input into the second text semantic feature extraction module to extract the quality grade text features, distortion type and grade text features. The quality grade text features, distortion type and grade text features are fused through the second semantic feature fusion module to obtain the quality and distortion text semantic feature matrix T2.

[0085] Step S8: Extract the quality and distortion visual feature matrix K1 based on the second visual feature matrix I2, calculate the similarity score between the quality and distortion visual feature matrix K1 and the quality and distortion text semantic feature matrix T2, and use the similarity score as the final image quality score .

[0086] The method for calculating the similarity score between the quality and distortion visual feature matrix K1 and the quality and distortion text semantic feature matrix T2 is as follows:

[0087] Assume that the semantic feature matrix of quality and distortion text is , quality and distortion visual feature matrix ,in m is the quality and number of semantic features of distorted text, n is the quality and number of distorted visual features, d is the feature dimension, R represents the set of real numbers;

[0088] First, calculate the cosine similarity matrix C, the formula is as follows:

[0089] ;

[0090] in Represents matrix multiplication;

[0091] Second, all values ​​in the cosine similarity matrix C are averaged to get the similarity score:

[0092] ;

[0093] in, Represents the first cosine similarity matrix C j elements, i Represents the number of all values ​​in the cosine similarity matrix C.

[0094] This example uses a dataset constructed based on the underwater image dataset UID2021 to verify the effectiveness and robustness of the proposed no-reference underwater image quality assessment method based on subject and scene semantic cues. Several classic underwater image quality assessment methods are also selected for experimental comparison. The experimental results are shown in Table 1.

[0095] Table 1. Comparison of experimental results of different underwater image quality evaluation methods

[0096] ;

[0097] The evaluation indicators selected in the experiment are Spearman rank correlation coefficient (SRCC), Pearson linear correlation coefficient (PLCC) and root mean square error (RMSE). The larger the SRCC and PLCC values ​​are and the smaller the RMSE value is, the better the performance of the algorithm is.

[0098] In Table 1, CCF, UIQM, UCIQE, and FDUM represent existing underwater image quality assessment methods. CCF stands for "An imaging-inspired no-reference underwater color image quality assessment metric"; UIQM stands for "Human-visual-system inspired underwater image quality measures"; UCIQE stands for "An Underwater Color Image Quality Evaluation Metric"; and FDUM stands for "A reference-free underwater image quality assessment metric in the frequency domain." As can be seen from Table 1, the proposed method performs better than CCF, UIQM, UCIQE, and FDUM in terms of Spearman's rank correlation coefficient, Spearman's rank correlation coefficient, and root mean square error.

[0099] Those skilled in the art may make various modifications and variations to the present invention. If these modifications and variations are within the scope of the claims of the present invention and their equivalents, then these modifications and variations are also within the scope of protection of the present invention.

[0100] The contents not described in detail in the specification are prior art known to those skilled in the art.

Claims

1. A no-reference underwater image quality assessment method based on subject and scene semantic cues, characterized by: The following steps are involved: Step S1: constructing learnable text prompt information for underwater images and constructing an underwater image dataset; The learnable text prompt information includes the subject text prompt information, scene text prompt information, quality level text prompt information, distortion type and level text prompt information corresponding to the underwater image; Step S2: constructing an underwater image quality assessment model, which includes a subject and scene semantic similarity assessment module and an image quality assessment module, wherein the subject and scene semantic similarity assessment module is connected to the image quality assessment module; Step S3: Input the underwater images in the underwater image dataset into the subject and scene semantic similarity assessment module to extract shallow features and deep features of the images; fuse the shallow features and the deep features to obtain a first visual feature matrix I1; input the subject text prompt information and the scene text prompt information corresponding to the underwater images in the underwater image dataset into the subject and scene semantic similarity assessment module to extract the subject text prompt features and the scene text prompt features; fuse the subject text prompt features and the scene text prompt features to obtain the subject and scene text semantic feature matrix T1; Step S4: Calculate the semantic similarity between the underwater image and the subject and scene text based on the first visual feature matrix I1 and the subject and scene text semantic feature matrix T1 to obtain a semantic similarity score; Step S5: training the subject and scene semantic similarity assessment module according to the semantic similarity score to obtain a trained subject and scene semantic similarity assessment module; Step S6: Input the underwater images in the underwater image dataset into the trained subject and scene semantic similarity evaluation module to extract subject visual cue features and scene visual cue features; Step S7: Input the underwater images in the underwater image dataset into the image quality assessment module to extract the shallow features and deep features of the images; fuse the subject visual cue features, the scene visual cue features, the shallow features of the images, and the deep features of the images to obtain a second visual feature matrix I2; input the quality grade text prompt information, the distortion type, and the grade text prompt information corresponding to the underwater images in the underwater image dataset into the image quality assessment module, extract the quality grade text features, the distortion type, and the grade text features, and fuse the quality grade text features, the distortion type, and the grade text features to obtain a quality and distortion text semantic feature matrix T2; Step S8: Extract the quality and distortion visual feature matrix K1 based on the second visual feature matrix I2, calculate the similarity score between the quality and distortion visual feature matrix K1 and the quality and distortion text semantic feature matrix T2, and use the similarity score as the final image quality score .

2. The no-reference underwater image quality assessment method based on subject and scene semantic cues according to claim 1, characterized in that: In step S1, collect M In this paper, we propose a method for constructing underwater original images, adding 6 distortion types to each original underwater image, and adding 10 distortion levels to each distortion type. We construct multiple underwater images with different distortion conditions, score each underwater image with different distortion conditions, and obtain multiple underwater images with subjective quality score labels. We also construct learnable text prompt information for multiple underwater images with subjective quality score labels, and obtain an underwater image dataset.

3. The no-reference underwater image quality assessment method based on subject and scene semantic cues according to claim 2, characterized in that: The subjects corresponding to underwater images are divided into fish, shipwrecks, divers, and anchors; the scenes corresponding to underwater images are divided into reefs, corals, sunken stones, and sediments; the quality level is divided into 5 levels based on the quality score, namely very poor, poor, fair, good, and perfect; there are 6 types of distortion, namely blur distortion, JPEG compression distortion, Gaussian noise distortion, fog distortion, low brightness distortion, and turbidity distortion; each distortion type is divided into 10 distortion levels, represented by 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10, which represent the degree of distortion from low to high, with level 1 indicating the lightest distortion and level 10 indicating the most severe distortion.

4. The method for no-reference underwater image quality assessment based on subject and scene semantic cues according to claim 1, characterized in that: The subject and scene semantic similarity assessment module includes a first text semantic feature extraction module, a first semantic feature fusion module, a first image visual feature extraction module, a first visual feature fusion module, a sparse perception similarity calculation module, and a pre-training loss calculation module. The first text semantic feature extraction module is connected to the first semantic feature fusion module, and the first semantic feature fusion module is connected to the sparse perception similarity calculation module; the first image visual feature extraction module is connected to the first visual feature fusion module, the first visual feature fusion module is connected to the sparse perception similarity calculation module, the sparse perception similarity calculation module is connected to the pre-training loss calculation module, and the pre-training loss calculation module is connected to the first image visual feature extraction module; The first image visual feature extraction module is used to extract the shallow image features, deep image features, subject visual cue features, and scene visual cue features of underwater images in the underwater image dataset; The first visual feature fusion module is used to fuse shallow features and deep features to obtain a first visual feature matrix I1; The first text semantic feature extraction module is used to extract the main text prompt features and scene text prompt features corresponding to the underwater image; The first semantic feature fusion module is used to fuse the subject text prompt feature and the scene text prompt feature to obtain the subject and scene text semantic feature matrix T1; The sparse-aware similarity calculation module is used to calculate the semantic similarity between the underwater image and the subject and scene text to obtain the semantic similarity score; The pre-training loss calculation module is used to calculate the loss according to the semantic similarity score, and back-propagate the calculated loss to update the first image visual feature extraction module.

5. The method for no-reference underwater image quality assessment based on subject and scene semantic cues according to claim 4, characterized in that: The image quality assessment module includes a second text semantic feature extraction module, a second semantic feature fusion module, a second image visual feature extraction module, a second visual feature fusion module, a quality and distortion visual feature extraction module, and a maximization similarity calculation module. The first image visual feature extraction module is connected to the second image visual feature extraction module, the second image visual feature extraction module is connected to the second visual feature fusion module, the second visual feature fusion module is connected to the quality and distortion visual feature extraction module, and the quality and distortion visual feature extraction module is connected to the maximization similarity calculation module; the second text semantic feature extraction module is connected to the second semantic feature fusion module, and the second semantic feature fusion module is connected to the maximization similarity calculation module; The second text semantic feature extraction module is used to extract quality grade text features, distortion type and grade text features; The second semantic feature fusion module is used to fuse the quality grade text features, distortion type and grade text features to obtain the quality and distortion text semantic feature matrix T2; The second image visual feature extraction module is used to extract shallow image features and deep image features of underwater images in the underwater image dataset; The second visual feature fusion module is used to fuse the subject visual cue features, the scene visual cue features, the image shallow features, and the image deep features to obtain the second visual feature matrix I2; The quality and distortion visual feature extraction module is used to extract the quality and distortion visual feature matrix K1 according to the second visual feature matrix I2; The maximization similarity calculation module is used to calculate the similarity score between the quality and distortion visual feature matrix K1 and the quality and distortion text semantic feature matrix T2.

6. The method for no-reference underwater image quality assessment based on subject and scene semantic cues according to claim 1, characterized in that: In steps S3 and S7, the shallow image features include clarity, sharpness, contrast, noise, and color shift of the underwater image; the deep image features include image content features, image structure features, and contextual association features.

7. The method for no-reference underwater image quality assessment based on subject and scene semantic cues according to claim 4, characterized in that: The sparse perception similarity calculation module includes a text multi-head self-attention module, a first multi-scale pooling module, a visual self-attention module, a second multi-scale pooling module, a cross-modal semantic interaction module, a feature fusion module, and a sparse processing module. The text multi-head self-attention module is connected to the first multi-scale pooling module, the visual self-attention module is connected to the second multi-scale pooling module, the first multi-scale pooling module and the second multi-scale pooling module are connected to the cross-modal semantic interaction module, the cross-modal semantic interaction module is connected to the feature fusion module, and the feature fusion module is connected to the sparse processing module.

8. The method for no-reference underwater image quality assessment based on subject and scene semantic cues according to claim 7, characterized in that: The process of calculating semantic similarity in the sparse perception similarity calculation module is as follows: the subject and scene text semantic feature matrix T1 is enhanced by the text multi-head self-attention module to obtain the enhanced text semantic feature T1_att, and the text semantic feature T1_att is decomposed into the global text semantic feature T1_global and the local text semantic feature T1_local by the first multi-scale pooling module; the first visual feature matrix I1 is enhanced by the image visual self-attention module to obtain the enhanced image visual feature I1_att, and the image visual feature I1_att is decomposed into the global image visual feature I1_global and the local image visual feature I1_local by the second multi-scale pooling module; the global text semantic feature T1_global, the local text semantic feature T1_local, the global image visual feature I1_global, and the local image visual feature I1_local are passed through the cross-modal semantic interaction module to calculate the semantic similarity between the global text semantic feature T1_global and the global image visual feature I1_global by the dot product method, and obtain the global semantic interaction feature T1_global I1_global, calculates the semantic similarity between the local text semantic feature T1_local and the local image visual feature I1_local by dot product, and obtains the local semantic interaction feature T1_local I1_local, global semantic interaction feature T1_global I1_global, local semantic interaction feature T1_local I1_local is spliced ​​and fused by the feature fusion module to obtain the fusion feature, and the fusion feature is then Through the sparse processing module, the threshold is dynamically adjusted in combination with the text-image semantic complexity, and the fused features are sparsely screened to obtain sparse features. Finally, the sparse features are globally pooled and the semantic similarity score is output to achieve deep semantic association.

9. The no-reference underwater image quality assessment method based on subject and scene semantic cues according to claim 1, characterized in that: In step S8, the method for calculating the similarity score between the quality and distortion visual feature matrix K1 and the quality and distortion text semantic feature matrix T2 is as follows: Assume that the semantic feature matrix of quality and distortion text is , quality and distortion visual feature matrix ,in m is the quality and number of semantic features of distorted text, n is the quality and number of distorted visual features, d is the feature dimension, R represents the set of real numbers; First, calculate the cosine similarity matrix C, the formula is as follows: ； in Represents matrix multiplication; Second, all values ​​in the cosine similarity matrix C are averaged to get the similarity score: ； in, Represents the first cosine similarity matrix C j elements, i Represents the number of all values ​​in the cosine similarity matrix C.

Images