Content and distortion bilingual guidance-based non-reference image quality evaluation model construction method

Abstract

The invention discloses a content and distortion bilingual guidance-based no-reference image quality evaluation model construction method, which comprises the following steps of: designing two semantic description models of content text description and distortion text description by utilizing a pre-training model CLIP of visual language comparative learning, respectively extracting content features and distortion features of an image by two branches of CNN and Transformer, and constructing a non-reference image quality evaluation model based on content and distortion bilingual guidance. Carrying out cosine similarity calculation on the distortion characteristics of the image and the distortion type and quality grade text description characteristics to obtain the matching probability of the image with various distortion types and quality grades; combining the quality grade probability with a Like scale to predict a quality score; and respectively calculating a content type, a distortion type, quality score loss and weighted summation to guide model training. The semantic content sensitivity characteristics of a human visual system in quality perception are fully combined, and the semantic understanding ability and the quality evaluation accuracy of the model in a complex distortion situation are effectively enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image quality assessment, and specifically relates to a method for constructing a reference-free image quality assessment model based on dual semantic guidance of content and distortion. Background Art

[0002] With the rapid development of deep learning technology, researchers have begun designing deeper neural networks to automatically extract image features, which has achieved significant progress in multiple NR-IQA tasks. However, most of these methods focus on image degradation information at the perceptual level, and rarely consider the intrinsic connection between the semantic content of the image and human cognitive mechanisms. In fact, when evaluating image quality, humans often do not rely solely on low-level perceptual features, but instead integrate high-level information such as the objects and contextual semantics contained in the image. In recent years, with the rise of multimodal learning and visual language pre-training models, image quality assessment tasks have gradually introduced mechanisms for image semantic understanding. Among them, the CLIP (Contrastive Language-Image Pretraining) model, with its large-scale image and text processing capabilities, can align images and natural language into a unified semantic embedding space, thereby achieving a deeper perception of image content.

[0003] CNN-based image quality assessment methods are widely used in various vision tasks due to their excellent feature extraction capabilities. They have significant advantages in modeling local image details and structure. However, they struggle to capture dependencies between distant regions within an image. To address this deficiency, recent research has gradually introduced attention mechanisms, particularly multi-head self-attention, into vision tasks. This mechanism, originally proposed in the Transformer architecture, can dynamically model the mutual influence between different locations, significantly improving the model's global modeling capabilities in the spatial dimension. Although the Transformer has advantages in non-local modeling, it is weak in extracting local features, particularly in image texture analysis, which can cause its attention areas to deviate from critical quality information. Combining the Transformer's global feature extraction capabilities with the CNN's local detail feature extraction capabilities, fully considering the relationships and detailed features between image regions, shows great promise in NR-IQA tasks. Summary of the Invention

[0004] The present invention aims to solve one of the technical problems existing in the related art at least to a certain extent.

[0005] The purpose of the present invention is to provide a method for constructing a reference-free image quality assessment model based on dual semantic guidance of content and distortion. The method adopts the pre-training model CLIP of visual language contrastive learning, designs two semantic description models: content text description and distortion text description. The CNN and Transformer branches respectively extract the content features and distortion features of the image, compare and match them with the corresponding text descriptions, and perform image quality assessment based on the semantic guidance of content and distortion.

[0006] In order to achieve the above objectives, the present invention provides, on one hand, a method for constructing a no-reference image quality assessment model based on dual semantic guidance of content and distortion, comprising:

[0007] S1. Obtain an input image, input a text template statement describing the image content type based on the input image, and use the Text Transformer encoder of the CLIP model to extract content text features;

[0008] The local features of the input image are extracted through the CNN network, and the local features are input into the multi-head self-attention mechanism for global semantic understanding to obtain the content features of the input image;

[0009] Calculate the cosine similarity between the input image content features and the content text features to obtain the matching probability between the input and various content types;

[0010] S2. Input a text template sentence describing the image distortion type and quality level according to the input image, and extract distorted text features by the Text Transformer encoder of the CLIP model;

[0011] Input the input image to the Vision Transformer encoder to extract global features, and fuse them with local features through cross attention to obtain the distortion features of the input image;

[0012] Calculate the cosine similarity between the distortion features of the input image and the distorted text features to obtain the matching probability between the input image and various distortion types and quality levels;

[0013] The quality score is predicted based on the matching probability of the obtained quality levels combined with the Likert scale;

[0014] S3. Calculate the losses of content type, distortion type, and quality score respectively, and use the weighted sum as the loss function. Train the network framework constructed in steps S1-S2 to obtain an image quality evaluation model.

[0015] A further preferred technical solution of the present invention is that, in step S1, a text template statement describing the image content type is input according to the input image, and the text features of the content are extracted by the Text Transformer encoder of the CLIP model; specifically,

[0016] Enter a text template statement describing the image content type: , where c is the content type, which is defined as 9 types:

[0017]

[0018] According to the defined content text description, the Text Transformer encoder of the CLIP model extracts content text features .

[0019] Preferably, the step S1 extracts local features of the input image through the CNN network, inputs the local features into the multi-head self-attention mechanism for global semantic understanding, and obtains content features of the input image; specifically includes:

[0020] The input image passes through three convolutional layers in sequence. Each convolutional layer is connected to a batch normalization layer and an activation function. The preprocessed image features are then obtained by average pooling. The preprocessed image features are input into a four-layer residual network to extract the local features of the image. ,exist On the basis of [1], a multi-head self-attention mechanism is introduced to strengthen the key semantic areas and increase the understanding of image content, which is expressed as:

[0021]

[0022] Where Q, K, and V represent query, key, and value respectively. After adding the position embedded features, Represents the weight matrix, Concat represents the connection operation of the tensor, is the i-th attention head, expressed as:

[0023]

[0024] in , , , , is the corresponding weight matrix, is the feature dimension;

[0025] Multiple attention heads are connected and mapped to obtain image content features .

[0026] Preferably, the step S1 performs cosine similarity calculation on the content features of the input image and the content text features to obtain the matching probability between the input and various content types; specifically comprising:

[0027] The content features of the input image and content text features Perform contrastive learning to match the input image with its corresponding text and calculate the input image belongs to each content type The probability is calculated as:

[0028]

[0029] in Represents the similarity calculation function, Describe the corresponding text features for 9 different content types. ,when When Representation and image features Matching content text features, is the learnable temperature coefficient.

[0030] Preferably, in step S2, a text template statement describing the image distortion type and quality level is input according to the input image, and the distorted text features are extracted by the Text Transformer encoder of the CLIP model; specifically,

[0031] Enter the text template statement that describes the image distortion type and quality level: ;

[0032] Where d is the distortion type, which is defined as 11 types:

[0033]

[0034] a is the quality grade, which is defined as 5 types:

[0035]

[0036] A total of 11*5=55 distorted text descriptions are combined. According to the defined distorted text descriptions, the TextTransformer encoder of the CLIP model extracts distorted text features. .

[0037] Preferably, in step S2, the input image is input into the Vision Transformer encoder to extract global features, and the global features are fused with the local features through cross attention to obtain the distortion features of the input image; including:

[0038] The input image is divided into several 16×16 image blocks and mapped to the feature embedding space to obtain the initial embedding sequence;

[0039] After adding category tags and position encoding, the image is input into the Vision Transformer module containing a 12-layer encoder to extract global perceptual features used to characterize the overall quality of the image. ;

[0040] Adjust local features The dimension is obtained ,right and Perform cross attention to fuse local and global information, expressed as:

[0041]

[0042] in , , , is the characteristic dimension of Q, K, and V, is the distortion feature of the input image.

[0043] Preferably, the step S2 calculates the cosine similarity between the distortion features of the input image and the distorted text features to obtain the matching probability between the input image and various distortion types and quality levels; specifically comprising:

[0044] The distortion features of the input image and distorted text features Perform contrastive learning to obtain the joint probability distribution of distortion type and quality level, which is expressed as:

[0045]

[0046] in Represents the similarity calculation function, Describe the corresponding text features for 55 different distorted texts. ,when When Representation and image features Matched distorted text features, is the learnable temperature coefficient;

[0047] Marginalized Joint Probability Then we can get the probability of distortion type and quality level. and .

[0048] Preferably, the quality score is predicted based on the obtained matching probability of the quality level in combination with the Likert scale in step S2; specifically:

[0049] Likert scale with five quality levels, represented by the image Quality The predicted probability Calculate the quality score of the image using the following formula:

[0050]

[0051] in For images The prediction quality score of For images Quality The prediction probability of , a is one of five quality levels, indicating the degree of image quality distortion.

[0052] Preferably, in step S3, the losses of content type, distortion type and quality score are calculated respectively, and the weighted sum is used as the loss function, specifically:

[0053] Calculate the image content loss, image distortion loss, and image quality loss respectively. The calculation formula is:

[0054]

[0055]

[0056]

[0057] in Predict the probability for the image content type, is the predicted probability of image distortion type, is the predicted quality score of the image, is the true quality score of the image, and N is the number of training images;

[0058] The weighted sum of the three losses is used to guide model training, which is expressed as:

[0059]

[0060] in is the image content loss, is the image distortion loss, For image quality loss, is the weight coefficient.

[0061] Another aspect of the present invention provides a non-transitory computer-readable storage medium having computer instructions stored thereon, which enable a computer to execute the above-mentioned method for constructing a reference-free image quality assessment model based on dual semantic guidance of content and distortion.

[0062] Yet another aspect of the present invention provides an electronic device comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus, and the processor calls logic instructions in the memory to execute the above-mentioned method for constructing a reference-free image quality assessment model based on dual semantic guidance of content and distortion.

[0063] On the other hand, the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer executes the above-mentioned method for constructing a reference-free image quality assessment model based on dual semantic guidance of content and distortion.

[0064] Beneficial effects: The present invention is based on a method for constructing a no-reference image quality assessment model with dual semantic guidance of content and distortion, proposes a no-reference image quality assessment framework with dual semantic guidance, and designs differentiated semantic guidance mechanisms for the content features and distortion features of the image. Specifically, the model of the present invention constructs a dual-branch semantic guidance structure, and by introducing two types of text information, namely, image content type description statements and distortion type and quality level description statements, through the CLIP pre-training model, cross-modal semantic alignment of images and texts is achieved, thereby guiding the feature extraction process. This design fully combines the characteristics of the human visual system's sensitivity to semantic content in quality perception, and effectively enhances the model's semantic understanding ability and quality assessment accuracy in complex distortion scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Flowchart of the method for constructing a no-reference image quality assessment model based on dual semantic guidance of content and distortion;

[0066] Figure 2 Schematic diagram of the content image feature extraction process in an embodiment of the present invention;

[0067] Figure 3 Schematic diagram of the distorted image feature extraction process in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0069] The following combination Figure 1-Figure 3 The present invention describes a method for constructing a no-reference image quality assessment model based on dual semantic guidance of content and distortion.

[0070] Example 1: This example provides a method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance, such as Figure 1 As shown, including:

[0071] S1. Obtain an input image and, based on the input image, input a text template statement describing the image content type: , where c is the content type, which is defined as 9 types:

[0072]

[0073] According to the defined content text description, the Text Transformer encoder of the CLIP model extracts content text features , output content text features The dimension is (9,1024).

[0074] Then, if Figure 2 As shown in the figure, the input image passes through three convolutional layers in sequence, each convolutional layer is connected to a batch normalization layer and an activation function, and then the preprocessed image features are obtained by average pooling. The preprocessed image features are input into a four-layer residual network to extract the local features of the image. ,exist On the basis of [1], a multi-head self-attention mechanism is introduced to strengthen the key semantic areas and increase the understanding of image content, which is expressed as:

[0075]

[0076] Where Q, K, and V represent query, key, and value respectively. After adding the position embedded features, Represents the weight matrix, Concat represents the connection operation of the tensor, is the i-th attention head, expressed as:

[0077]

[0078] in , , , , is the corresponding weight matrix, is the feature dimension;

[0079] Multiple attention heads are connected and mapped to obtain image content features .

[0080] In this embodiment, a distorted image x is input, and the size of the distorted image is adaptively adjusted, and the distorted image is scaled so that the shortest side of the distorted image is 768 pixels;

[0081] The adjusted distorted image is divided into several 224*224 pixel sub-images with a stride of 48. N sub-images are randomly selected. In this embodiment, N is set to 3 and the batch size is 4, resulting in a tensor of size (12, 3, 224*224).

[0082] Input a tensor of size (12, 3, 224*224) into CNN, output a tensor of size (12, 2048, 7, 7), and rearrange it to get local features The dimensions are (12,7*7,2048);

[0083] Adjust local features The size is (12, 7*7+1, 2048). The position encoding is added to the multi-head self-attention mechanism to obtain a tensor of size (12, 7*7+1, 2048). It is projected into the same embedding space as the content text feature to obtain the image content feature. The size is (12,1024);

[0084] For content text features of size (9,1024) and image content features of size (12,1024) Perform contrastive learning to match the input image with its corresponding text and calculate the input image belongs to each content type Probability , the calculation formula is:

[0085]

[0086] in Represents the similarity calculation function, Describe the corresponding text features for 9 different content types. ,when When Representation and image features Matching content text features, is the learnable temperature coefficient.

[0087] S2. Based on the input image, enter a text template statement describing the image distortion type and quality level: ;

[0088] Where d is the distortion type, which is defined as 11 types:

[0089]

[0090] a is the quality grade, which is defined as 5 types:

[0091]

[0092] A total of 11*5=55 distorted text descriptions are combined. According to the defined distorted text descriptions, the TextTransformer encoder of the CLIP model extracts distorted text features. .

[0093] like Figure 3 As shown in Figure 1, the input image is divided into several 16×16 image blocks and mapped to the feature embedding space to obtain the initial embedding sequence;

[0094] After adding category tags and position encoding, the image is input into the Vision Transformer module containing a 12-layer encoder to extract global perceptual features used to characterize the overall quality of the image. ;

[0095] Adjust local features The dimension is obtained ,right and Perform cross attention to fuse local and global information, expressed as:

[0096]

[0097] in , , , is the characteristic dimension of Q, K, and V, is the distortion feature of the input image.

[0098] In this example, the 224*224 pixel sub-image is divided into 7*7 32*32 image blocks with a dimension of 768, forming an initial embedding sequence of size (12, 7*7, 768). After adding the category label and position encoding, the size is (12, 7*7+1,768), which is input into the Transformer to output the global feature. The size is (12,7*7+1,512);

[0099] Local features Resize to get is (12,7*7+1,512), With global features After linear mapping, a cross-attention operation is performed to obtain a tensor of size (12,7*7+1,512), which is projected into the same embedding space as the distorted text feature to obtain the image distortion feature. The size is (12,512);

[0100] For distorted text features of size (55,512) and image distortion features of size (12,512) Perform contrastive learning to obtain the joint probability distribution of distortion type and quality level , expressed as:

[0101]

[0102] in Represents the similarity calculation function, Describe the corresponding text features for 55 different distorted texts. ,when When Representation and image features Matched distorted text features, is the learnable temperature coefficient;

[0103] Marginalized Joint Probability Then we can get the probability of distortion type and quality level. and .

[0104] Likert scale with five quality levels, according to the Likert scale, by the image Quality The predicted probability Calculate the quality score of the image using the following formula:

[0105]

[0106] in For images The prediction quality score of For images Quality The prediction probability of , a is one of five quality levels, indicating the degree of image quality distortion.

[0107] right 、 、 Loss calculations are performed separately, that is, image content loss, image distortion loss, and image quality loss are calculated separately. The calculation formula is:

[0108]

[0109]

[0110]

[0111] in Predict the probability for the image content type, is the predicted probability of image distortion type, is the predicted quality score of the image, is the true quality score of the image, and N is the number of training images;

[0112] The weighted sum of the three losses is used to guide model training, which is expressed as:

[0113]

[0114] in is the image content loss, is the image distortion loss, For image quality loss, is the weight coefficient.

[0115] The weighted summation loss function is used to train the above-constructed network framework to obtain an image quality evaluation model.

[0116] Embodiment 2: This embodiment provides a non-transitory computer-readable storage medium having computer instructions stored thereon. The computer instructions cause a computer to execute a method for constructing a no-reference image quality assessment model based on dual semantic guidance of content and distortion. The method comprises the following steps:

[0117] S1. Obtain an input image, input a text template statement describing the image content type based on the input image, and use the Text Transformer encoder of the CLIP model to extract content text features;

[0118] The local features of the input image are extracted through the CNN network, and the local features are input into the multi-head self-attention mechanism for global semantic understanding to obtain the content features of the input image;

[0119] Calculate the cosine similarity between the content features of the input image and the content text features to obtain the matching probability between the input and various content types;

[0120] S2. Input a text template sentence describing the image distortion type and quality level according to the input image, and extract distorted text features by the Text Transformer encoder of the CLIP model;

[0121] Input the input image to the Vision Transformer encoder to extract global features, and fuse them with local features through cross attention to obtain the distortion features of the input image;

[0122] Calculate the cosine similarity between the distortion features of the input image and the distorted text features to obtain the matching probability between the input image and various distortion types and quality levels;

[0123] The quality score is predicted based on the matching probability of the obtained quality levels combined with the Likert scale;

[0124] S3. Calculate the losses of content type, distortion type, and quality score respectively, and use the weighted sum as the loss function. Train the network framework constructed in steps S1-S2 to obtain an image quality evaluation model.

[0125] Embodiment 3: This embodiment provides an electronic device, which may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may invoke logic instructions in the memory to execute a method for constructing a no-reference image quality assessment model based on dual semantic guidance of content and distortion, the method comprising the following steps:

[0126] S1. Obtain an input image, input a text template statement describing the image content type based on the input image, and use the Text Transformer encoder of the CLIP model to extract content text features;

[0127] The local features of the input image are extracted through the CNN network, and the local features are input into the multi-head self-attention mechanism for global semantic understanding to obtain the content features of the input image;

[0128] Calculate the cosine similarity between the content features of the input image and the content text features to obtain the matching probability between the input and various content types;

[0129] S2. Input a text template sentence describing the image distortion type and quality level according to the input image, and extract distorted text features by the Text Transformer encoder of the CLIP model;

[0130] Input the input image to the Vision Transformer encoder to extract global features, and fuse them with local features through cross attention to obtain the distortion features of the input image;

[0131] Calculate the cosine similarity between the distortion features of the input image and the distorted text features to obtain the matching probability between the input image and various distortion types and quality levels;

[0132] The quality score is predicted based on the matching probability of the obtained quality levels combined with the Likert scale;

[0133] S3. Calculate the losses of content type, distortion type, and quality score respectively, and use the weighted sum as the loss function. Train the network framework constructed in steps S1-S2 to obtain an image quality evaluation model.

[0134] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0135] Embodiment 4: This embodiment provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform a method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance. The method includes the following steps:

[0136] S1. Obtain an input image, input a text template statement describing the image content type based on the input image, and use the Text Transformer encoder of the CLIP model to extract content text features;

[0137] The local features of the input image are extracted through the CNN network, and the local features are input into the multi-head self-attention mechanism for global semantic understanding to obtain the content features of the input image;

[0138] Calculate the cosine similarity between the input image content features and the content text features to obtain the matching probability between the input and various content types;

[0139] S2. Input a text template sentence describing the image distortion type and quality level according to the input image, and extract distorted text features by the Text Transformer encoder of the CLIP model;

[0140] Input the input image to the Vision Transformer encoder to extract global features, and fuse them with local features through cross attention to obtain the distortion features of the input image;

[0141] Calculate the cosine similarity between the distortion features of the input image and the distorted text features to obtain the matching probability between the input image and various distortion types and quality levels;

[0142] The quality score is predicted based on the matching probability of the obtained quality levels combined with the Likert scale;

[0143] S3. Calculate the losses of content type, distortion type, and quality score respectively, and use the weighted sum as the loss function. Train the network framework constructed in steps S1-S2 to obtain an image quality evaluation model.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0145] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for constructing a no-reference image quality assessment model based on dual semantic guidance of content and distortion, characterized in that: include: S1. Obtain an input image, input a text template statement describing the image content type based on the input image, and use the Text Transformer encoder of the CLIP model to extract content text features; The local features of the input image are extracted through the CNN network, and the local features are input into the multi-head self-attention mechanism for global semantic understanding to obtain the content features of the input image; Calculate the cosine similarity between the input image content features and the content text features to obtain the matching probability between the input and various content types; S2. Input a text template sentence describing the image distortion type and quality level according to the input image, and extract distorted text features by the Text Transformer encoder of the CLIP model; Input the input image to the Vision Transformer encoder to extract global features, and fuse them with local features through cross attention to obtain the distortion features of the input image; Calculate the cosine similarity between the distortion features of the input image and the distorted text features to obtain the matching probability between the input image and various distortion types and quality levels; The quality score is predicted based on the matching probability of the obtained quality levels combined with the Likert scale; S3. Calculate the losses of content type, distortion type, and quality score respectively, and use the weighted sum as the loss function. Train the network framework constructed in steps S1-S2 to obtain an image quality evaluation model.

2. The method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance according to claim 1, characterized in that: In step S1, a text template statement describing the image content type is input according to the input image, and the text features of the content are extracted by the Text Transformer encoder of the CLIP model; Specifically: Enter a text template statement describing the image content type: , where c is the content type, which is defined as 9 types: ； According to the defined content text description, the Text Transformer encoder of the CLIP model extracts content text features .

3. The method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance according to claim 2, characterized in that: Step S1 extracts local features of the input image through the CNN network, inputs the local features into the multi-head self-attention mechanism for global semantic understanding, and obtains the content features of the input image; specifically includes: The input image passes through three convolutional layers in sequence. Each convolutional layer is connected to a batch normalization layer and an activation function. The preprocessed image features are then obtained by average pooling. The preprocessed image features are input into a four-layer residual network to extract the local features of the image. ,exist On the basis of [1], a multi-head self-attention mechanism is introduced to strengthen the key semantic areas and increase the understanding of image content, which is expressed as: ； Where Q, K, and V represent query, key, and value respectively. After adding the position embedded features, Represents the weight matrix, Concat represents the connection operation of the tensor, is the i-th attention head, expressed as: ； in , , , , is the corresponding weight matrix, is the feature dimension; Multiple attention heads are connected and mapped to obtain image content features .

4. The method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance according to claim 3, characterized in that: Step S1 performs cosine similarity calculation on the content features of the input image and the content text features to obtain the matching probability between the input and various content types; specifically, the steps include: The content features of the input image and content text features Perform contrastive learning to match the input image with its corresponding text and calculate the input image belongs to each content type The probability is calculated as: ； in Represents the similarity calculation function, Describe the corresponding text features for 9 different content types. ,when When Representation and image features Matching content text features, is the learnable temperature coefficient.

5. The method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance according to claim 1, characterized in that: In step S2, a text template statement describing the image distortion type and quality level is input based on the input image, and the text transformer encoder of the CLIP model extracts the distorted text features; specifically: Enter the text template statement that describes the image distortion type and quality level: ; Where d is the distortion type, which is defined as 11 types: ； a is the quality grade, which is defined as 5 types: ； A total of 11*5=55 distorted text descriptions are combined. According to the defined distorted text descriptions, the TextTransformer encoder of the CLIP model extracts distorted text features. .

6. The method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance according to claim 5, characterized in that: In step S2, the input image is input to the Vision Transformer encoder to extract global features, and then fused with local features through cross attention to obtain the distortion features of the input image; including: The input image is divided into several 16×16 image blocks and mapped to the feature embedding space to obtain the initial embedding sequence; After adding category tags and position encoding, the image is input into the Vision Transformer module containing a 12-layer encoder to extract global perceptual features used to characterize the overall quality of the image. ; Adjust local features The dimension is obtained ,right and Perform cross attention to fuse local and global information, expressed as: ； in , , , is the characteristic dimension of Q, K, and V, is the distortion feature of the input image.

7. The method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance according to claim 6, characterized in that: Step S2 calculates the cosine similarity between the distortion features of the input image and the distorted text features to obtain the matching probability between the input image and various distortion types and quality levels; specifically, it includes: The distortion features of the input image and distorted text features Perform contrastive learning to obtain the joint probability distribution of distortion type and quality level, which is expressed as: ； in Represents the similarity calculation function, Describe the corresponding text features for 55 different distorted texts. ,when When Representation and image features Matched distorted text features, is the learnable temperature coefficient; Marginalized Joint Probability Then we can get the probability of distortion type and quality level. and .

8. The method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance according to claim 7, characterized in that: Step S2 predicts the quality score based on the obtained matching probability of the quality level in combination with the Likert scale; specifically: Likert scale with five quality levels, represented by the image Quality The predicted probability Calculate the quality score of the image using the following formula: ； in For images The prediction quality score of For images Quality The prediction probability of , a is one of five quality levels, indicating the degree of image quality distortion.

9. The method for constructing a no-reference image quality assessment model based on content and distortion dual semantic guidance according to claim 1, characterized in that: In step S3, the losses of content type, distortion type and quality score are calculated respectively, and the weighted sum is used as the loss function, specifically: Calculate the image content loss, image distortion loss, and image quality loss respectively. The calculation formula is: ； ； ； in Predict the probability for the image content type, is the predicted probability of image distortion type, is the predicted quality score of the image, is the true quality score of the image, and N is the number of training images; The weighted sum of the three losses is used to guide model training, which is expressed as: ； in is the image content loss, is the image distortion loss, For image quality loss, is the weight coefficient.

10. A non-transitory computer-readable storage medium, characterized in that Computer instructions are stored thereon, and the computer instructions enable the computer to execute the method for constructing a reference-free image quality assessment model based on content and distortion dual semantic guidance as described in any one of claims 1 to 9.

Images