Image quality evaluation method considering text context

Abstract

The invention provides an image quality evaluation method considering a text context. The method comprises the steps of obtaining a to-be-evaluated image and a text description of the to-be-evaluated image; performing feature extraction on the to-be-evaluated image and the text description through a multi-modal feature extraction module to correspondingly obtain image features and text features; performing interactive fusion on the image features and the text features through a local enhancement module to obtain a semantic mask, and performing region perception enhancement on the image features based on the semantic mask to obtain enhanced image features; and splicing and evaluating the enhanced image features and the image features through a quality regression module to obtain a first perceptual quality score. According to the method, the quality information of the text semantic related region can be explicitly enhanced, so that a real subjective scoring behavior is better fitted, and the accuracy and reliability of image quality evaluation in a multi-modal scene are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image quality assessment method considering text context. Background Art

[0002] Image quality assessment is a technique that automatically quantifies the visual quality of an image by building a mathematical model. Image quality assessment is widely used in various fields, such as image coding and image super-resolution. An image quality assessment algorithm that considers textual context comprehensively evaluates the perceived quality of an image in multimodal scenarios where both images and text coexist, combining the image's visual features with the semantic information of the text. This image quality assessment algorithm not only focuses on the quality attributes of the image itself but also considers the impact of the textual context on users' visual attention and quality judgments, thereby more comprehensively reflecting users' true perception of image quality in the context of both image and text.

[0003] Currently, existing image quality assessment methods in multimodal scenarios where images and text coexist suffer from the following technical issues: First, the introduction of text distracts users' visual attention from the main image, affecting their judgment of image quality; second, image distortion leads to semantic inconsistencies between the image and text, interfering with users' perception of image quality. These issues hinder the accurate assessment of image perceptual quality in multimodal scenarios. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an image quality assessment method that takes text context into consideration to improve the assessment accuracy of image perceptual quality.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0006] In a first aspect, the present invention provides an image quality assessment method considering text context, the method comprising:

[0007] Obtaining an image to be evaluated and a text description of the image to be evaluated;

[0008] Performing feature extraction on the image to be evaluated and the text description respectively through a multimodal feature extraction module to obtain image features and text features respectively;

[0009] Interactively fusing the image features and the text features through a local enhancement module to obtain a semantic mask, and performing region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features;

[0010] The enhanced image features and the image features are spliced ​​and evaluated by a quality regression module to obtain a first perceptual quality score.

[0011] In some embodiments of the present invention, the local enhancement module includes a cross-modal fusion submodule based on an attention mechanism and a mask generation submodule;

[0012] The interactive fusion of the image features and the text features by the local enhancement module to obtain a semantic mask includes:

[0013] generating an attention map by using the cross-modal fusion submodule, taking the image features as queries and the text features as keys and values, and obtaining fused features based on the attention map and the text features;

[0014] The mask generation submodule calculates the fusion feature and the image feature to obtain a semantic mask, and the semantic mask is used to locate the area related to the text description in the image to be evaluated.

[0015] In some embodiments of the present invention, after obtaining the image to be evaluated and the text description of the image to be evaluated, the method further includes:

[0016] Calculating a semantic consistency score between the image to be evaluated and the text description to obtain a second perceptual quality score;

[0017] After obtaining the first perceptual quality score, the method further includes:

[0018] A final perceptual quality score is obtained based on the first perceptual quality score and the second perceptual quality score.

[0019] In some embodiments of the present invention, calculating the semantic consistency score between the image to be evaluated and the text description to obtain a second perceptual quality score includes:

[0020] Performing normalized feature extraction on the image to be evaluated and the text description respectively, and obtaining normalized visual features and normalized language features accordingly;

[0021] A similarity measurement calculation is performed on the normalized visual feature and the normalized language feature, and the calculation result is used as the second perceptual quality score.

[0022] In some embodiments of the present invention, the quality regression module includes a concatenation layer, a convolution layer, a pooling layer, and a regression layer;

[0023] The step of combining and evaluating the enhanced image features and the image features by a quality regression module to obtain a first perceptual quality score includes:

[0024] splicing the enhanced image features and the image features through the splicing layer to obtain spliced ​​features;

[0025] Compressing the concatenated features through the convolution layer to obtain compressed features;

[0026] Performing dimensionality reduction on the compressed features through the pooling layer to obtain compact features;

[0027] The compact feature is regressed by the regression layer to obtain the first perceptual quality score.

[0028] In some embodiments of the present invention, performing normalized feature extraction on the image to be evaluated and the text description respectively to obtain normalized visual features and normalized language features, respectively, includes:

[0029] Performing normalized feature extraction on the image to be evaluated using a visual encoder in a pre-trained CLIP model to obtain the normalized visual features;

[0030] The text encoder in the CLIP model is used to perform normalized feature extraction on the text description to obtain the normalized language features.

[0031] In some embodiments of the present invention, the multimodal feature extraction module includes an image encoder and a text encoder;

[0032] The multimodal feature extraction module is used to extract features from the image to be evaluated and the text description, respectively, to obtain image features and text features, including:

[0033] Extracting features from the image to be evaluated by the image encoder to obtain the image features;

[0034] The text encoder performs feature extraction on the text description to obtain the text features.

[0035] In a second aspect, the present invention further provides an image quality assessment device that considers text context, the device comprising:

[0036] A data acquisition unit, configured to acquire an image to be evaluated and a text description of the image to be evaluated;

[0037] A feature extraction unit, configured to extract features from the image to be evaluated and the text description respectively through a multimodal feature extraction module, and obtain image features and text features respectively;

[0038] an enhancement unit, configured to interactively fuse the image features and the text features through a local enhancement module to obtain a semantic mask, and perform region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features;

[0039] An evaluation unit is configured to combine and evaluate the enhanced image features and the image features through a quality regression module to obtain a first perceptual quality score.

[0040] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the program.

[0041] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0042] Beneficial effects of the present invention: The present invention provides an image quality assessment method that considers textual context. A multimodal feature extraction module extracts features from the image to be assessed and its corresponding textual description, obtaining corresponding multi-scale image features and textual features. A local enhancement module receives these multi-scale image and textual features and constructs a semantic mask to locate the textual description region in the image, thereby obtaining enhanced image features focused on the text-related region. A quality regression module calculates a quality perception score based on the enhanced image features and the image features. Considering that users' attention will significantly shift to the image region indicated by the textual description, users will be more sensitive to the quality of that region. Existing image quality assessment techniques primarily rely on the visual features of the image itself to make quality judgments, ignoring the issue of attention shift caused by textual context. Therefore, the present invention provides the above-mentioned method, which models attention shift guided by text as a key factor in image quality assessment. Through an attention-guided local enhancement mechanism, it explicitly enhances quality information in regions related to textual semantics, thereby more closely aligning with real-world subjective scoring behavior and improving the accuracy and reliability of image quality assessment in multimodal scenarios.

[0043] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 One of the flowcharts of an image quality assessment method considering text context provided by an embodiment of the present invention;

[0046] Figure 2The second flowchart of an image quality assessment method considering text context is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0048] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of features, integers, steps, operations, elements and / or modules, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0049] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.

[0050] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0051] Example 1

[0052] See also Figure 1 as well as Figure 2 An embodiment of the present invention provides an image quality assessment method considering text context, the method comprising the following steps:

[0053] S101: Acquire an image to be evaluated and a text description of the image to be evaluated.

[0054] The image to be evaluated can be a high-quality or distorted image, an image uploaded by a user, an intermediate / final result of an image processing process, a frame captured by a surveillance camera, a compressed or transmitted image, a medical image, a satellite remote sensing image, etc. The text description is an objective description of the content of the image to be evaluated. Schematically, Figure 2 The image to be evaluated is a surfing image, and the corresponding text description is "A person attempts to raise the sail on his surfboard." The text description can be generated through annotation, automatically generated by an image description model, or provided by the user when uploading the image. There are no restrictions on this.

[0055] S102: Perform feature extraction on the image to be evaluated and the text description respectively through a multimodal feature extraction module to obtain image features and text features accordingly.

[0056] In this step, the multimodal feature extraction module may include a submodule for image feature extraction and a submodule for text extraction. The submodule for image feature extraction performs feature extraction on the image to be evaluated to obtain image features; the submodule for text extraction performs feature extraction on the text description to obtain text features.

[0057] The submodule used for image feature extraction is a deep convolutional neural network, such as ResNet, VGG, EfficientNet, etc., which can extract rich multi-scale image features.

[0058] The sub-module for text extraction can be a Transformer-based model, which understands the semantics and contextual relationships of the text description through the sub-module for text extraction.

[0059] It is understandable that before the submodule for image feature extraction performs feature extraction on the image to be evaluated, an image preprocessing process is also included, such as scaling, normalization, etc., which is not limited.

[0060] S103: interactively fuse the image features and the text features through a local enhancement module to obtain a semantic mask, and perform region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features.

[0061] In this step, the attention between text and image is calculated using a cross-attention mechanism. This attention is used to quantify the semantic relevance between text features and image features, and a semantic mask is used to characterize the degree of semantic relevance between the image and the text description. Utilizing this semantic mask enables the model provided by the present invention (which includes the aforementioned multimodal feature extraction module, local enhancement module, quality regression module, etc.) to focus on image regions related to the text description. The quality of these image regions has a significant impact on the perception of the overall image.

[0062] S104: Splicing and evaluating the enhanced image features and the image features through a quality regression module to obtain a first perceptual quality score.

[0063] In this step, the enhanced image features focus on the image areas related to the text description. The image features retain complete global and underlying visual information. By splicing in the channel dimension, they provide more comprehensive and discriminative features for regression, making the final prediction score as close as possible to the human subjective score.

[0064] It can be understood that the model provided by the present invention is pre-trained based on training images and training texts corresponding to the training images, and the training process will not be described in detail here.

[0065] An embodiment of the present invention provides an image quality assessment method that considers textual context. A multimodal feature extraction module extracts features from the image to be assessed and its corresponding textual description, obtaining multi-scale image features and textual features. A local enhancement module receives these multi-scale image and textual features and constructs a semantic mask to locate the textual description region in the image, thereby obtaining enhanced image features focused on the text-related region. A quality regression module calculates a quality perception score based on the enhanced image features and the aforementioned image features. Considering that users' attention is significantly biased toward the image region indicated by the textual description, users are more sensitive to the quality of that region. Existing image quality assessment techniques primarily rely on the image's own visual features for quality judgment, ignoring the issue of attention shift caused by textual context. Therefore, the present invention provides the aforementioned method, which models attention shift guided by text as a key factor in image quality assessment. Through an attention-guided local enhancement mechanism, the method explicitly enhances quality information in regions related to textual semantics, thereby more closely aligning with real-world subjective scoring behavior and improving the accuracy and reliability of image quality assessment in multimodal scenarios.

[0066] In some embodiments of the present invention, the local enhancement module includes a cross-modal fusion submodule based on an attention mechanism and a mask generation submodule.

[0067] The interactive fusion of the image features and the text features by the local enhancement module to obtain a semantic mask includes:

[0068] The cross-modal fusion submodule uses the image features as queries and the text features as keys and values ​​to generate an attention map, and obtains fusion features based on the attention map and the text features. (Among them, C i , H i , W i The number of channels, height and width representing image features) as query, text features (Among them, C t , T represents the number of channels and words of text features) as the key (Key) and value (Value), and the attention map A is generated by scaling the dot product attention calculation i , and based on this we get the fusion feature G i The principle can be expressed by the following formula:

[0069]

[0070] G i =A i F t

[0071] Among them, Softmax is a normalization operation used to convert the input vector into a probability distribution; C t is the number of channels of text features.

[0072] The mask generation submodule calculates the fusion feature and the image feature to obtain a semantic mask, which is used to locate the area related to the text description in the image to be evaluated. i After dimension reshaping and original image feature F i The spatial alignment is then integrated into the image feature stream by element-by-element addition to obtain the enhanced feature E guided by text semantics. i , which is calculated as follows:

[0073] E i =F i +G i

[0074] Enhanced Feature E i Through convolutional layer dimensionality reduction, and adopting a top-down fusion strategy, that is, high-level features are sequentially upsampled and added to low-level features to finally form semantic fusion features, which are then Figure 2 The decoder process in generates a semantic mask M, which is used to locate the areas in the image that are highly semantically related to the input text.

[0075] Accordingly, performing region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features includes:

[0076] The semantic mask is multiplied element-by-element by the image features to obtain the enhanced image features. This method explicitly enhances the network's ability to perceive the quality information of the region. The calculation formula is as follows:

[0077]

[0078] in, Represents an element-by-element multiplication operation. The local enhancement module effectively models the perceptual behavior of users’ attention biased towards specific regions in the context of text, making the network output more consistent with actual human perception and effectively improving the robustness of quality assessment in the context of text.

[0079] An embodiment of the present invention provides a text-context-based image quality assessment method that interactively integrates image and text features to generate a semantic mask to locate image regions semantically related to the text description. The semantic mask is used to weight image features, explicitly enhancing the network's ability to perceive the quality of regions focused on by the text. This mechanism effectively models the perceptual behavior of users' attention biased towards the text description region in a textual context, thereby improving the accuracy and robustness of image quality assessment in this context.

[0080] In some embodiments of the present invention, after obtaining the image to be evaluated and the text description of the image to be evaluated, the method further includes:

[0081] A semantic consistency score between the image to be evaluated and the text description is calculated to obtain a second perceptual quality score.

[0082] After obtaining the first perceptual quality score, the method further includes:

[0083] A final perceptual quality score is obtained based on the first perceptual quality score and the second perceptual quality score.

[0084] Schematically, the first perceptual quality score and the second perceptual quality score are integrated by weighted fusion to generate a final image quality score in the text context. The specific calculation formula is as follows:

[0085] S=αS1+βS2∈[0,1]

[0086] Among them, α and β are learnable parameters, which are automatically optimized through back propagation during the training process and used to dynamically adjust the first perceptual quality score S1 (i.e. Figure 2 The score 1 in the first sentence) and the second perceptual quality score S2 (i.e. Figure 2 The relative contribution of the score in 2) in the final quality prediction, and the final output prediction quality score is normalized to the [0,1] interval.

[0087] An embodiment of the present invention provides an image quality assessment method that takes text context into consideration. In order to measure the semantic consistency between an image and its corresponding text description and address the problem of image-text semantic deviation caused by image distortion, the present invention calculates the semantic consistency score between the image to be evaluated and the text description to obtain a second perceptual quality score, and obtains a final perceptual quality score based on the first perceptual quality score and the second perceptual quality score. This overcomes the problem that, in a text context, distortion may cause the semantic information of the image to be inconsistent with the text description, thereby causing a deviation in the user's understanding of the image content, and further affecting their perception of image quality. The second perceptual quality score provides auxiliary information for the final quality prediction, which can more accurately reflect human perception of image quality in a text context.

[0088] In some embodiments of the present invention, calculating the semantic consistency score between the image to be evaluated and the text description to obtain a second perceptual quality score includes:

[0089] Normalized feature extraction is performed on the image to be evaluated and the text description respectively, and normalized visual features and normalized language features are obtained accordingly.

[0090] A similarity measurement calculation is performed on the normalized visual feature and the normalized language feature, and the calculation result is used as the second perceptual quality score.

[0091] Schematically, the similarity measure is calculated by the cosine similarity method:

[0092]

[0093] Among them, V i and V t are normalized visual features and normalized language features, respectively, ||V i || and ||V t || denotes the L2 norm of visual and linguistic features, respectively. This score reflects the degree of semantic alignment between an image and its textual description, providing complementary information from a semantic consistency perspective for subsequent image quality prediction. Through this mechanism, the present invention can more comprehensively integrate the semantic relationship between image and text, improving the accuracy and robustness of image quality assessment in the context of text.

[0094] It is understandable that, in addition to the cosine similarity method, similarity measurement can also be implemented using Euclidean distance, Mahalanobis distance, etc.

[0095] In some embodiments of the present invention, the quality regression module includes a concatenation layer, a convolution layer, a pooling layer, and a regression layer.

[0096] The step of combining and evaluating the enhanced image features and the image features by a quality regression module to obtain a first perceptual quality score includes:

[0097] The enhanced image features and the image features are spliced ​​together through the splicing layer to obtain spliced ​​features.

[0098] The concatenated features are compressed through the convolution layer to obtain compressed features.

[0099] The compressed features are reduced in dimension by the pooling layer to obtain compact features.

[0100] The compact feature is regressed by the regression layer to obtain the first perceptual quality score.

[0101] An embodiment of the present invention provides an image quality assessment method that takes textual context into consideration. Enhanced image features are concatenated with original image features along the channel dimension through a quality regression module. The concatenated features are then compressed and normalized through a convolution operation to enhance their expressiveness and discriminability. A global pooling operation is then used to reduce the dimensionality of the features, preserving key semantic information while generating a compact feature representation. This representation is then fed into a multi-layer perceptron (MLP) regression network to predict the image's first perceptual quality score S1. The quality regression module achieves a synergistic fusion of local and global information, thereby generating a more stable quality prediction result that aligns with human perceptual preferences.

[0102] In some embodiments of the present invention, performing normalized feature extraction on the image to be evaluated and the text description respectively to obtain normalized visual features and normalized language features, respectively, includes:

[0103] Normalized feature extraction is performed on the image to be evaluated using a visual encoder in a pre-trained CLIP model (Contrastive Language-Image Pre-training) to obtain the normalized visual features.

[0104] The text encoder in the CLIP model is used to perform normalized feature extraction on the text description to obtain the normalized language features.

[0105] An embodiment of the present invention provides an image quality assessment method that takes text context into consideration. Normalized feature extraction is performed through the visual encoder and text encoder in the CLIP model to achieve plug-and-play alignment of multimodal features, solving the problems of cross-modal bias and computational redundancy in traditional solutions, and providing a highly robust feature foundation for semantically guided quality assessment.

[0106] In some embodiments of the present invention, the multimodal feature extraction module includes an image encoder and a text encoder;

[0107] The multimodal feature extraction module is used to extract features from the image to be evaluated and the text description, respectively, to obtain image features and text features, including:

[0108] By the image encoder (ie Figure 2 Encoder in ) performing feature extraction on the image to be evaluated to obtain the image features. In this embodiment, the image encoder is ResNet50. In other embodiments of the present invention, other convolutional neural networks may also be used, without limitation.

[0109] By the text encoder (i.e. Figure 2 Encoder in ) performs feature extraction on the text description to obtain the text features. In this embodiment, the text encoder is the text encoder in the aforementioned CLIP model, and may also be other text feature extraction models, which is not limited to this.

[0110] Example 2

[0111] Based on Example 1, this Example 2 provides an image quality assessment device that considers text context. The image quality assessment device that considers text context corresponds to the above-mentioned image quality assessment method that considers text context, and specifically includes:

[0112] A data acquisition unit, configured to acquire an image to be evaluated and a text description of the image to be evaluated;

[0113] A feature extraction unit, configured to extract features from the image to be evaluated and the text description respectively through a multimodal feature extraction module, and obtain image features and text features respectively;

[0114] an enhancement unit, configured to interactively fuse the image features and the text features through a local enhancement module to obtain a semantic mask, and perform region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features;

[0115] An evaluation unit is configured to combine and evaluate the enhanced image features and the image features through a quality regression module to obtain a first perceptual quality score.

[0116] For details, please refer to the description of the image quality assessment method considering text context, which will not be repeated here.

[0117] Example 3

[0118] Embodiment 3 of the present invention provides an electronic device including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute an image quality assessment method considering text context, the method including the following process steps:

[0119] Obtaining an image to be evaluated and a text description of the image to be evaluated;

[0120] Performing feature extraction on the image to be evaluated and the text description respectively through a multimodal feature extraction module to obtain image features and text features respectively;

[0121] Interactively fusing the image features and the text features through a local enhancement module to obtain a semantic mask, and performing region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features;

[0122] The enhanced image features and the image features are spliced ​​and evaluated by a quality regression module to obtain a first perceptual quality score.

[0123] Example 4

[0124] Embodiment 4 of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, a method for image quality assessment considering text context is implemented. The method includes the following steps:

[0125] Obtaining an image to be evaluated and a text description of the image to be evaluated;

[0126] Performing feature extraction on the image to be evaluated and the text description respectively through a multimodal feature extraction module to obtain image features and text features respectively;

[0127] Interactively fusing the image features and the text features through a local enhancement module to obtain a semantic mask, and performing region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features;

[0128] The enhanced image features and the image features are spliced ​​and evaluated by a quality regression module to obtain a first perceptual quality score.

[0129] Example 5

[0130] Embodiment 5 of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements an image quality assessment method that considers text context. The method includes the following steps:

[0131] Obtaining an image to be evaluated and a text description of the image to be evaluated;

[0132] Performing feature extraction on the image to be evaluated and the text description respectively through a multimodal feature extraction module to obtain image features and text features respectively;

[0133] Interactively fusing the image features and the text features through a local enhancement module to obtain a semantic mask, and performing region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features;

[0134] The enhanced image features and the image features are spliced ​​and evaluated by a quality regression module to obtain a first perceptual quality score.

[0135] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0136] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for method or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The method and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Ordinary technicians in this field can understand and implement it without expending creative work.

[0137] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for image quality assessment considering text context, characterized in that: The method comprises: Obtaining an image to be evaluated and a text description of the image to be evaluated; Performing feature extraction on the image to be evaluated and the text description respectively through a multimodal feature extraction module to obtain image features and text features respectively; Interactively fusing the image features and the text features through a local enhancement module to obtain a semantic mask, and performing region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features; The enhanced image features and the image features are spliced ​​and evaluated by a quality regression module to obtain a first perceptual quality score.

2. The image quality assessment method considering text context according to claim 1, characterized in that The local enhancement module includes a cross-modal fusion submodule based on the attention mechanism and a mask generation submodule; The interactive fusion of the image features and the text features by the local enhancement module to obtain a semantic mask includes: generating an attention map by using the cross-modal fusion submodule, taking the image features as queries and the text features as keys and values, and obtaining fused features based on the attention map and the text features; The mask generation submodule calculates the fusion feature and the image feature to obtain a semantic mask, and the semantic mask is used to locate the area related to the text description in the image to be evaluated.

3. The image quality assessment method considering text context according to claim 1, characterized in that After obtaining the image to be evaluated and the text description of the image to be evaluated, the method further includes: Calculating a semantic consistency score between the image to be evaluated and the text description to obtain a second perceptual quality score; After obtaining the first perceptual quality score, the method further includes: A final perceptual quality score is obtained based on the first perceptual quality score and the second perceptual quality score.

4. The image quality assessment method considering text context according to claim 3, characterized in that The calculating the semantic consistency score between the image to be evaluated and the text description to obtain a second perceptual quality score includes: Performing normalized feature extraction on the image to be evaluated and the text description respectively, and obtaining normalized visual features and normalized language features accordingly; A similarity measurement calculation is performed on the normalized visual feature and the normalized language feature, and the calculation result is used as the second perceptual quality score.

5. The image quality assessment method considering text context according to claim 1, characterized in that The quality regression module includes a splicing layer, a convolution layer, a pooling layer and a regression layer; The step of combining and evaluating the enhanced image features and the image features by a quality regression module to obtain a first perceptual quality score includes: splicing the enhanced image features and the image features through the splicing layer to obtain spliced ​​features; Compressing the concatenated features through the convolution layer to obtain compressed features; Performing dimensionality reduction on the compressed features through the pooling layer to obtain compact features; The compact feature is regressed by the regression layer to obtain the first perceptual quality score.

6. The image quality assessment method considering text context according to claim 4, characterized in that The performing normalized feature extraction on the image to be evaluated and the text description respectively to obtain normalized visual features and normalized language features accordingly includes: Performing normalized feature extraction on the image to be evaluated using a visual encoder in a pre-trained CLIP model to obtain the normalized visual features; The text encoder in the CLIP model is used to perform normalized feature extraction on the text description to obtain the normalized language features.

7. The image quality assessment method considering text context according to claim 1, characterized in that The multimodal feature extraction module includes an image encoder and a text encoder; The multimodal feature extraction module is used to extract features from the image to be evaluated and the text description, respectively, to obtain image features and text features, including: Extracting features from the image to be evaluated by the image encoder to obtain the image features; The text encoder performs feature extraction on the text description to obtain the text features.

8. An image quality assessment device considering text context, characterized in that: The device comprises: A data acquisition unit, configured to acquire an image to be evaluated and a text description of the image to be evaluated; A feature extraction unit, configured to extract features from the image to be evaluated and the text description respectively through a multimodal feature extraction module, and obtain image features and text features respectively; an enhancement unit, configured to interactively fuse the image features and the text features through a local enhancement module to obtain a semantic mask, and perform region-aware enhancement on the image features based on the semantic mask to obtain enhanced image features; An evaluation unit is configured to combine and evaluate the enhanced image features and the image features through a quality regression module to obtain a first perceptual quality score.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the image quality assessment method considering text context are implemented as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps of the image quality assessment method considering text context according to any one of claims 1 to 7.

Images