A unified visual language model pre-training and fine-tuning method for image quality and aesthetics assessment

Abstract

A method for pre-training and adjusting a unified visual language model for image quality and aesthetics assessment includes the following steps: a. Using a multimodal large language model (MLLM) to generate text descriptions related to image quality and aesthetics for training images used in image quality and aesthetics assessment tasks, obtaining an image-text description dataset; b. Using a data purification strategy to filter and optimize the image-text description dataset using aesthetic relevance and information richness ranking (AIR); c. Pre-training a unified visual language model (UniQA) on the obtained image-text dataset, using a contrastive learning strategy to maximize the similarity of paired image and text features while minimizing the similarity of mismatched images and text; and d. Using a lightweight adapter to adjust the visual features of the pre-trained UniQA model to suit specific image assessment tasks. This invention provides an efficient, accurate, and adaptable solution for image assessment, with broad application prospects and practical value.

Description

Technical Field

[0001] The present invention relates to a unified visual language model pre-training and adjustment method for image quality and aesthetics evaluation. Background Art

[0002] In the fields of Image Quality Assessment (IQA) and Image Aesthetic Assessment (IAA), the research goal is to develop algorithms and models that can automatically evaluate image quality or aesthetic value.

[0003] Image Quality Assessment (IQA): IQA aims to evaluate the quality of an image, typically focusing on factors such as image clarity, contrast, color balance, and the presence of noise. This technology is crucial for areas such as image processing, compression, transmission, and storage, as it helps ensure that images maintain acceptable quality levels during these processes.

[0004] Image Aesthetic Assessment (IAA): IAA focuses on the aesthetic appeal of an image, that is, how visually pleasing it is. This involves aesthetic factors such as image content, composition, color, and lighting. IAA has a wide range of applications in social media, online advertising, photography competitions, and art galleries.

[0005] In recent years, with the rapid development of deep learning technology, the field of image quality assessment (IQA) has attracted widespread interest. The convolutional neural network (CNN)-based model initially achieved effectiveness in the IQA problem, although it only contained three convolutional layers. Subsequently, CNN-based IQA methods focused on different aspects, including deeper feature learning networks, multi-level feature aggregation, adaptive quality prediction, and learning from image patches to the whole. Recently, Transformer-based IQA methods have shown promising results in the field of IQA, which can compensate for the shortcomings of CNN in the ability to learn non-local features. Despite these significant breakthroughs, these methods usually transfer models pre-trained on classification datasets (such as ImageNet) to IQA tasks, which may not be optimal.

[0006] Image aesthetic assessment (IAA) aims to measure the aesthetic quality of images. With the advent of deep learning, IAA methods have evolved from hand-crafted feature extraction to end-to-end feature learning, marking a significant advancement in the field of IAA. Various techniques have been developed to promote progress in the field of IAA, including local and global feature integration, graph neural networks, knowledge distillation, and topic-aware learning. Recently, multimodal IAA methods have emerged that incorporate text as auxiliary supervision. However, these methods require the use of text during reasoning, which limits their flexible application as text is often not readily available.

[0007] Existing technologies in the fields of image quality assessment (IQA) and image aesthetic assessment (IAA) have the following major shortcomings: 1) Isolated task processing: Traditional methods usually treat IQA and IAA as independent tasks, ignoring the mutual correlation and influence between the two, resulting in the inability to fully utilize the commonalities between the tasks to improve evaluation performance. 2) Dataset limitations: IQA datasets lack text descriptions, while text descriptions in IAA datasets may contain noise, which limits the model's ability to learn effective features from the data. 3) Limitations of perceptual representation: Existing methods may not be able to fully capture humans' subjective perception of image quality and aesthetics, resulting in a gap between evaluation results and human perception. 4) Insufficient generalization ability: Many existing methods perform well on specific datasets, but lack generalization ability in other datasets or diverse real-world scenarios. 5) Insufficient zero-shot and few-label learning capabilities: In the case of scarce labeled data, existing technologies have difficulty in effectively performing image evaluation, limiting their widespread use in practical applications.

[0008] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0009] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide a unified visual language model pre-training and adjustment method for image quality and aesthetic assessment, which effectively solves the problem that existing methods fail to effectively integrate the intrinsic connection between the two tasks of image quality assessment (IQA) and image aesthetic assessment (IAA), resulting in the inability to learn shared representations consistent with human perception.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] A unified visual language model pre-training and tuning method for image quality and aesthetic evaluation, comprising the following steps:

[0012] a. Use a multimodal large language model (MLLM) to generate text descriptions related to image quality and aesthetics for training images used in image quality and aesthetics assessment tasks, and obtain an image-text description dataset;

[0013] b. Using data purification strategies to filter and optimize the image-text description dataset by ranking AIR by aesthetic relevance and information richness;

[0014] c. Pre-train the unified visual language model UniQA on the image-text dataset obtained by screening and optimizing in step b. Using a contrastive learning strategy, the model maximizes the similarity between paired image and text features while minimizing the similarity between mismatched images and text.

[0015] d. Adapting the visual features of the pre-trained UniQA model to specific image evaluation tasks through lightweight adapters.

[0016] Further:

[0017] In step a, the multimodal large language model MLLM and specific prompts designed for image quality and aesthetics assessment tasks are used, combined with text guidance based on image quality score MOS, to generate text descriptions related to image quality and aesthetics, forming the corresponding description dataset Y IQA and Y IAA ; The specific prompts are tailored for image quality assessment (IQA) and image aesthetic assessment (IAA) tasks respectively to guide MLLM to generate task-related text descriptions.

[0018] The text descriptions include descriptions involving sharpness, color balance, and noise level designed for the IQA task, and descriptions including content, color, lighting, and composition designed for the IAA task.

[0019] In step b, the aesthetic relevance ranking AR and the information richness ranking IR are calculated to generate the aesthetic relevance and information richness ranking AIR, and AIR is used to screen out texts that are closely related to image aesthetics to construct an optimized dataset.

[0020] Step c specifically includes:

[0021] Use the CLIP model to train on a large-scale image-text dataset;

[0022] Maximize the cosine similarity of paired image and text features through a contrastive learning strategy;

[0023] The model is trained using a temperature parameter τ and contrastive learning loss to generate aligned feature representations; the temperature parameter τ is used to adjust the sensitivity of the similarity calculation between image and text features.

[0024] In step c, the IAA dataset containing human comments is used together with the optimized image-text description dataset to train the UniQA model.

[0025] In step c, the contrastive learning loss is calculated by taking the average of the image loss and the text loss.

[0026] In step d, the lightweight adapter includes an image encoder and a text encoder that constitute the backbone network of the UniQA model, and is used to extract visual features of the image and semantic features of the text;

[0027] A learnable residual module is added after the pre-trained image encoder to adjust the visual features to adapt to a specific evaluation dataset; wherein the residual module includes two fully connected layers with ReLU activation functions, and adjusts the visual features of the image through normalization and residual connections, while keeping the image and text backbone networks frozen and only optimizing the residual module;

[0028] Among them, the prompt template "{level}image" and the predefined text level are used to construct multiple prompt groups, and the cosine similarity between the normalized text features and the adjusted visual features of each prompt group is calculated. The Softmax function is used to obtain the correlation value of the image-text correspondence, and the correlation value is weighted by the predefined incremental value scoring level to obtain the final evaluation score.

[0029] The method further comprises the following steps:

[0030] e. For the UniQA model fine-tuned with the lightweight adapter in step d, adopt a prompt ensemble strategy to comprehensively evaluate the quality and aesthetic features of the image by combining multiple prompt groups. The evaluation scores of multiple prompt groups are preferably averaged to obtain the final evaluation score.

[0031] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the unified visual language model pre-training and adjustment method for image quality and aesthetic evaluation.

[0032] The present invention has the following beneficial effects:

[0033] The method of the present invention uses a large multimodal model and language as a bridge to perform unified aesthetic and quality-related pre-training, learns beneficial representations of quality and aesthetic consistency, and then uses task-specific knowledge extraction to enable it to be used in downstream image quality evaluation and perception tasks. The method of the present invention can learn more effective and generalized representations by performing joint pre-training on quality and aesthetic image-text datasets, providing more general and effective representations for IQA tasks. The method of the present invention overcomes the limitation of previous multimodal IAA methods that combine text as auxiliary supervision in the reasoning process through visual-language pre-training. The pre-trained model can be flexibly applied to the field of IAA using only images.

[0034] Compared with the existing methods, the main advantages of the present invention are:

[0035] 1) Unified visual-language pre-training: By establishing a unified model, the quality and aesthetics of images are learned and evaluated simultaneously to overcome the problem of isolated task processing. 2) Data enhancement and purification: Multimodal large language models (MLLMs) are used to generate high-quality text descriptions, and data purification strategies are adopted to improve the quality of the dataset. 3) Learning human perception representations: Through large-scale pre-training, the model learns representations consistent with human visual perception, improving the accuracy and consistency of evaluation. 4) Enhanced generalization ability: By pre-training and adaptation on a variety of datasets, the model's generalization ability in different datasets and scenarios is improved. 5) Improve zero-shot and few-label learning capabilities: Design lightweight adapters and strategies so that the model can still effectively perform image evaluation when there is limited labeled data, expanding its potential in practical applications.

[0036] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flowchart of a unified visual language model pre-training and adjustment method for image quality and aesthetic evaluation according to an embodiment of the present invention.

[0038] Figure 2 This is an overall algorithm block diagram of the unified visual language model pre-training and adjustment method for image quality and aesthetic evaluation according to an embodiment of the present invention, wherein (a) shows the data purification process and (b) shows the multi-cue integration adapter. DETAILED DESCRIPTION

[0039] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0040] See Figure 1 , an embodiment of the present invention provides a unified visual language model pre-training and adjustment method for image quality and aesthetics evaluation, comprising the following steps:

[0041] a. Use a multimodal large language model (MLLM) to generate text descriptions related to image quality and aesthetics for training images used in image quality and aesthetics assessment tasks, and obtain an image-text description dataset;

[0042] b. Using a data purification strategy to screen and optimize the image-text description dataset by ranking by aesthetic relevance and information richness to improve the quality of training data; wherein the data purification strategy includes removing noisy text that is irrelevant to the image aesthetics and enhancing descriptions that are closely related to the image content;

[0043] c. Pre-train the unified visual language model UniQA on the image-text dataset obtained by screening and optimizing in step b. Using a contrastive learning strategy, the model maximizes the similarity between paired image and text features while minimizing the similarity between mismatched images and text.

[0044] d. Adapting the visual features of the pre-trained UniQA model to specific image evaluation tasks through lightweight adapters.

[0045] In a preferred embodiment, in step a, the multimodal large language model MLLM and specific prompts designed for image quality and aesthetics assessment tasks are used, combined with text guidance based on image quality score MOS, to generate text descriptions related to image quality and aesthetics, forming a corresponding description dataset Y IQA and Y IAA The specific prompts are tailored for the image quality assessment (IQA) and image aesthetics assessment (IAA) tasks, respectively, to guide the MLLM to generate task-related text descriptions. Preferably, the text descriptions include descriptions related to sharpness, color balance, and noise level designed for the IQA task, and descriptions related to content, color, lighting, and composition designed for the IAA task.

[0046] In a preferred embodiment, in step b, the aesthetic relevance ranking AR and the information richness ranking IR are calculated to generate the aesthetic relevance and information richness ranking AIR, and AIR is used to screen out high-quality texts that are closely related to image aesthetics to construct an optimized dataset.

[0047] In a preferred embodiment, step c specifically includes: training the CLIP model on a large-scale image-text dataset; maximizing the cosine similarity of paired image and text features using a contrastive learning strategy; training the model using a temperature parameter τ and a contrastive learning loss to generate aligned feature representations; wherein the temperature parameter τ is used to adjust the sensitivity of the similarity calculation between image and text features. Preferably, the contrastive learning loss is calculated by taking the average of the image loss and the text loss.

[0048] In a preferred embodiment, in step c, the IAA dataset containing human comments is also used together with the optimized image-text description dataset to train the UniQA model.

[0049] In a preferred embodiment, in step d, the lightweight adapter includes an image encoder and a text encoder constituting the backbone network of the UniQA model, which are used to extract visual features of images and semantic features of text.

[0050] In a preferred embodiment, the lightweight adapter includes two aspects: visual feature adaptation and multi-cue integrated prediction.

[0051] Visual feature adaptation: A learnable residual module is added after the pre-trained image encoder to adjust the visual features to adapt to the specific evaluation dataset; wherein, the residual module includes two fully connected layers with ReLU activation function, which adjusts the visual features of the image through normalization and residual connections, while keeping the image and text backbone networks frozen and only optimizing the residual module to achieve efficient parameter adjustment.

[0052] Multi-cue integrated prediction: Using the prompt template "{level}image" and predefined text levels, multiple prompt groups are constructed. The cosine similarity between the normalized text features and the adjusted visual features of each prompt group is calculated. The softmax function is used to obtain the correlation value of the image-text correspondence. The correlation value is weighted by a predefined incremental value rating scale to obtain the final evaluation score. The absolute similarity scores of the prompt groups are then converted into relative values ​​for weighting. This allows for comprehensive exploration of the capabilities of pre-trained UniQA through diverse cues to improve evaluation accuracy.

[0053] In a preferred embodiment, the method further comprises the following steps:

[0054] e. For the UniQA model fine-tuned with the lightweight adapter in step d, adopt a prompt ensemble strategy to comprehensively evaluate the quality and aesthetic features of the image by combining multiple prompt groups. The evaluation scores of multiple prompt groups are preferably averaged to obtain the final evaluation score.

[0055] Embodiments of the present invention address a key technical issue in image quality assessment (IQA) and image aesthetic assessment (IAA): existing methods fail to effectively integrate the intrinsic connection between the two tasks, resulting in a failure to learn shared representations consistent with human perception. To address this, the present invention proposes a unified vision-language pre-training method that generates high-quality text descriptions using a multimodal large language model (MLLM), cleans data noise, and utilizes lightweight adapters to adapt to downstream tasks, thereby achieving efficient and accurate quality assessment for IQA and IAA tasks.

[0056] The method of the embodiment of the present invention generates high-quality text descriptions related to image quality and aesthetics by unifying the vision-language pre-training framework and combining it with a multimodal large-scale language model. The embodiment of the present invention utilizes a data purification strategy to optimize the training dataset, and preferably uses a multi-cue integration adapter and a prompt integration strategy to improve the accuracy and generalization ability of the model in evaluating image quality and aesthetics. The data purification strategy and multi-cue integration adapter adopted significantly improve the robustness of the model to noisy data and enhance the model's adaptability to new tasks. This method is particularly suitable for environments with limited annotated data, can achieve zero-sample and few-label image evaluation, and has broad application potential.

[0057] Specific embodiments of the present invention are further described below.

[0058] Quality and aesthetics pre-training

[0059] Vision-language pre-training aims to achieve comprehensive cross-modal understanding by training on large-scale image-text datasets. Thanks to this large-scale pre-training, CLIP (a well-known vision-language model) has demonstrated great potential in assisting multiple vision tasks. Specifically, CLIP consists of an image encoder f and a text encoder g, which are jointly trained to build a shared latent space for images and text through contrastive learning.

[0060] Given a batch of N pairs of images and texts CLIP Extraction of Image Features and text features Use the corresponding encoder. In the pre-training stage, CLIP aims to maximize the cosine similarity of paired image and text features while minimizing the similarity of unmatched images and text. The contrastive learning objective can be expressed as:

[0061]

[0062] Among them I i and T i is the i-th feature in the batch, and τ is the temperature parameter. The final contrastive learning loss can be obtained by taking the average: With this training strategy, CLIP can generate aligned features in the latent space for paired image-text samples.

[0063] In order to achieve visual-language pre-training in the field of image evaluation, text is generated for IQA and IAA datasets, because the IQA dataset lacks text and the text in the IAA dataset contains noise. A multimodal large language model (MLLM) can be used as a teacher model to generate high-quality and quality- and aesthetic-related text data for images. Previous studies have emphasized that it is challenging for MLLM to directly and accurately perceive the quality and aesthetics of input images, which often leads to expressions that are biased towards the positive and have strong hallucinations. Therefore, in order to obtain correct and fine-grained descriptions of quality and aesthetics, a MLLM (M T ) and a task-specific hint Pt and MOS-based text guidance G to obtain a large number of descriptions Y t :

[0064] Y t ~M t (x I ,P t |G)

[0065] To obtain G, images are classified into 5 levels according to MOS, namely {bad, poor, fair, good, perfect}. If an image's MOS ranking is in the top 20% of the score range, its level is assigned to perfect. This method standardizes IQA and IAA datasets of different scales and avoids the bias introduced when directly using MOS scores for training. In addition, P t For IQA(P IQA ) and IAA(P IAA ) tasks to customize. IQA involves sharpness, color balance and noise level, while P IAA Including content, color, lighting and composition. Through these designs, M t was guided to perform image evaluation and obtain the generated description dataset Y IQA and Y IAA .

[0066] In addition to the aesthetic title Y generated by MLLM IAA In addition, there are also IAA datasets that contain human comments, which directly reflect human perceptions of image aesthetics. Combining comments from different people can provide a more comprehensive description of image aesthetics. However, while increasing the diversity of text descriptions, this can introduce noise into the data, as individuals may provide comments that are irrelevant to the image aesthetics. To address this issue, we propose a novel data purification strategy to refine the raw comments in the original dataset.

[0067] Specifically, the aesthetic relevance and information richness ranking (AIR) is introduced to measure the quality of the text corresponding to an image. AIR consists of the aesthetic relevance ranking (AR) and information richness ranking (IR), as Figure 2 In order to obtain AR, we first use the generated aesthetic data YIAA to pre-train a CLIP model to obtain a CLIP model with aesthetic awareness, denoted as CLIP aes Then, it is used to measure the aesthetic relevance score of the image-text pair. For an image with n descriptions, AR can be defined as:

[0068]

[0069] in Represents the i-th title Its corresponding image x IThe aesthetic relevance score between . Note that AR consists of a long integer representing the ranking of the captions in the entire set. To obtain IR, we simply use sentence length as an information richness score to measure the information content of the text. Therefore, for an image with n text captions, IR can be expressed as:

[0070]

[0071] Where Length(·) can output the length of the input sentence. Therefore, the AIR between the image and n captions is:

[0072] AIR=Rank((AR 1 +IR 1 ),···,(AR n +IR n )).

[0073] We select texts with Top-K ranking AIR to build a high-quality and semantically rich aesthetic title dataset, denoted as This strategy ensures that texts relevant to aesthetic perception and rich in information are preserved, thereby improving the quality and richness of the original dataset.

[0074] So far, a high-quality dataset Y on image quality and aesthetics has been obtained, namely Based on this dataset, we pre-trained the CLIP model to obtain UniQA. In this way, the model learns general perceptions of image quality and aesthetics, as well as task-specific perceptions. This pre-trained model can provide strong perceptual priors for downstream image assessment tasks.

[0075] Next, we further propose a lightweight adapter to easily and effectively convert pre-trained models to different datasets.

[0076] Applying pre-trained models to image evaluation tasks

[0077] The pre-trained UniQA contains extensive quality and aesthetic perception information and can be effectively applied to specific IQA and IAA tasks in a zero-shot or supervised manner. Through carefully designed adapters and prompt engineering, the performance of the model in downstream evaluation tasks is further improved.

[0078] During the pre-training phase, the model aligns images and captions relevant to evaluation, giving it a strong ability to understand image quality and aesthetics. Based on this basic model, the visual features can be slightly adjusted to effectively transform image-text alignment into image score predictions for a specific dataset for image evaluation. To this end, a lightweight adapter, the multi-cue integration adapter, is introduced to adapt visual features and inject rich cues for fine-tuning downstream tasks. The adapter consists of two key processes: visual feature adaptation and multi-cue integration prediction, as shown in the figure. Figure 2 As shown in (b).

[0079] Visual feature adaptation step. A learnable residual module is added after the pre-trained image encoder to adjust the visual features to fit the specific evaluation dataset. This module is optimized while keeping the image and text backbone networks frozen, achieving efficient parameter adjustment. The structure of the adapter is as follows Figure 2 As shown in (b). Let I represent the image features extracted from the frozen image encoder, the visual feature adaptation process can be expressed as:

[0080] I'=Normalize(Adapter(I)+I)

[0081] Where Adapter(·) consists of two fully connected layers with ReLU activation function, and I′ represents the adjusted visual features.

[0082] Multi-cue integrated prediction step. A straightforward approach to using the CLIP model for perceptual evaluation is to use a "good image" as an anchor and the cosine similarity between a given image and the text anchor as the evaluation score. However, this approach has two drawbacks: 1) using the absolute value of the similarity as the perceptual score may not be optimal because it only reflects the semantic similarity between the image and the text; 2) a single cue may not fully utilize the rich knowledge of the pre-trained model. Therefore, we propose to use diverse cues to fully explore the capabilities of pre-trained UniQA and convert the absolute similarity scores into relative values ​​for weighting.

[0083] Specifically, the prompt template "{level}image" and five text levels, namely {bad, poor, fair, good, perfect}, are used to construct the prompt. Next, the cosine similarity between the normalized text features of the five prompts and the adjusted visual features I′ is calculated, and the correlation values ​​of the five image-text correspondences are obtained using Softmax(·). These correlation values ​​are weighted by the predefined scoring level to obtain the final evaluation score. This process can be expressed as follows:

[0084]

[0085] Among them C i is the text level score with increasing value, set to {0.2, 0.4, 0.6, 0.8, 1.0}; τ is the temperature parameter, and q is the final evaluation score of a given image.

[0086] In addition, a hint integration strategy is introduced that combines more hint groups to derive the final evaluation score, thereby achieving a more comprehensive understanding of image quality and aesthetics. For example, one can use {extremely blurry, blurry, fair, sharp, extremely sharp} as another set of five text levels. Now, the final evaluation score q f is the average value of all prompt groups and can be described as:

[0087]

[0088] where m denotes the number of prompt groups. This strategy can more fully exploit the multimodal understanding capabilities of pre-trained UniQA and shows significant performance improvements in zero-shot and few-label learning.

[0089] In summary, the present invention proposes a unified visual language multimodal pre-training model for quality and aesthetics, which can promote accurate quality and aesthetic perception in image evaluation tasks. The present invention includes the following improvements:

[0090] (1) Unified Vision-Language Pre-training Method (UniQA): This paper proposes a novel pre-training method that combines image quality and aesthetics assessment tasks and uses a multimodal large language model (MLLM) to generate high-quality text descriptions related to the image, thereby achieving a unified evaluation of image quality and aesthetics.

[0091] (2) Data purification strategy: In order to solve the problem of text noise in the existing IAA dataset, this paper designs a data purification strategy to screen and optimize the dataset through aesthetic relevance and information richness ranking (AIR), thereby improving the quality of training data.

[0092] (3) Design of a multi-cue integration adapter: This paper introduces a lightweight multi-cue integration adapter that can adjust the visual features of the pre-trained model and combine multiple cues to fine-tune the model to adapt to specific image evaluation tasks.

[0093] (4) Prompt integration strategy: In order to further improve the evaluation performance of the model, the prompt integration strategy of the present invention comprehensively considers the quality and aesthetic characteristics of the image by combining multiple prompt groups, thereby obtaining more accurate evaluation results.

[0094] Compared with traditional methods, the main advantages of the embodiments of the present invention include the following aspects: 1) Unified evaluation framework: The present invention provides a unified pre-training framework that can simultaneously handle image quality assessment (IQA) and image aesthetic assessment (IAA) tasks, overcoming the limitation of the existing technology that treats these two tasks in isolation. 2) Dataset optimization: Through innovative data purification strategies, the present invention can effectively extract high-quality image-text pairs, reduce the impact of noisy data on evaluation model training, and improve the efficiency and accuracy of model training. 3) Enhanced model adaptability: The multi-cue integration adapter adopted by the present invention allows the model to be quickly adjusted for different downstream tasks. This lightweight design enables the model to adapt to new tasks with fewer parameters and computational costs. 4) Improved generalization performance: The method of the present invention has been tested on multiple public datasets, demonstrating that it can achieve excellent performance on different datasets and show strong generalization capabilities. 5) Zero-shot and few-label learning capabilities: The present invention is particularly suitable for environments where labeled data is scarce and can perform effective image evaluation with no or only a small amount of labeled data, which is particularly important for data-driven tasks in practical applications. 6) Multimodal Understanding: By leveraging text descriptions generated by a multimodal large language model (MLLM), this paper enhances the model's understanding of image content, making the evaluation results more consistent with human subjective perception. 7) Flexible Prompt Strategy: The prompt integration strategy adopted by this paper can flexibly combine multiple prompts, allowing the model to evaluate images from different perspectives, improving the comprehensiveness and accuracy of the evaluation.

[0095] This invention provides a more accurate and efficient solution for image quality and aesthetics assessment, and is applicable to multiple fields such as image editing, social media content screening, and photographic work evaluation.

[0096] In summary, the present invention provides an efficient, accurate and adaptable solution for image evaluation, which has broad application prospects and practical value.

[0097] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0098] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0099] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0100] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0101] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0102] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0104] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0105] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0106] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0107] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0108] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0109] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A unified visual language model pre-training and adjustment method for image quality and aesthetic evaluation, characterized by: The following steps are involved: a. Use a multimodal large language model (MLLM) to generate text descriptions related to image quality and aesthetics for training images used in image quality and aesthetics assessment tasks, and obtain an image-text description dataset; b. Using data purification strategies to filter and optimize the image-text description dataset by ranking AIR by aesthetic relevance and information richness; In step b, the aesthetic relevance ranking AR and the information richness ranking IR are calculated to generate the aesthetic relevance and information richness ranking AIR, and AIR is used to filter out texts that are closely related to image aesthetics to construct an optimized dataset; c. Pre-train the unified visual language model UniQA on the image-text dataset obtained by screening and optimizing in step b. Using a contrastive learning strategy, the model maximizes the similarity between paired image and text features while minimizing the similarity between mismatched images and text. Step c specifically includes: Use the CLIP model to train on a large-scale image-text dataset; Maximize the cosine similarity of paired image and text features through a contrastive learning strategy; The model is trained using a temperature parameter τ and contrastive learning loss to generate aligned feature representations; the temperature parameter τ is used to adjust the sensitivity of the similarity calculation between image and text features; d. Adapt the visual features of the pre-trained UniQA model to specific image evaluation tasks through lightweight adapters; In step d, the lightweight adapter includes an image encoder and a text encoder that constitute the backbone network of the UniQA model, and is used to extract visual features of the image and semantic features of the text; A learnable residual module is added after the pre-trained image encoder to adjust the visual features to adapt to a specific evaluation dataset; wherein the residual module includes two fully connected layers with ReLU activation functions, and adjusts the visual features of the image through normalization and residual connections, while keeping the image and text backbone networks frozen and only optimizing the residual module; Among them, using the prompt template "{level} image" and predefined text levels, multiple prompt groups are constructed. The cosine similarity between the normalized text features and the adjusted visual features of each prompt group is calculated. The Softmax function is used to obtain the correlation value of the image-text correspondence, and the correlation value is weighted by the predefined incremental value scoring level to obtain the final evaluation score.

2. The unified visual language model pre-training and adjustment method for image quality and aesthetic evaluation according to claim 1, characterized in that: In step a, the multimodal large language model MLLM and specific prompts designed for image quality and aesthetics assessment tasks are used, combined with text guidance based on image quality score MOS, to generate text descriptions related to image quality and aesthetics to form a corresponding description dataset and ; The specific prompts are tailored for image quality assessment (IQA) and image aesthetic assessment (IAA) tasks respectively to guide MLLM to generate task-related text descriptions.

3. The unified visual language model pre-training and adjustment method for image quality and aesthetic evaluation according to claim 2, characterized in that: The text descriptions include descriptions involving sharpness, color balance, and noise level designed for the IQA task, and descriptions including content, color, lighting, and composition designed for the IAA task.

4. The unified visual language model pre-training and adjustment method for image quality and aesthetics evaluation according to any one of claims 1 to 3, characterized in that: In step c, the IAA dataset containing human comments is used together with the optimized image-text description dataset to train the UniQA model.

5. The unified visual language model pre-training and adjustment method for image quality and aesthetics evaluation according to any one of claims 1 to 3, characterized in that: In step c, the contrastive learning loss is calculated by taking the average of the image loss and the text loss.

6. The unified visual language model pre-training and adjustment method for image quality and aesthetics evaluation according to any one of claims 1 to 3, characterized in that: The following steps are also included: e. For the UniQA model fine-tuned with the lightweight adapter in step d, a prompt ensemble strategy is adopted to comprehensively evaluate the quality and aesthetic features of the image by combining multiple prompt groups.

7. The unified visual language model pre-training and adjustment method for image quality and aesthetic evaluation according to claim 6, characterized in that: In step e, the evaluation scores of the multiple prompt groups are averaged to obtain the final evaluation score.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the unified visual language model pre-training and adjustment method for image quality and aesthetic evaluation as described in any one of claims 1 to 7.

Images