A general image assessment method based on a mixture of expert models

Abstract

A universal image evaluation method based on a hybrid expert model can be applied to various image evaluation scenarios to automatically evaluate the quality or aesthetics of an image. The method utilizes a visual encoder and a text encoder to extract the visual and text features of the image, respectively, and calculates the quality or aesthetic score through an adapter. A hybrid evaluation expert module is integrated on the visual encoder and the text encoder, which combines shared evaluation experts and adaptive evaluation experts. The former is responsible for learning general evaluation knowledge, while the latter dynamically learns for a specific data set, adjusts the expert contribution through a router, and effectively manages the annotation deviation in the training of multiple data sets. Furthermore, the present invention also implements a scene-based difference prompt strategy, which enhances the adaptability of the model through scene-specific prompt words, reduces the annotation deviation between data sets, and improves the evaluation accuracy. This method not only improves the generalization and adaptability of the model, but also optimizes parameter efficiency and training efficiency, and has broad application prospects.

Description

Technical Field

[0001] The present invention relates to image evaluation technology, in particular to a universal image evaluation method based on a hybrid expert model. Background Art

[0002] In the fields of Image Quality Assessment (IQA) and Image Aesthetic Assessment (IAA), the research goal is to develop algorithms and models that can automatically assess image quality or aesthetic scores. Furthermore, multi-dataset hybrid training and hybrid expert models are also technologies covered by this patent.

[0003] Image evaluation is a classic image processing task, which mainly includes two subtasks: image quality assessment (IQA) and image aesthetic assessment (IAA). The IQA task focuses on the degree of distortion of the image, while IAA aims to evaluate the beauty of the image. In the era of deep learning, both tasks have achieved significant breakthroughs. For the IQA task, researchers have developed a variety of advanced technologies to improve performance, including multi-level feature aggregation, adaptive convolution, Transformer methods, visual language models, and large language models. In addition to natural image evaluation, there are also a variety of IQA methods for other scenarios, such as face IQA, AI-generated image IQA, and underwater image IQA. For the IAA task, many methods have also been proposed to improve model performance, including loss functions, new Transformer architectures, topic information, and multimodal pre-training.

[0004] Recently, some works have attempted to develop general image assessment algorithms that can be applied to a variety of scenarios. To achieve this goal, one approach is to perform mixed training on multiple IQA datasets. For example, UNIQUE uses multiple real and synthetic natural scene IQA datasets for mixed training, but they only focus on natural image evaluation. Q-Align uses a large language model to unify IQA and IAA tasks, but its inference speed is slow and it only focuses on natural images. In addition, PromptIQA uses image score pairs as prompts to make quality predictions, but requires multiple images as references during inference, resulting in inflexible inference. In addition, these methods have also failed to match the performance of methods fine-tuned for specific tasks.

[0005] Current image evaluation models have the following shortcomings: 1) Insufficient scene versatility: Most existing image evaluation models are designed for specific scenarios, such as natural scene images and face images. These models often perform poorly when applied across scenarios and lack wide applicability. 2) Unable to effectively solve the problem of bias in multiple datasets during mixed training: There is a labeling bias between different datasets, which makes it difficult for the model to be generalized to other datasets after being trained on one dataset. Especially when performing mixed training on multiple datasets, this bias will seriously affect the performance of the model. 3) Limited generalization ability: Existing models may perform well on specific tasks, but when faced with diverse and complex image evaluation tasks, their generalization ability is insufficient, and it is difficult to maintain high performance in unseen images or scenes.

[0006] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0007] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a universal image evaluation method based on a hybrid expert model.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A general image assessment method based on a hybrid expert model, comprising:

[0010] extract visual features of an input image using a visual encoder, extract text features corresponding to the image using a text encoder, and calculate a quality or aesthetic score of the image based on the visual features and text features using an adapter;

[0011] A hybrid evaluation expert module is integrated and used on the visual encoder and text encoder to adaptively learn shared and specific knowledge of different datasets, wherein the hybrid evaluation expert module includes a shared evaluation expert and an adaptive evaluation expert; the shared evaluation expert is used to learn general knowledge of image evaluation, and the adaptive evaluation expert dynamically learns dataset-specific knowledge and distributes the processing contributions of different experts to the input features through a router, thereby achieving effective management of labeling deviations in multi-dataset hybrid training.

[0012] Furthermore, the output of the adapter is obtained by weighted summing the features of multiple evaluation-related textual cues, where the weights are adjusted by a Softmax function and a temperature parameter τ to achieve sensitivity adjustment to different quality or aesthetic levels; wherein the features of the multiple evaluation-related textual cues represent different evaluation levels of the image, and through processing by the adapter, the visual features are matched with the evaluation levels to calculate the final image quality or aesthetic score.

[0013] Furthermore, the process of the adapter calculating the final image quality or aesthetic score is expressed as:

[0014]

[0015] Where q is the score of image quality or aesthetics, Represents the features of five assessment-related text prompts, each T k is a d-dimensional vector representing different textual cues related to image quality or aesthetic evaluation; C k is the weight coefficient corresponding to each text prompt, which is used to adjust the contribution of each text prompt to the final score; I' is the intermediate feature representation obtained after the adapter processes the input image I; Softmax is the normalization function used to combine I' with each T k The dot product result is converted into a probability distribution, where the temperature parameter τ is used to control the smoothness of the Softmax function.

[0016] Furthermore, the adapter includes two fully connected layers and an intermediate ReLU activation function for obtaining an image quality or aesthetic score, and the intermediate feature representation obtained after the adapter processes the input image is calculated through the two fully connected layers and the ReLU activation function in the adapter.

[0017] Furthermore, the shared evaluation expert utilizes the pre-trained weights of the original multimodal model to inherit and preserve the general image evaluation capabilities learned by the model in different contexts;

[0018] During the training of the model, the shared evaluation expert remains frozen;

[0019] The output of the shared evaluation expert is obtained by passing the input features directly to the original feed-forward network FFN, which is part of the CLIP model and is used to generate the output of the shared evaluation expert; it is expressed as:

[0020] Given an input feature x∈R d , the output of the shared evaluation expert is:

[0021] y shared =E shared (x),

[0022] Among them E shared The original feed-forward network FFN implemented as a CLIP model.

[0023] Furthermore, the adaptive evaluation expert includes multiple experts and a router, each expert is responsible for capturing different aspects of information from multiple data sets; the router dynamically adjusts the contribution of each expert based on input features to optimize the adaptability of the model to a specific data set;

[0024] The output of the adaptive evaluation expert is calculated by the weighted sum of the weights assigned to the router and the output of each expert. The router performs a linear transformation on the input features to determine the weight of each expert. The weight assignment is implemented using the Softmax function to reflect the contribution of each expert. It is expressed as:

[0025] Given an input feature x∈R d , output y adaptive Calculated as:

[0026]

[0027] G(x)=Softmax(Wx).

[0028] The router G is a linear transformation of the input feature x, i is the index of each expert, represents the i-th expert, and W is the linear transformation matrix.

[0029] Furthermore, the hybrid evaluation expert module combines the outputs of the shared evaluation expert and the adaptive evaluation expert, and adjusts the contribution of the adaptive expert through a learnable factor σ;

[0030] The output of the hybrid evaluation expert module is the sum of the output of the shared evaluation expert and the output of the adaptive evaluation expert weighted by the σ factor, expressed as:

[0031] y MoAE =y shared +σ·y adaptive

[0032] The initial value of the σ factor is set to zero so that the visual and text encoders generate aligned features;

[0033] In practical applications, the shared evaluation expert remains fixed, while the adaptive evaluation expert is set to be trainable.

[0034] Furthermore, the hybrid evaluation expert module is integrated into the last layers of the visual encoder and the text encoder;

[0035] Before the hybrid evaluation expert module, the visual encoder sequentially extracts the deep visual features of the image layer by layer; starting from a layer close to the output to the last layer, the visual encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module;

[0036] Among them, before the hybrid evaluation expert module, the text encoder extracts the in-depth features of the text layer by layer in sequence; starting from a layer close to the output to the last layer, the text encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module.

[0037] Furthermore, the method also includes: executing a scene-based difference prompt strategy, using different prompts for data sets of different scenes, and providing useful scene-based guidance for mixed data set training; wherein, the difference prompt strategy enhances the adaptability of the model to different scenes by using scene-specific prompt words, thereby reducing the score annotation deviation between different data sets.

[0038] Furthermore, the execution of the scene-based difference prompting strategy includes implementing a scene-based difference prompting strategy, which promotes the model's learning of scene-specific knowledge by customizing prompts for data sets of different image evaluation scenarios; wherein, multiple data sets are classified according to scene features, and differentiated prompts are designed for each set of data sets to enhance the model's recognition and evaluation of each scene feature.

[0039] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the universal image assessment method based on a hybrid expert model is implemented.

[0040] The present invention has the following beneficial effects:

[0041] This paper proposes a universal image evaluation method based on a hybrid expert model and designs a universal image evaluation model with strong generalization performance for quality and aesthetics, which can be applied to various image evaluation scenarios.

[0042] The present invention is a universal image assessment method applicable across a variety of scenarios. By combining a visual encoder and a text encoder, along with the use of adapters, the method enables in-depth analysis of the visual and textual features of an input image, and calculates the image's quality or aesthetic score accordingly. The present invention integrates a hybrid assessment expert module with the visual encoder and text encoder. This module contains shared assessment experts and adaptive assessment experts, enabling the model to not only learn general knowledge about image assessment but also dynamically adapt and learn knowledge specific to different datasets. Through a router mechanism, the model can assign processing contributions from different experts to different input features, effectively managing annotation bias in hybrid training of multiple datasets.

[0043] Furthermore, this paper further enhances the model's adaptability to different scenarios by introducing a scene-based differential cueing strategy. This strategy uses scene-specific cue words to guide model learning, thereby mitigating the bias in score annotations across different datasets. This approach not only improves the model's generalization, enabling it to excel across a wide range of image quality and aesthetic assessment tasks, but also enhances its adaptability, allowing it to demonstrate strong performance on specific tasks and maintain high performance even in complex scenarios.

[0044] The model design of this invention also considers parameter and training efficiency. By fine-tuning only the parameters of the hybrid expert module, while keeping other parameters unchanged, computing resources are saved and training efficiency is improved. The model's flexibility and scalability are also significantly improved. It can be used as a base model through fine-tuning to adapt to new assessment areas, such as medical image quality assessment, demonstrating the model's wide applicability.

[0045] This paper provides a unified evaluation framework capable of handling a variety of image evaluation tasks, reducing the need to develop and maintain multiple specialized models, lowering development costs and time, and offering an efficient, flexible, and broadly applicable technical solution for the field of image evaluation.

[0046] In summary, the present invention provides an efficient, robust, and highly generalized technical solution in the field of multi-scene and multi-task joint image evaluation, which has broad application prospects and practical value.

[0047] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a model architecture diagram of a general image assessment method based on a hybrid expert model according to an embodiment of the present invention.

[0049] Figure 2 Schematic diagram of the structure of the hybrid assessment expert module in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0051] In response to the shortcomings of the aforementioned prior art, the objectives of the present invention include: 1) Developing a universal evaluation model: The present invention aims to develop a universal image evaluation model that can perform effective evaluation across a variety of scenarios, so as to overcome the scene-specificity problem of the prior art. 2) Solving the dataset bias problem: By proposing a new model architecture and training strategy, the present invention aims to reduce the labeling bias between different datasets and improve the performance of the model in multi-dataset mixed training. 3) Enhancing generalization capabilities: The present invention is committed to improving the generalization capabilities of the model so that it can maintain high performance in a variety of image evaluation tasks and unknown scenarios. The pre-trained model can also be applied as a basic model to other image evaluation fields.

[0052] To this end, this paper proposes a general image evaluation model with strong generalization performance for quality and aesthetics, which can be applied to various image evaluation scenarios. The present invention specifically includes the following key points:

[0053] (1) Hybrid Evaluation Expert Module: This paper proposes a novel hybrid evaluation expert module that contains shared experts and adaptive experts to dynamically learn common and specific knowledge from different datasets. This structure allows the model to capture common features across datasets while maintaining adaptability to different datasets.

[0054] (2) Scenario-based Differential Hint Strategy: This paper introduces a scenario-based hint engineering strategy that uses different hints depending on the context of the dataset, providing the model with scenario-specific prior knowledge and guidance. This approach helps the model more effectively learn and adapt to the characteristics of different scenarios during mixed dataset training.

[0055] (3) Universal image evaluation capability: The present invention achieves unified training across multiple datasets, solves the problem of annotation bias between different datasets, improves the generalization capability of the model, and can achieve excellent image evaluation performance in different image quality and aesthetic evaluation tasks.

[0056] (4) Flexibility and scalability of the model: The model proposed in this paper can not only be used for training specific tasks, but can also be used as a basic model to enhance other image assessment tasks, such as medical image quality assessment, through fine-tuning, thus achieving the flexibility and scalability of the model.

[0057] See Figure 1 and Figure 2The embodiment of the present invention provides a general image evaluation method based on a hybrid expert model, comprising: using a visual encoder to extract visual features of an input image, using a text encoder to extract text features corresponding to the image, and using an adapter to calculate the quality or aesthetic score of the image based on the visual features and text features; wherein a hybrid evaluation expert module is integrated and used on the visual encoder and the text encoder to adaptively learn shared and specific knowledge of different data sets, wherein the hybrid evaluation expert module MoAE includes a shared evaluation expert E shared and adaptive assessment experts Among them, the shared evaluation expert is used to learn general knowledge of image evaluation, while the adaptive evaluation expert dynamically learns dataset-specific knowledge and distributes the processing contributions of different experts to the input features through the router, thereby achieving effective management of labeling bias in multi-dataset hybrid training to solve the labeling bias problem in multi-dataset hybrid training.

[0058] like Figure 1 As shown, in a preferred embodiment, the hybrid evaluation expert module MoAE is integrated into the last few layers of the visual encoder and the text encoder to improve the efficiency and effectiveness of the model in processing visual and text features; wherein, before the hybrid evaluation expert module MoAE is integrated, the visual encoder sequentially extracts the in-depth visual features of the image layer by layer; starting from a certain layer close to the output to the last layer, the visual encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module MoAE; before the hybrid evaluation expert module MoAE is integrated, the text encoder sequentially extracts the in-depth features of the text layer by layer; starting from a certain layer close to the output to the last layer, the text encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module MoAE. This integration strategy enables the model to enhance its ability to learn and adapt to the features of different datasets while maintaining parameter efficiency, thereby improving the accuracy and generalization of image evaluation.

[0059] like Figure 2 As shown, in a preferred embodiment, the hybrid evaluation expert module MoAE includes a shared evaluation expert E shared and adaptive evaluation experts, the shared evaluation experts E shared The pre-trained weights of the original multimodal model are used to inherit and retain the general image evaluation capabilities learned by the model in different contexts; during the training process of the model, the shared evaluation expert E shared Keep it frozen to ensure that the knowledge it has learned is not covered or lost in the subsequent training process; the shared evaluation expert E shared The output of is obtained by passing the input features directly to the original feed-forward network FFN, which is part of the CLIP model and is used to generate the output of the shared evaluation expert; it is expressed as:

[0060] Given an input feature x∈R d , the output of the shared evaluation expert is:

[0061] y shared =E shared (x),

[0062] Among them E shared The original feed-forward network FFN implemented as a CLIP model.

[0063] The design of the shared evaluation expert captures common feature representations across scenarios while maintaining the model's multimodal evaluation capabilities, providing a stable basic evaluation capability for the hybrid evaluation expert module.

[0064] like Figure 2 As shown, adaptive assessment experts include multiple experts And a router, each expert is responsible for capturing different aspects of information in multiple data sets to enhance the model's ability to identify the characteristics of a specific data set; the router dynamically adjusts the contribution of each expert according to the input features to optimize the model's adaptability to a specific data set; the output of the adaptive evaluation expert is combined with the weights assigned to the router by the expert The weighted sum of the output is calculated, where the router performs a linear transformation on the input features to determine the weights of each expert. The weight distribution is realized by the Softmax function to reflect the weight of each expert. Contribution; expressed as:

[0065] Given an input feature x∈R d , output y adaptive Calculated as:

[0066]

[0067] G(x)=Softmax(Wx).

[0068] The router G is a linear transformation of the input feature x, i is the index of each expert, represents the i-th expert, and W is the linear transformation matrix.

[0069] In a preferred embodiment, the hybrid evaluation expert module combines the outputs of the shared evaluation expert and the adaptive evaluation expert, and adjusts the contribution of the adaptive expert through a learnable factor σ to achieve feature fusion and knowledge complementarity; the output of the hybrid evaluation expert module is the sum of the output of the shared evaluation expert and the output of the adaptive evaluation expert weighted by the σ factor, expressed as:

[0070] yMoAE =y shared +σ·y adaptive

[0071] The initial value of the σ factor is set to zero to ensure that the ability of the shared evaluation expert is mainly relied on in the early stage of training so that the visual and text encoders can generate aligned features;

[0072] In practical applications, the shared evaluation expert remains fixed, while the adaptive evaluation expert is set to be trainable.

[0073] In a preferred embodiment, the output of the adapter is obtained by weighted summing the features of multiple evaluation-related textual cues, where the weights are adjusted by a Softmax function and a temperature parameter τ to achieve sensitivity adjustment for different quality or aesthetic levels. The features of the multiple evaluation-related textual cues represent different evaluation levels of the image. Through the adapter's processing, the visual features are matched with the evaluation levels to calculate the final image quality or aesthetic score. Preferably, the adapter includes two fully connected layers and an intermediate ReLU activation function to obtain the image quality or aesthetic score. The intermediate feature representation obtained after the adapter processes the input image is calculated using the two fully connected layers and the ReLU activation function in the adapter.

[0074] The process by which the adapter calculates the final image quality or aesthetic score is expressed as:

[0075]

[0076] Where q is the score of image quality or aesthetics, Represents the features of five assessment-related text prompts, each T k is a d-dimensional vector representing different textual cues related to image quality or aesthetic evaluation, such as “bad image, poor image, fair image, good image, perfect image”; C k is the weight coefficient corresponding to each text prompt, which is used to adjust the contribution of each text prompt to the final score. It is a learnable parameter vector. I' is the intermediate feature representation obtained after the adapter processes the input image I. I' is calculated by the two fully connected layers and ReLU activation function in the adapter. Softmax is a normalization function used to combine I' with each T k The dot product result is converted into a probability distribution, where the temperature parameter τ is used to control the smoothness of the Softmax function.

[0077] In a further preferred embodiment, the general image evaluation method based on the hybrid expert model of the present invention also includes: executing a scene-based difference prompting strategy, using different prompts for data sets of different scenes, and providing useful scene-based guidance for hybrid data set training; wherein, the difference prompting strategy enhances the adaptability of the model to different scenes by using scene-specific prompt words, thereby reducing the score annotation deviation between different data sets.

[0078] Preferably, the execution of the scene-based difference prompting strategy includes implementing a scene-based difference prompting strategy, which promotes the model's learning of scene-specific knowledge by customizing prompts for data sets of different image evaluation scenarios; wherein, multiple data sets are classified according to scene features, and differentiated prompts are designed for each set of data sets to enhance the model's recognition and evaluation of each scene feature.

[0079] The present invention is based on a universal image evaluation method of a hybrid expert model and designs an innovative hybrid expert model framework. By integrating shared evaluation experts and adaptive evaluation experts, the framework effectively improves the generalization ability of the model in a variety of image evaluation scenarios. The shared evaluation expert is responsible for capturing common features across datasets, while the adaptive evaluation expert dynamically learns dataset-specific knowledge. The combination of the two enables the model to flexibly adapt to different image evaluation tasks. In addition, the present invention adopts a scene-based difference prompting strategy to further enhance the adaptability of the model to specific scenes and reduce the labeling deviation between different datasets. The integration of the hybrid evaluation expert module not only optimizes parameter efficiency and improves training efficiency, but also enhances the flexibility and scalability of the model, making it easy to adapt to new evaluation fields. This unified evaluation framework reduces the need to develop and maintain multiple dedicated models, reduces development costs and time, and provides an efficient, flexible and broadly applicable technical solution for the field of image evaluation.

[0080] Specific embodiments of the present invention are further described below.

[0081] The present invention proposes a universal image evaluation model that can effectively evaluate the quality and aesthetics of images across multiple scenarios. Specifically, an embodiment of the present invention designs an image evaluation model applicable to multiple scenarios, enabling it to simultaneously handle image quality assessment and image aesthetics assessment tasks in multiple scenarios, including natural images, artificial intelligence-generated images, underwater images, facial images, etc. To achieve this goal, a multi-dataset hybrid training strategy is used. In order to solve the labeling bias problem that exists when training mixed data sets, the present invention proposes a hybrid evaluation expert module and a scene-based difference prompting strategy to learn different feature representations for different evaluation scenarios, effectively realizing multi-data unified pre-training. The mixture of experts (MoE) divides the parameters of a specific part into several subsets, each subset is called an expert, and a router is set to assign different experts to different inputs. The present invention utilizes the MoE structure integrated into the last few layers of the visual encoder and text encoder to alleviate the label bias problem in multi-dataset training and realize effective multi-dataset unified pre-training.

[0082] The overall architecture of the model

[0083] CLIP, as a foundational visual-language model, has demonstrated great potential across a wide range of vision tasks. Specifically, CLIP consists of a Transformer-based visual encoder and a text encoder that generate aligned visual and textual representations for each image-text pair. Leveraging these features, cosine similarity scores can be computed between image and text pairs in different domains or tasks to perform task-specific predictions, including image assessment. Recently, to enhance CLIP's capabilities in image assessment, UniQA fine-tuned CLIP on large-scale synthetic and real-world image-text datasets related to image quality and aesthetics. After task-specific fine-tuning, this approach demonstrated excellent performance on IQA and IAA tasks. However, the model lacks versatility across various image assessment scenarios. Building on the unified training framework proposed by UniQA, this paper proposes two methods to address the annotation bias problem in different datasets, thereby developing a universal image assessment model. The following details its components.

[0084] like Figure 1 As shown in Figure 1, the proposed model uses a visual encoder V to extract visual features I and a text encoder T to extract text features T. After the visual encoder, the model uses a fine-tunable adapter to obtain a score q for image quality or aesthetics. This process can be described as:

[0085]

[0086] in, Features representing five evaluation-related text cues, e.g., “bad image, poor image, fair image, good image, perfect image”. is a learnable vector with an initialization value of [0.2, 0.4, 0.6, 0.8, 1.0], and τ is a temperature parameter. In implementation, Adapter consists of two fully connected layers with a ReLU activation function in the middle. In order to deal with the labeling bias in mixed datasets and perform unified pre-training efficiently, this paper proposes a Mixture of Assessment Experts (MoAE) module to adaptively learn the shared and specific knowledge of different datasets. This paper only integrates the MoAE module into the last few layers of the encoder, such as Figure 1 It is noteworthy that the present invention only fine-tunes the parameters of the MoAE module and keeps other parameters unchanged, making the fine-tuning parameter-efficient, which is a significant advantage of the method of the present invention.

[0087] In addition, the present invention introduces scene-based differential prompts (SDP), which uses different prompts for datasets of different scenes, thereby providing useful scene-based guidance for mixed dataset training.

[0088] Detailed principles of the algorithm model

[0089] In order to develop a unified and universal image evaluation model, the goal of this paper is to combine multiple image evaluation datasets for joint training. Unfortunately, the Mean Opinion Score (MOS) has significant deviations between different datasets, which hinders joint training. To address this challenge, this paper proposes a Mixture of Evaluation Experts (MoAE) module, in which multiple experts are used to learn the diversity biases of different datasets. Figure 2 As shown in Figure 2, the proposed MoAE module consists of a shared evaluation expert for learning general knowledge of image evaluation, and several adaptive evaluation experts for dynamically learning dataset-specific knowledge. The following will introduce them separately:

[0090] Shared Assessment Expert

[0091] Shared Evaluation Expert E sharedBy using the original multimodal model weights, the image evaluation ability of the original model is inherited. During training, the expert remains frozen to ensure that the learned knowledge is preserved. Therefore, the model can capture universal representations in different contexts and maintain its original multimodal capabilities. Given an input feature x∈slice d , the output of the shared evaluation expert is:

[0092] y shared =E shared (x),

[0093] Among them E shared Implemented as the original feed-forward network (FFN) of the CLIP model.

[0094] Adaptive Assessment Expert

[0095] The adaptive evaluation expert module consists of two components: (1) n experts Used to capture different aspects of information from multiple datasets; (2) A router G that adjusts the contributions of different experts based on the input features. Given an input feature x∈R d , output y adaptive It can be calculated as:

[0096]

[0097] G(x)=Softmax(Wx).

[0098] Here, router G is a linear transformation of the input feature x; W is the linear transformation matrix. To avoid unreasonable weights, the present invention uses a Softmax operation to normalize the contribution weights. This ensures that the model can effectively learn dataset-specific knowledge.

[0099] Hybrid Assessment Expert Module (MoAE Module)

[0100] Based on the above experts, the hybrid evaluation expert module combines the features of the two types of experts using a learnable factor σ, as Figure 2 Therefore, the output of the hybrid evaluation expert module can be expressed as:

[0101] y MoAE =y shared +σ·y adaptive

[0102] The σ factor is zero-initialized so that the visual and textual encoders can generate aligned features at the beginning. In practice, we freeze the shared expert and only set the adaptive expert to be trainable. This approach maintains parameter efficiency during hybrid training and preserves the multimodal capabilities of the original model.

[0103] The present invention integrates the hybrid evaluation expert module into the last K layers of the visual and text encoder, as Figure 1 This strategy makes the method of the present invention both effective and efficient. The visual feature extraction process can be described as follows:

[0104] I i =V i (I {i-1} ),i=1,2,…,LK

[0105]

[0106] Where L represents the number of layers of the visual encoder; I i represents the visual features of the i-th layer encoder; V MoAE Represents the visual encoder layer equipped with the MoAE module. The operation of the text branch is similar to that of the visual branch.

[0107] Scenario-based prompting strategy

[0108] To facilitate scene-guided learning, we introduce scene-based differential prompts (SDPs) to help the model acquire scene-specific knowledge from different datasets. We use a hybrid training approach using 12 datasets covering six image assessment scenarios, including synthetically distorted natural image quality assessment (SDN-IQA), realistically distorted natural image quality assessment (ADN-IQA), facial image quality assessment (F-IQA), artificial intelligence-generated image quality assessment (AG-IQA), underwater image quality assessment (U-IQA), and image aesthetic assessment (IAA). We categorize these datasets into five groups based on their scenarios: natural quality, AI-generated quality, underwater quality, facial quality, and natural aesthetics. Specifically, for the face quality assessment dataset, we use terms like "face bad-quality," "face poor-quality," "face fair-quality," "face good-quality," and "face perfect-quality" plus the word "image" as a boost. This strategy effectively distinguishes the feature space of images from different scenes and enhances scene-specific knowledge, thereby mitigating the score annotation bias between different datasets.

[0109] Compared with traditional technologies, the main advantages of the present invention include the following aspects: 1) Enhanced generalization capability: Through a hybrid evaluation expert module and a scenario-based difference prompting strategy, the present invention enables the model to learn across multiple datasets, improving its generalization capability in different scenarios. 2) Improved model adaptability: The adaptive experts in the hybrid expert module can dynamically adjust to specific datasets, enhancing the model's adaptability to the characteristics of different datasets and improving the model's performance on diverse data. 3) Addressing dataset bias: The integrated hybrid evaluation expert module effectively addresses the labeling bias between different datasets, enabling unified training and improving the model's performance on different datasets. 4) Optimizing parameter efficiency: The method of the present invention only fine-tunes the parameters of the hybrid expert module, keeping other parameters unchanged, saving computing resources and improving training efficiency. 5) Scalability and flexibility: The model of the present invention can not only be used for a variety of image evaluation tasks, but can also be adapted to new evaluation domains through fine-tuning, demonstrating the model's flexibility and scalability. 6) Unified evaluation framework: The present invention provides a unified framework that can handle a variety of image evaluation tasks, reducing the need to develop and maintain multiple specialized models. 7) Reduced development costs: Due to the versatility and scalability of the model, developers can quickly develop new image assessment applications based on the model of the present invention, reducing development costs and time.

[0110] In summary, the present invention provides an efficient, robust, and highly generalized technical solution in the field of multi-scene and multi-task joint image evaluation, which has broad application prospects and practical value.

[0111] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0112] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0113] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0114] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0115] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0116] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0118] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0119] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0120] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0121] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0122] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0123] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A general image assessment method based on a hybrid expert model, characterized in that: include: extract visual features of an input image using a visual encoder, extract text features corresponding to the image using a text encoder, and calculate a quality or aesthetic score of the image based on the visual features and text features using an adapter; A hybrid evaluation expert module is integrated and used on the visual encoder and text encoder to adaptively learn shared and specific knowledge of different datasets. The hybrid evaluation expert module includes a shared evaluation expert and an adaptive evaluation expert. The shared evaluation expert is used to learn general knowledge of image evaluation, while the adaptive evaluation expert dynamically learns dataset-specific knowledge. The processing contribution of different experts to input features is distributed through a router, thereby achieving effective management of labeling bias in multi-dataset hybrid training. The shared evaluation expert utilizes the pre-trained weights of the original multimodal model to inherit and preserve the general image evaluation capabilities learned by the model in different contexts; During the training of the model, the shared evaluation expert remains frozen; The output of the shared evaluation expert is obtained by passing the input features directly to the original feed-forward network FFN, which is part of the CLIP model and is used to generate the output of the shared evaluation expert; it is expressed as: Given an input feature , the output of the shared evaluation expert is: ， in The original feed-forward network FFN implemented as a CLIP model; The adaptive evaluation expert includes multiple experts and a router, each of which is responsible for capturing different aspects of information from multiple datasets. The router dynamically adjusts the contribution of each expert based on input features to optimize the model's adaptability to a specific dataset. The output of the adaptive evaluation expert is calculated by the weighted sum of the weights assigned to the router and the output of each expert. The router performs a linear transformation on the input features to determine the weight of each expert. The weight assignment is implemented using the Softmax function to reflect the contribution of each expert. It is expressed as: Given an input feature , output Calculated as: ， ， The router G is a linear transformation of the input feature x, i is the index of each expert, represents the i-th expert, and W is the linear transformation matrix.

2. The universal image assessment method based on a hybrid expert model according to claim 1, wherein: The output of the adapter is obtained by weighted summing the features of multiple evaluation-related text cues, where the weights are adjusted by the Softmax function and the temperature parameter τ to achieve sensitivity adjustment for different quality or aesthetic levels; wherein the features of the multiple evaluation-related text cues represent different evaluation levels of the image, and through the processing of the adapter, the visual features are matched with the evaluation levels to calculate the final image quality or aesthetic score.

3. The universal image assessment method based on a hybrid expert model according to claim 2, wherein: The adapter includes two fully connected layers and a ReLU activation function in the middle, which is used to obtain an image quality or aesthetic score. The intermediate feature representation obtained after the adapter processes the input image is calculated by the two fully connected layers and the ReLU activation function in the adapter.

4. The universal image assessment method based on a hybrid expert model according to claim 2, wherein: The process by which the adapter calculates the final image quality or aesthetic score is expressed as: ， Where q is the score of image quality or aesthetics, Represents the features of five assessment-related text prompts, each is a d dimensional vectors, representing different textual cues related to image quality or aesthetic evaluation; is the weight coefficient corresponding to each text prompt, which is used to adjust the contribution of each text prompt to the final score; It is an adapter Adapter For the input image I The intermediate feature representation obtained after processing; Softmax is a normalization function used to With each The dot product result is converted into a probability distribution where the temperature parameter τ Used to control the smoothness of the Softmax function.

5. The universal image assessment method based on a hybrid expert model according to any one of claims 1 to 4, characterized in that: The hybrid evaluation expert module combines the outputs of the shared evaluation expert and the adaptive evaluation expert through a learnable factor σ To adjust the contribution of adaptive experts; The output of the hybrid evaluation expert module is the output of the shared evaluation expert and the output of the adaptive evaluation expert. σ The sum of the weighted factors is expressed as: ， σ The initial value of the factors is set to zero so that the visual and text encoders generate aligned features; In practical applications, the shared evaluation expert remains fixed, while the adaptive evaluation expert is set to be trainable.

6. The universal image assessment method based on a hybrid expert model according to any one of claims 1 to 4, characterized in that: Integrate the hybrid evaluation expert module into the last few layers of the visual encoder and text encoder; Before the hybrid evaluation expert module, the visual encoder sequentially extracts the deep visual features of the image layer by layer; starting from a layer close to the output to the last layer, the visual encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module; Among them, before the hybrid evaluation expert module, the text encoder extracts the in-depth features of the text layer by layer in sequence; starting from a layer close to the output to the last layer, the text encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module.

7. The universal image assessment method based on a hybrid expert model according to any one of claims 1 to 4, characterized in that: Also includes: A scene-based differential prompt strategy is implemented, using different prompts for datasets of different scenarios to provide useful scene-based guidance for mixed dataset training; wherein, the differential prompt strategy enhances the model's adaptability to different scenarios by using scene-specific prompt words, thereby alleviating the score annotation bias between different datasets.

8. The universal image assessment method based on a hybrid expert model according to claim 7, wherein: The execution of the scene-based difference prompting strategy includes implementing a scene-based difference prompting strategy, which promotes the model's learning of scene-specific knowledge by customizing prompts for datasets of different image evaluation scenarios; wherein, multiple datasets are classified according to scene features, and differentiated prompts are designed for each dataset to enhance the model's recognition and evaluation of each scene feature.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the universal image assessment method based on a hybrid expert model according to any one of claims 1 to 8 is implemented.

Images