Universal image evaluation method based on hybrid expert model

Abstract

The invention discloses a general image evaluation method based on a hybrid expert model, which can be applied to various image evaluation scenes to automatically evaluate the quality or aesthetics of images. According to the method, a visual encoder and a text encoder are used for extracting visual and text features of an image respectively, and a quality or aesthetic score is calculated through an adapter. The visual encoder and the text encoder are integrated with a hybrid evaluation expert module, the hybrid evaluation expert module combines a shared evaluation expert and a self-adaptive evaluation expert, the shared evaluation expert is responsible for learning general evaluation knowledge, the self-adaptive evaluation expert dynamically learns specific data sets, expert contribution is adjusted through a router, and labeling deviation in multi-data-set training is effectively managed. Furthermore, a scene-based difference prompt strategy is implemented, the model adaptability is enhanced through a scene specific prompt word, the annotation deviation between data sets is reduced, and the evaluation accuracy is improved. The method not only improves generalization and adaptability of the model, but also optimizes parameter efficiency and training efficiency, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to image evaluation technology, in particular to a universal image evaluation method based on a hybrid expert model. Background Art

[0002] In the fields of Image Quality Assessment (IQA) and Image Aesthetic Assessment (IAA), the research goal is to develop algorithms and models that can automatically evaluate image quality or aesthetic scores. In addition, multi-dataset hybrid training and hybrid expert models are also technical parts involved in this patent.

[0003] Image evaluation is a classic image processing task, which mainly includes two subtasks: image quality assessment (IQA) and image aesthetic assessment (IAA). The IQA task focuses on the degree of distortion of the image, while IAA aims to evaluate the beauty of the image. In the era of deep learning, both tasks have achieved remarkable breakthroughs. For the IQA task, researchers have developed a variety of advanced techniques to improve performance, including multi-level feature aggregation, adaptive convolution, Transformer methods, visual language models, and large language models. In addition to natural image evaluation, there are also a variety of IQA methods for other scenarios, such as face IQA, AI-generated image IQA, and underwater image IQA. For the IAA task, many methods have also been proposed to improve model performance, including loss functions, new Transformer architectures, topic information, and multimodal pre-training, etc.

[0004] Recently, some works have attempted to develop general image assessment algorithms that can be applied to a variety of scenarios. To achieve this goal, one approach is to perform hybrid training on multiple IQA datasets. For example, UNIQUE uses multiple real and synthetic natural scene IQA datasets for hybrid training, but they only focus on natural image evaluation. Q-Align uses a large language model to unify IQA and IAA tasks, but its inference speed is slow and it only focuses on natural images. In addition, PromptIQA uses image score pairs as prompts to make quality predictions, but requires multiple images as references during inference, resulting in inflexible inference. In addition, these methods have also failed to match the performance of methods fine-tuned for specific tasks.

[0005] The current image evaluation models have the following shortcomings: 1) Insufficient scene versatility: Most existing image evaluation models are designed for specific scenes, such as natural scene images and face images. These models often perform poorly when applied across scenes and lack wide applicability. 2) Unable to effectively solve the problem of bias in multiple datasets during mixed training: There are annotation biases between different datasets, which makes it difficult for the model to generalize to other datasets after being trained on one dataset, especially when performing mixed training on multiple datasets. This bias will seriously affect model performance. 3) Limited generalization ability: Existing models may perform well on specific tasks, but when faced with diverse and complex image evaluation tasks, their generalization ability is insufficient and it is difficult to maintain high performance in unseen images or scenes.

[0006] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0007] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a universal image evaluation method based on a hybrid expert model.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A general image assessment method based on a hybrid expert model, comprising:

[0010] extracting visual features of an input image using a visual encoder, extracting text features corresponding to the image using a text encoder, and calculating a quality or aesthetic score of the image based on the visual features and the text features using an adapter;

[0011] A hybrid evaluation expert module is integrated and used on the visual encoder and text encoder to adaptively learn the shared and specific knowledge of different datasets, wherein the hybrid evaluation expert module includes a shared evaluation expert and an adaptive evaluation expert; the shared evaluation expert is used to learn general knowledge of image evaluation, and the adaptive evaluation expert dynamically learns dataset-specific knowledge, and distributes the processing contributions of different experts to the input features through a router, thereby achieving effective management of annotation deviations in multi-dataset hybrid training.

[0012] Furthermore, the output of the adapter is obtained by weighted summing the features of multiple evaluation-related text prompts, wherein the weights are adjusted by the Softmax function and the temperature parameter τ to achieve sensitivity adjustment to different quality or aesthetic levels; wherein the features of the multiple evaluation-related text prompts represent different evaluation levels of the image, and through the processing of the adapter, the visual features are matched with the evaluation levels to calculate the final image quality or aesthetic score.

[0013] Furthermore, the process of the adapter calculating the final image quality or aesthetic score is expressed as:

[0014]

[0015] Where q is the score of image quality or aesthetics, Represents the features of five evaluation-related textual cues, each T k is a d-dimensional vector representing different textual cues related to image quality or aesthetic evaluation; C k is the weight coefficient corresponding to each text prompt, which is used to adjust the contribution of each text prompt to the final score; I' is the intermediate feature representation obtained by the adapter after processing the input image I; Softmax is a normalization function used to combine I' with each T k The dot product result of is converted into a probability distribution, where the temperature parameter τ is used to control the smoothness of the Softmax function.

[0016] Furthermore, the adapter includes two fully connected layers and an intermediate ReLU activation function for obtaining an image quality or aesthetic score, and the intermediate feature representation obtained by the adapter after processing the input image is calculated through the two fully connected layers and the ReLU activation function in the adapter.

[0017] Furthermore, the shared evaluation expert utilizes the pre-trained weights of the original multimodal model, thereby inheriting and retaining the general image evaluation capabilities learned by the model in different contexts;

[0018] During the training of the model, the shared evaluation expert remains frozen;

[0019] The output of the shared evaluation expert is obtained by passing the input features directly to the original feed-forward network FFN, which is part of the CLIP model and is used to generate the output of the shared evaluation expert; it is expressed as:

[0020] Given an input feature x∈R d , the output of the shared evaluation expert is:

[0021] y shared =E shared (x),

[0022] Where E shared The original feed-forward network FFN implemented as a CLIP model.

[0023] Furthermore, the adaptive evaluation expert includes multiple experts and a router, each expert is responsible for capturing different aspects of information in multiple data sets; the router dynamically adjusts the contribution of each expert according to the input features to optimize the adaptability of the model to a specific data set;

[0024] The output of the adaptive evaluation expert is calculated by the weighted sum of the weight assigned to the router and the output of each expert, wherein the router performs a linear transformation on the input features to determine the weight of each expert, and the weight assignment is implemented by the Softmax function to reflect the contribution of each expert; expressed as:

[0025] Given an input feature x∈R d , output y adaptive Calculated as:

[0026]

[0027] G(x)=Softmax(Wx).

[0028] The router G is a linear transformation of the input feature x, i is the index of each expert, represents the i-th expert, and W is the linear transformation matrix.

[0029] Furthermore, the hybrid evaluation expert module combines the outputs of the shared evaluation expert and the adaptive evaluation expert, and adjusts the contribution of the adaptive expert through a learnable factor σ;

[0030] The output of the hybrid evaluation expert module is the sum of the output of the shared evaluation expert and the output of the adaptive evaluation expert weighted by the σ factor, expressed as:

[0031] y MoAE =y shared +σ·y adaptive

[0032] The initial value of the σ factor is set to zero so that the visual and text encoders generate aligned features;

[0033] In practical applications, the shared evaluation expert is kept fixed, while the adaptive evaluation expert is set to be trainable.

[0034] Further, the hybrid evaluation expert module is integrated into the last layers of the visual encoder and the text encoder;

[0035] Before the hybrid evaluation expert module, the visual encoder extracts the deep visual features of the image layer by layer in sequence; starting from a layer close to the output to the last layer, the visual encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module;

[0036] Among them, before the hybrid evaluation expert module, the text encoder extracts the in-depth features of the text layer by layer in sequence; starting from a layer close to the output to the last layer, the text encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module.

[0037] Furthermore, the method also includes: executing a scene-based difference prompt strategy, using different prompts for data sets of different scenes, and providing useful scene-based guidance for mixed data set training; wherein, the difference prompt strategy enhances the adaptability of the model to different scenes by using scene-specific prompt words, thereby reducing the score annotation deviation between different data sets.

[0038] Furthermore, the execution of the scene-based difference prompting strategy includes implementing a scene-based difference prompting strategy, which promotes the model's learning of scene-specific knowledge by customizing prompts for data sets of different image evaluation scenarios; wherein, multiple data sets are classified according to scene features, and differentiated prompts are designed for each set of data sets to enhance the model's recognition and evaluation of each scene feature.

[0039] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the universal image assessment method based on a hybrid expert model is implemented.

[0040] The present invention has the following beneficial effects:

[0041] The present invention proposes a universal image evaluation method based on a hybrid expert model and designs a universal image evaluation model with strong generalization performance for quality and aesthetics, which can be applied to various image evaluation scenarios.

[0042] The present invention is a universal image evaluation method applicable across multiple scenarios. The method achieves in-depth analysis of the visual and text features of the input image by combining a visual encoder and a text encoder, and using an adapter, and calculates the quality or aesthetic score of the image accordingly. The present invention integrates a hybrid evaluation expert module on the visual encoder and the text encoder. The module contains a shared evaluation expert and an adaptive evaluation expert, so that the model can not only learn general knowledge of image evaluation, but also dynamically adapt and learn knowledge specific to different data sets. Through the router mechanism, the model can assign processing contributions of different experts to different input features, effectively managing annotation deviations in hybrid training of multiple data sets.

[0043] In addition, the present invention further enhances the adaptability of the model to different scenes by introducing a scene-based difference prompt strategy, using scene-specific prompt words to guide model learning, thereby reducing the score annotation bias between different data sets. This approach not only improves the generalization ability of the model, enabling it to perform well in a variety of image quality and aesthetic assessment tasks, but also enhances the adaptability of the model, allowing the model to perform well on specific tasks and maintain high performance when facing complex scenes.

[0044] The model design of the present invention also takes into account parameter efficiency and training efficiency. By only fine-tuning the parameters of the hybrid expert module and keeping other parameters unchanged, computing resources are saved and training efficiency is improved. The flexibility and scalability of the model are also significantly improved. It can be used as a basic model to adapt to new evaluation fields, such as medical image quality assessment, through fine-tuning, showing the wide applicability of the model.

[0045] The present invention provides a unified evaluation framework that can handle a variety of image evaluation tasks, reduces the need to develop and maintain multiple dedicated models, reduces development costs and time, and provides an efficient, flexible and broadly applicable technical solution for the field of image evaluation.

[0046] In summary, the present invention provides an efficient, robust, and highly generalized technical solution in the field of multi-scene and multi-task joint image evaluation, which has broad application prospects and practical value.

[0047] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a model architecture diagram of a general image assessment method based on a hybrid expert model according to an embodiment of the present invention.

[0049] Figure 2 Schematic diagram of the structure of the hybrid assessment expert module in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0051] In view of the shortcomings of the aforementioned prior art, the objectives of the present invention include: 1) Developing a universal evaluation model: The present invention aims to develop a universal image evaluation model that can perform effective evaluation across a variety of scenarios, so as to overcome the scene-specificity problem of the prior art. 2) Solving the dataset bias problem: By proposing a new model architecture and training strategy, the present invention aims to reduce the labeling bias between different datasets and improve the performance of the model in multi-dataset mixed training. 3) Enhancing generalization capabilities: The present invention is committed to improving the generalization capabilities of the model so that it can maintain high performance in a variety of image evaluation tasks and unknown scenarios. The pre-trained model can also be used as a basic model in other image evaluation fields.

[0052] To this end, the present invention proposes a general image evaluation model with strong generalization performance for quality and aesthetics, which can be applied to various image evaluation scenarios. The present invention specifically includes the following key points:

[0053] (1) Hybrid Evaluation Expert Module: This paper proposes a novel hybrid evaluation expert module, which contains shared experts and adaptive experts to dynamically learn common and specific knowledge from different datasets. This structure allows the model to capture common features across datasets while maintaining adaptability to different datasets.

[0054] (2) Scenario-based Differential Hint Strategy: This paper introduces a scenario-based hint engineering strategy, which uses different hints according to the scenario of the dataset to provide the model with scenario-specific prior knowledge and guidance. This approach helps the model learn and adapt to the characteristics of different scenarios more effectively during mixed dataset training.

[0055] (3) Universal image assessment capability: The present invention achieves unified training across multiple datasets, solves the problem of annotation bias between different datasets, improves the generalization capability of the model, and can achieve excellent image assessment performance in different image quality and aesthetic assessment tasks.

[0056] (4) Flexibility and scalability of the model: The model proposed in this invention can not only be used for training of specific tasks, but also can be used as a basic model to enhance other image assessment tasks, such as medical image quality assessment, through fine-tuning, thus achieving the flexibility and scalability of the model.

[0057] See also Figure 1 and Figure 2The embodiment of the present invention provides a general image evaluation method based on a hybrid expert model, comprising: using a visual encoder to extract visual features of an input image, using a text encoder to extract text features corresponding to the image, and using an adapter to calculate the quality or aesthetic score of the image based on the visual features and text features; wherein a hybrid evaluation expert module is integrated and used on the visual encoder and the text encoder to adaptively learn shared and specific knowledge of different data sets, wherein the hybrid evaluation expert module MoAE includes a shared evaluation expert E shared and adaptive assessment experts Among them, the shared evaluation expert is used to learn general knowledge of image evaluation, while the adaptive evaluation expert dynamically learns dataset-specific knowledge and distributes the processing contribution of different experts to the input features through the router, thereby achieving effective management of labeling deviation in multi-dataset mixed training to solve the labeling deviation problem in multi-dataset mixed training.

[0058] like Figure 1 As shown, in a preferred embodiment, the hybrid evaluation expert module MoAE is integrated into the last few layers of the visual encoder and the text encoder to improve the efficiency and effect of the model in processing visual and text features; wherein, in the stage before the integration of the hybrid evaluation expert module MoAE, the visual encoder extracts the deepening visual features of the image layer by layer in sequence; from a certain layer close to the output to the last layer, the visual encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module MoAE; in the stage before the integration of the hybrid evaluation expert module MoAE, the text encoder extracts the deepening features of the text layer by layer in sequence; from a certain layer close to the output to the last layer, the text encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module MoAE. This integration strategy enables the model to enhance the learning and adaptability of the features of different data sets while maintaining parameter efficiency, thereby improving the accuracy and generalization of image evaluation.

[0059] like Figure 2 As shown, in a preferred embodiment, the hybrid evaluation expert module MoAE includes a shared evaluation expert E shared and adaptive evaluation experts, the shared evaluation experts E shared The pre-trained weights of the original multimodal model are used to inherit and retain the general image evaluation capabilities learned by the model in different contexts; during the training process of the model, the shared evaluation expert E shared Keep it frozen to ensure that the knowledge it has learned is not covered or lost in the subsequent training process; the shared evaluation expert E shared The output of is obtained by passing the input features directly to the original feed-forward network FFN, which is part of the CLIP model and is used to generate the output of the shared evaluation expert; it is expressed as:

[0060] Given an input feature x∈R d , the output of the shared evaluation expert is:

[0061] y shared =E shared (x),

[0062] Where E shared The original feed-forward network FFN implemented as a CLIP model.

[0063] The design of the shared evaluation expert captures common feature representations across scenarios while maintaining the multimodal evaluation capabilities of the model, providing a stable basic evaluation capability for the hybrid evaluation expert module.

[0064] like Figure 2 As shown, adaptive assessment experts include multiple experts and a router, each expert is responsible for capturing different aspects of information in multiple data sets to enhance the model's ability to identify the features of a specific data set; the router dynamically adjusts the contribution of each expert according to the input features to optimize the model's adaptability to a specific data set; the output of the adaptive evaluation expert is combined with the weights assigned to the router by the experts The weighted sum of the output is calculated, where the router performs a linear transformation on the input features to determine the weights of each expert. The weight distribution is realized through the Softmax function to reflect the weight of each expert Contribution; expressed as:

[0065] Given an input feature x∈R d , output y adaptive Calculated as:

[0066]

[0067] G(x)=Softmax(Wx).

[0068] The router G is a linear transformation of the input feature x, i is the index of each expert, represents the i-th expert, and W is the linear transformation matrix.

[0069] In a preferred embodiment, the hybrid evaluation expert module combines the outputs of the shared evaluation expert and the adaptive evaluation expert, and adjusts the contribution of the adaptive expert through a learnable factor σ to achieve feature fusion and knowledge complementarity; the output of the hybrid evaluation expert module is the sum of the output of the shared evaluation expert and the output of the adaptive evaluation expert weighted by the σ factor, expressed as:

[0070] yMoAE =y shared +σ·y adaptive

[0071] The initial value of the σ factor is set to zero to ensure that the ability of the shared evaluation experts is mainly relied on in the early stage of training so that the visual and text encoders can generate aligned features;

[0072] In practical applications, the shared evaluation expert is kept fixed, while the adaptive evaluation expert is set to be trainable.

[0073] In a preferred embodiment, the output of the adapter is obtained by weighted summing the features of multiple evaluation-related text prompts, wherein the weights are adjusted by the Softmax function and the temperature parameter τ to achieve sensitivity adjustment for different quality or aesthetic levels; wherein the features of the multiple evaluation-related text prompts represent different evaluation levels of the image, and through the processing of the adapter, the visual features are matched with the evaluation levels to calculate the final image quality or aesthetic score. Preferably, the adapter includes two fully connected layers and an intermediate ReLU activation function for obtaining an image quality or aesthetic score, and the intermediate feature representation obtained by the adapter after processing the input image is calculated by the two fully connected layers and the ReLU activation function in the adapter.

[0074] The process by which the adapter calculates the final image quality or aesthetic score is expressed as:

[0075]

[0076] Where q is the score of image quality or aesthetics, Represents the features of five evaluation-related textual cues, each T k is a d-dimensional vector representing different textual cues related to image quality or aesthetic evaluation, such as “bad image, poor image, fair image, good image, perfect image”; C k is the weight coefficient corresponding to each text prompt, which is used to adjust the contribution of each text prompt to the final score and is a learnable parameter vector; I' is the intermediate feature representation obtained by the adapter after processing the input image I, and I' is calculated by two fully connected layers and ReLU activation function in the adapter; Softmax is a normalization function used to normalize I' with each T k The dot product result of is converted into a probability distribution, where the temperature parameter τ is used to control the smoothness of the Softmax function.

[0077] In a further preferred embodiment, the general image evaluation method based on a hybrid expert model of the present invention also includes: executing a scene-based difference prompt strategy, using different prompts for data sets of different scenes, and providing useful scene-based guidance for mixed data set training; wherein, the difference prompt strategy enhances the adaptability of the model to different scenes by using scene-specific prompt words, thereby reducing the score annotation deviation between different data sets.

[0078] Preferably, the execution of the scene-based difference prompting strategy includes implementing a scene-based difference prompting strategy, which promotes the model's learning of scene-specific knowledge by customizing prompts for data sets of different image evaluation scenarios; wherein multiple data sets are classified according to scene features, and differentiated prompts are designed for each set of data sets to enhance the model's recognition and evaluation of each scene feature.

[0079] The present invention is based on a universal image evaluation method of a hybrid expert model, and designs an innovative hybrid expert model framework, which effectively improves the generalization ability of the model in a variety of image evaluation scenarios by integrating shared evaluation experts and adaptive evaluation experts. The shared evaluation experts are responsible for capturing common features across data sets, while the adaptive evaluation experts dynamically learn data set-specific knowledge. The combination of the two enables the model to flexibly adapt to different image evaluation tasks. In addition, the present invention adopts a scene-based difference prompting strategy to further enhance the adaptability of the model to specific scenes and reduce the annotation deviation between different data sets. The integration of the hybrid evaluation expert module not only optimizes parameter efficiency and improves training efficiency, but also enhances the flexibility and scalability of the model, making it easy to adapt to new evaluation fields. This unified evaluation framework reduces the need to develop and maintain multiple dedicated models, reduces development costs and time, and provides an efficient, flexible and broadly applicable technical solution for the field of image evaluation.

[0080] The specific embodiments of the present invention are further described below.

[0081] The present invention proposes a universal image evaluation model, which can effectively evaluate the quality and aesthetics of images across multiple scenes. Specifically, the embodiment of the present invention designs an image evaluation model applicable to multiple scenes, so that it can simultaneously process image quality evaluation and image aesthetic evaluation tasks in multiple scenes including natural images, artificial intelligence generated images, underwater images, facial images, etc. In order to achieve this goal, a multi-dataset mixed training strategy is used. In order to solve the labeling bias problem existing in mixed training of different data sets, the present invention proposes a mixed evaluation expert module and a scene-based difference prompt strategy to learn different feature representations for different evaluation scenes, and effectively realize multi-data unified pre-training. Mixture-of-Experts (MoE) divides the parameters of a specific part into several subsets, each subset is called an expert, and a router is set to assign different experts to different inputs. The present invention uses the MoE structure integrated in the last few layers of the visual encoder and the text encoder to alleviate the label bias problem in multi-dataset training and realize effective multi-dataset unified pre-training.

[0082] The overall architecture of the model

[0083] CLIP, as a basic visual language model, has shown great potential in a wide range of visual tasks. Specifically, CLIP consists of a Transformer-based visual encoder and a text encoder, which generate aligned visual representations and text representations for each pair of image-text. Using these features, cosine similarity scores between image and text pairs in different fields or tasks can be calculated to perform predictions for specific tasks, including image evaluation tasks. Recently, in order to enhance the capabilities of CLIP in the field of image evaluation, UniQA fine-tuned CLIP on large-scale synthetic and real image quality and aesthetics-related image-text datasets. This method performs well on IQA and IAA tasks after task-specific fine-tuning. However, the model lacks versatility in various image evaluation scenarios. Based on the unified training framework proposed by UniQA, this paper proposes two methods to deal with the annotation bias problem in different datasets, thereby developing a universal image evaluation model. Its components are elaborated in detail below.

[0084] like Figure 1 As shown, the model of the present invention uses a visual encoder V to extract visual features I and a text encoder T to extract text features T. After the visual encoder, the model uses a fine-tunable adapter to obtain a score q for image quality or aesthetics. This process can be described as:

[0085]

[0086] in, Features representing five evaluation-related textual cues, e.g., “bad image, poor image, fair image, good image, perfect image”. is a learnable vector with an initialization value of [0.2, 0.4, 0.6, 0.8, 1.0], and τ is a temperature parameter. In implementation, Adapter consists of two fully connected layers with a ReLU activation function in the middle. In order to cope with the labeling bias in mixed datasets and effectively perform unified pre-training, the present invention proposes a Mixture of Assessment Experts (MoAE) module to adaptively learn the shared and specific knowledge of different datasets. The present invention only integrates the MoAE module into the last few layers of the encoder, such as Figure 1 It is worth noting that the present invention only fine-tunes the parameters of the MoAE module and keeps other parameters unchanged, so that the fine-tuning is parameter-efficient, which is a significant advantage of the method of the present invention.

[0087] In addition, the present invention introduces scene-based differential prompts (SDP), which uses different prompts for datasets of different scenes, thereby providing useful scene-based guidance for mixed dataset training.

[0088] Detailed principles of the algorithm model

[0089] In order to develop a unified and universal image evaluation model, this paper aims to combine multiple image evaluation datasets for joint training. Unfortunately, the Mean Opinion Score (MOS) has significant deviations between different datasets, which hinders joint training. To address this challenge, this paper proposes a Mixed Evaluation Expert (MoAE) module, where multiple experts are used to learn the diversity biases of different datasets. Figure 2 As shown in Figure 1, the proposed MoAE module consists of a shared evaluation expert for learning general knowledge of image evaluation and several adaptive evaluation experts for dynamically learning dataset-specific knowledge. The following will introduce them separately:

[0090] Shared Assessment Expert

[0091] Shared Evaluation Expert sharedBy using the original multimodal model weights, the image evaluation capabilities of the original model are inherited. During training, this expert is kept frozen to ensure that the learned knowledge is preserved. Therefore, the model can capture universal representations in different contexts and maintain its original multimodal capabilities. Given an input feature x∈slice d , the output of the shared evaluation expert is:

[0092] y shared =E shared (x),

[0093] Where E shared Implemented as the original Feed Forward Network (FFN) of the CLIP model.

[0094] Adaptive Assessment Expert

[0095] The adaptive evaluation expert module consists of two components: (1) n experts Used to capture different aspects of information from multiple datasets; (2) A router G that adjusts the contributions of different experts based on the input features. Given an input feature x∈R d , output y adaptive It can be calculated as:

[0096]

[0097] G(x)=Softmax(Wx).

[0098] Here, router G is a linear transformation of input feature x; W is a linear transformation matrix. In order to avoid unreasonable weights, the present invention uses a Softmax operation to normalize the contribution weights. This ensures that the model can effectively learn dataset-specific knowledge.

[0099] Hybrid Assessment Expert Module (MoAE Module)

[0100] Based on the above experts, the hybrid evaluation expert module combines the features of the two types of experts using a learnable factor σ, as Figure 2 Therefore, the output of the hybrid evaluation expert module can be expressed as:

[0101] y MoAE =y shared +σ·y adaptive

[0102] The σ factor is zero-initialized so that the visual and text encoders can generate aligned features at the beginning. In practice, the present invention freezes the shared expert and only sets the adaptive expert to be trainable. This approach maintains parameter efficiency during hybrid training and preserves the multimodal capabilities of the original model.

[0103] The present invention integrates the hybrid evaluation expert module into the last K layers of the visual and text encoders, such as Figure 1 This strategy makes the method of the present invention both effective and efficient. The visual feature extraction process can be described as follows:

[0104] I i =V i (I {i-1} ),i=1,2,…,LK

[0105]

[0106] Where L represents the number of layers of the visual encoder; I i represents the visual features of the i-th layer encoder; V MoAE Represents the visual encoder layer equipped with the MoAE module. The operation of the text branch is similar to that of the visual branch.

[0107] Scenario-based prompting strategy

[0108] In order to promote scene-guided learning, the present invention introduces scene-based differential prompts (SDP) to help the model acquire scene-specific knowledge from different datasets. The present invention uses 12 datasets covering 6 image evaluation scenarios for mixed training, including synthetic distortion natural image quality assessment (SDN-IQA), real distortion natural image quality assessment (ADN-IQA), facial image quality assessment (F-IQA), artificial intelligence generated image quality assessment (AG-IQA), underwater image quality assessment (U-IQA) and image aesthetics assessment (IAA). The present invention classifies these datasets into five groups according to their scenes: natural quality, AI generated quality, underwater quality, facial quality and natural aesthetics. Specifically, for example, for the face quality assessment dataset, the present invention uses words like "face bad-quality", "face poor-quality", "face fair-quality", "face good-quality", "face perfect-quality" plus the word "image" as a boost. This strategy effectively distinguishes the feature space of images from different scenes and enhances scene-specific knowledge, thereby alleviating the score annotation bias between different datasets.

[0109] Compared with traditional technologies, the main advantages of the present invention include the following aspects: 1) Enhanced generalization ability: The present invention enables the model to learn across multiple data sets through a hybrid evaluation expert module and a scene-based difference prompt strategy, thereby improving the generalization ability in different scenarios. 2) Improved model adaptability: The adaptive experts in the hybrid expert module can dynamically adjust to specific data sets, enhancing the adaptability of the model to the characteristics of different data sets and improving the performance of the model on diversified data. 3) Solving the problem of data set deviation: The integrated hybrid evaluation expert module effectively copes with the annotation deviation between different data sets, making unified training possible and improving the performance of the model on different data sets. 4) Optimizing parameter efficiency: The method of the present invention only fine-tunes the parameters of the hybrid expert module and keeps other parameters unchanged, saving computing resources and improving training efficiency. 5) Scalability and flexibility: The model of the present invention can not only be used for a variety of image evaluation tasks, but also can be adapted to new evaluation fields through fine-tuning, showing the flexibility and scalability of the model. 6) Unified evaluation framework: The present invention provides a unified framework that can handle a variety of image evaluation tasks, reducing the need to develop and maintain multiple dedicated models. 7) Reduce development costs: Due to the versatility and scalability of the model, developers can quickly develop new image assessment applications based on the model of the present invention, reducing development costs and time.

[0110] In summary, the present invention provides an efficient, robust, and highly generalized technical solution in the field of multi-scene and multi-task joint image evaluation, which has broad application prospects and practical value.

[0111] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0112] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0113] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0114] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0115] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0116] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0117] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0118] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0119] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0120] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0121] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0122] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0123] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A general image assessment method based on a hybrid expert model, characterized in that: include: extracting visual features of an input image using a visual encoder, extracting text features corresponding to the image using a text encoder, and calculating a quality or aesthetic score of the image based on the visual features and the text features using an adapter; Among them, a hybrid evaluation expert module is integrated and used on the visual encoder and the text encoder to adaptively learn the shared and specific knowledge of different datasets, wherein the hybrid evaluation expert module includes a shared evaluation expert and an adaptive evaluation expert; the shared evaluation expert is used to learn general knowledge of image evaluation, and the adaptive evaluation expert dynamically learns dataset-specific knowledge, and distributes the processing contributions of different experts to the input features through a router, thereby achieving effective management of labeling deviations in multi-dataset hybrid training.

2. The general image assessment method based on a hybrid expert model as claimed in claim 1, characterized in that: The output of the adapter is obtained by weighted summing the features of multiple evaluation-related text prompts, wherein the weights are adjusted by the Softmax function and the temperature parameter τ to achieve sensitivity adjustment for different quality or aesthetic levels; wherein the features of the multiple evaluation-related text prompts represent different evaluation levels of the image, and through the processing of the adapter, the visual features are matched with the evaluation levels to calculate the final image quality or aesthetic score; preferably, the adapter includes two fully connected layers and an intermediate ReLU activation function for obtaining an image quality or aesthetic score, and the intermediate feature representation obtained after the adapter processes the input image is calculated by two fully connected layers and the ReLU activation function in the adapter.

3. The general image assessment method based on a hybrid expert model as claimed in claim 2, characterized in that: The process by which the adapter calculates the final image quality or aesthetic score is expressed as: Where q is the score of image quality or aesthetics, Represents the features of five evaluation-related textual cues, each T k is a d-dimensional vector representing different textual cues related to image quality or aesthetic evaluation; C k is the weight coefficient corresponding to each text prompt, which is used to adjust the contribution of each text prompt to the final score; I' is the intermediate feature representation obtained by the adapter after processing the input image I; Softmax is a normalization function used to combine I' with each T k The dot product result of is converted into a probability distribution, where the temperature parameter τ is used to control the smoothness of the Softmax function.

4. The universal image assessment method based on a hybrid expert model according to any one of claims 1 to 3, characterized in that: The shared evaluation expert utilizes the pre-trained weights of the original multimodal model to inherit and preserve the general image evaluation capabilities learned by the model in different contexts; During the training of the model, the shared evaluation expert remains frozen; The output of the shared evaluation expert is obtained by passing the input features directly to the original feed-forward network FFN, which is part of the CLIP model and is used to generate the output of the shared evaluation expert; it is expressed as: Given an input feature x∈R d , the output of the shared evaluation expert is: and shared =E shared (x), Where E shared The original feed-forward network FFN implemented as a CLIP model.

5. The universal image assessment method based on a hybrid expert model according to any one of claims 1 to 4, characterized in that: The adaptive evaluation expert includes multiple experts and a router, each expert is responsible for capturing different aspects of information in multiple data sets; the router dynamically adjusts the contribution of each expert according to the input features to optimize the adaptability of the model to a specific data set; The output of the adaptive evaluation expert is calculated by the weighted sum of the weight assigned to the router and the output of each expert, wherein the router performs a linear transformation on the input features to determine the weight of each expert, and the weight assignment is implemented by the Softmax function to reflect the contribution of each expert; expressed as: Given an input feature x∈R d , output y adaptive Calculated as: G(x)=Softmax(Wx). The router G is a linear transformation of the input feature x, i is the index of each expert, represents the i-th expert, and W is the linear transformation matrix.

6. The universal image assessment method based on a hybrid expert model according to any one of claims 1 to 5, characterized in that: The hybrid evaluation expert module combines the outputs of the shared evaluation expert and the adaptive evaluation expert, and adjusts the contribution of the adaptive expert through a learnable factor σ; The output of the hybrid evaluation expert module is the sum of the output of the shared evaluation expert and the output of the adaptive evaluation expert weighted by the σ factor, expressed as: and MoAE =and shared +σ·y adaptive The initial value of the σ factor is set to zero so that the visual and text encoders generate aligned features; In practical applications, the shared evaluation expert is kept fixed, while the adaptive evaluation expert is set to be trainable.

7. The universal image assessment method based on a hybrid expert model according to any one of claims 1 to 6, characterized in that: Integrate the hybrid evaluation expert module into the last few layers of the visual encoder and the text encoder; Before the hybrid evaluation expert module, the visual encoder extracts the deep visual features of the image layer by layer in sequence; starting from a layer close to the output to the last layer, the visual encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module; Among them, before the hybrid evaluation expert module, the text encoder extracts the in-depth features of the text layer by layer in sequence; starting from a layer close to the output to the last layer, the text encoder adaptively adjusts the feature extraction process through the hybrid evaluation expert module.

8. The universal image assessment method based on a hybrid expert model according to any one of claims 1 to 7, characterized in that: Also includes: A scene-based difference prompt strategy is implemented to use different prompts for datasets of different scenarios to provide useful scene-based guidance for mixed dataset training; wherein the difference prompt strategy enhances the adaptability of the model to different scenarios by using scene-specific prompt words, thereby alleviating the score annotation deviation between different datasets.

9. The universal image assessment method based on a hybrid expert model as claimed in claim 8, characterized in that: The execution of the scene-based difference prompting strategy includes implementing a scene-based difference prompting strategy, which promotes the model's learning of scene-specific knowledge by customizing prompts for data sets of different image evaluation scenes; wherein, multiple data sets are classified according to scene features, and differentiated prompts are designed for each set of data sets to strengthen the model's recognition and evaluation of each scene feature.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the universal image assessment method based on a hybrid expert model as claimed in any one of claims 1 to 9 is implemented.

Images