Image quality evaluation model training and evaluation method, system, equipment and medium

Abstract

The invention provides an image quality evaluation model training and evaluation method, system and device and a medium, and the method comprises the steps: collecting a basic image, and obtaining a plurality of reference images shot by a shooting device; multiple degradation resources are obtained based on the reference image, the degradation resources are added into a degradation resource pool, and each degradation resource corresponds to a degradation category and a degradation degree; for each basic image, obtaining one or more degradation resources from the degradation resource pool, processing the basic images to obtain degradation images, and adding the degradation images and the basic images into a sample image set; obtaining sample images in the sample image set and label description evaluations corresponding to various evaluation task types; and based on the sample image set and the label description, evaluating and training an image quality evaluation model constructed based on the multi-modal large model. Through adoption of the method and the device, the basic image is degraded so as to enhance the sample image set, and the model has multiple task evaluation capabilities, so that the generalization capability of the model is improved.

Description

Background Art

[0002] Internet Protocol Camera (IPC) plays a vital role in the field of security monitoring. With the advancement of technology and the evolution of market demand, the application scope and functions of IPC continue to expand and optimize. IPC must be able to capture clear images under changing lighting conditions, and usually pursue high resolution to facilitate magnified observation of details, such as facial features, license plates, etc. However, the assessment of IPC image quality (Image Quality Assessment, IQA) has always been a key challenge in this field. This assessment process aims to evaluate the visual quality of IPC images and has a wide range of application value in security monitoring applications, including image compression, transmission, enhancement and restoration.

[0003] IPC needs to adapt to all-weather working environment, covering a variety of complex lighting conditions such as daytime, nighttime, strong light, weak light, rainy weather, etc. In outdoor environment, the captured images may also be disturbed by environmental factors such as rain, fog, dust, etc., which makes it more difficult to evaluate the noise and clarity of the image. In addition, because IPC has PTZ (Pan-Tilt-Zoom) function, the angle and focal length may change when shooting at different times, which further increases the complexity of image quality evaluation.

[0004] Traditional image quality assessment technology can no longer adapt to increasingly complex and changing needs. With the advancement of large-scale model technology, some researchers have begun to try to use visual language models to describe image quality from a linguistic perspective, and have made some progress in the field of general image quality assessment. Previous research methods have mostly focused on specific types of evaluation tasks. In scenarios where multiple types of evaluation tasks are required, multiple image quality assessment models need to be trained separately. In addition, when training image quality assessment models in the existing technology, the available sample image sets fail to meet the needs of practical applications in terms of quantity and quality, and the smaller sample image sets will affect the accuracy of model evaluation. Summary of the invention

[0005] In response to the problems in the prior art, the purpose of this application is to provide an image quality assessment model training and evaluation method, system, device and medium, which degrades the basic image based on the reference image captured by the captured image to enhance the sample image set, and uses label description evaluation training of various evaluation task types to enable the model to have multiple task evaluation capabilities and improve the model generalization ability.

[0006] The present invention provides an image quality assessment model training method, comprising the following steps:

[0007] Collecting a basic image and obtaining a plurality of reference images taken by a photographing device;

[0008] Acquire multiple degraded resources based on the reference image, add the degraded resources into a degraded resource pool, each degraded resource corresponds to a degradation category and a degradation degree;

[0009] For each basic image, one or more degraded resources are obtained from the degraded resource pool, the basic image is processed to obtain a degraded image, and the degraded image and the basic image are added to the sample image set;

[0010] Obtain label description evaluations corresponding to sample images in the sample image set and multiple evaluation task types;

[0011] An image quality assessment model built on a multimodal large model is trained based on sample image sets and label description evaluation.

[0012] In some embodiments, obtaining a plurality of degraded resources based on a reference image includes the following steps:

[0013] Based on multiple benchmark images, the benchmark degraded resources of various degradation categories are extracted by using the degraded resource extraction algorithms corresponding to various degradation categories;

[0014] Based on the benchmark degraded resources of various degradation categories and a variety of degradation coefficients, degraded resources with different degradation degrees under the degradation category are obtained respectively.

[0015] In some embodiments, the image quality assessment model includes a proxy algorithm module and an evaluation large model; training the image quality assessment model constructed based on the multimodal large model based on the sample image set and the label description evaluation includes the following steps:

[0016] Presetting a first prompt word template corresponding to a plurality of evaluation task types; wherein the first prompt word template is used to prompt the image quality evaluation model with the required evaluation task type;

[0017] Inputting the quality evaluation problem and the sample image obtained from the sample image set into the image quality evaluation model, the proxy algorithm module is configured to determine the corresponding evaluation task type and the first prompt word template based on the sample image and the quality evaluation problem, and inputting the sample image and the first prompt word template into the evaluation large model, and the evaluation large model is configured to output the reasoning description evaluation based on the input sample image and the first prompt word template;

[0018] The image quality evaluation model is optimized based on the inference description evaluation and the corresponding label description evaluation.

[0019] In some embodiments, the evaluation large model includes a visual encoder, a visual focus projection layer, and a large language model; the evaluation task type includes a reference image evaluation task type and a non-reference image task type, and / or the evaluation task type includes a single image evaluation task type and a multiple image evaluation task type;

[0020] After the sample images and the first prompt word templates corresponding to various evaluation task types are input into the large evaluation model, the visual encoder extracts visual features from the sample images, and the visual focus projection layer performs difference feature attention calculation on the visual features. The projected visual features are obtained based on the attention calculation results, and the projected visual features and the first prompt word template are input into the large language model to obtain the inference description evaluation output by the large language model.

[0021] In some embodiments, the evaluation task types include evaluation task types with reference images, single image evaluation types without reference images, and multiple image evaluation types without reference images;

[0022] The visual focus projection layer performs differential feature attention calculation on visual features, including the following steps:

[0023] For the evaluation type with reference image, the query is obtained according to the visual feature encoding of the reference image in the sample image, the key is obtained according to the visual feature encoding of the image to be evaluated in the sample image, and the value is obtained according to the difference visual feature encoding between the image to be evaluated and the reference image, and the difference feature attention calculation is performed;

[0024] For the single image evaluation type without reference image, query, key and value are obtained according to the visual feature encoding of the sample image, and differential feature attention calculation is performed;

[0025] For the multiple image evaluation type without reference images, select two adjacent sample images, obtain the query according to the visual feature encoding of the previous sample image, obtain the key according to the visual feature encoding of the next sample image, and obtain the value according to the difference visual feature encoding of the two adjacent sample images, and perform the difference feature attention calculation.

[0026] In some embodiments, the evaluation task type includes a single image evaluation type and a multiple image evaluation type;

[0027] Obtaining label description evaluations of sample images in a sample image set includes the following steps:

[0028] The degradation category and degradation degree corresponding to the sample image are used as the degradation label of each sample image;

[0029] Select multiple sample image groups, and obtain the sorting positions of multiple first sample images in each sample image group, wherein, for sample image groups of the same degradation category, sort the first sample images according to the degree of degradation, and determine the sorting position of each first sample image; for sample image groups of different degradation categories, input the first sample images into a trained image scoring model to obtain image scores, and sort the first sample images according to the image scores;

[0030] Pre-constructing second prompt word templates corresponding to various evaluation task types, the second prompt word templates are used to indicate the format of the label description evaluation to the description evaluation generation model;

[0031] Corresponding to the single image evaluation type, the sample image, the degraded label and the corresponding second prompt word template are input into the description evaluation generation model constructed based on the multimodal large model to obtain the label description evaluation;

[0032] For multiple image evaluation types, each sample image group, the degradation label of each first sample image, the sorting position and the corresponding second prompt word template are input into the description evaluation generation model constructed based on the multimodal large model to obtain the label description evaluation.

[0033] In some embodiments, the following steps are used to train an image quality assessment model based on a multimodal large model:

[0034] Perform multiple batches of image quality assessment model training based on sample image sets and label description evaluations;

[0035] After each batch of training is completed, if the training of the image quality assessment model is still not completed, continue with the following steps:

[0036] Select a test sample image from the sample image set, input the image quality evaluation model, obtain the test reasoning description evaluation, and determine the corresponding test label description evaluation;

[0037] Input the test sample image, the test reasoning description evaluation, and the test label description evaluation into the result evaluation model to obtain the description evaluation professional comparison result;

[0038] When it is determined that the test reasoning description evaluation is more professional according to the description evaluation professional comparison result, the test reasoning description evaluation is used as the label description evaluation of the test sample image in the next batch of training process.

[0039] The present application also provides an image quality assessment method, which is implemented based on the image quality assessment model trained by the above-mentioned image quality assessment model training method, and the method includes the following steps:

[0040] Obtaining the image to be processed taken by the shooting device;

[0041] The image to be processed and the quality assessment problem are input into the image quality assessment model to obtain the reasoning description evaluation output by the image quality assessment model.

[0042] The embodiment of the present application also provides an image quality evaluation system for implementing the above-mentioned image quality evaluation method, the system comprising:

[0043] An image acquisition module is used to acquire the image to be processed taken by the shooting device;

[0044] The image evaluation module is used to input the image to be processed and the quality evaluation problem into the image quality evaluation model, and obtain the reasoning description evaluation output by the image quality evaluation model.

[0045] The present application also provides an image quality evaluation device, including:

[0046] processor;

[0047] a memory storing executable instructions of the processor;

[0048] The processor is configured to perform the steps of the image quality assessment method by executing the executable instructions.

[0049] The embodiment of the present application also provides a computer-readable storage medium for storing a program, and when the program is executed by a processor, the steps of the image quality assessment method are implemented.

[0050] By adopting the image quality assessment model training and assessment method, system, device and medium of the present application, the following beneficial effects are achieved:

[0051] By adopting the image quality assessment model training method of the present application, first, a degraded resource pool is constructed based on a reference image, where the reference image is an actual image captured by a shooting device, so that the degraded resources can be closer to the degradation that may occur in actual application scenarios. The basic image is degraded based on the degraded resource pool to obtain multiple degraded images. Through degradation processing of different degrees of degradation categories, richer data enhancement means are introduced, thereby performing high-quality enhancement on the sample image set, which can improve the generalization ability of the image quality assessment model when facing a small sample data set in actual applications, and improve the evaluation accuracy of the trained image quality assessment model. When training the image quality assessment model, the image quality assessment model is given multiple task evaluation capabilities through label description evaluation training of multiple evaluation task types, and multiple different evaluation tasks are unified by one image quality assessment model, thereby expanding the scope of application of the model and enabling the model to meet the usage requirements in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings.

[0053] Figure 1 is a flowchart of an image quality assessment model training method according to an embodiment of the present application;

[0054] Figure 2 is a flow chart of performing quality degradation processing on a basic image according to an embodiment of the present application;

[0055] Figure 3 is a flowchart describing the evaluation generation process of an embodiment of the present application;

[0056] Figure 4 is a schematic diagram of a sample image and a quality assessment problem input into an image quality assessment model according to an embodiment of the present application;

[0057] Figure 5 is a flow chart of an image quality evaluation method according to an embodiment of the present application;

[0058] Figure 6 is a schematic diagram of an image quality evaluation system according to an embodiment of the present application;

[0059] Figure 7 is a structural schematic diagram of an image quality evaluation device according to an embodiment of the present application;

[0060] Figure 8 It is a schematic diagram of the structure of a computer storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and the concepts of the example embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted. Although "first" or "second" etc. are used in this specification to represent certain features, they are only used to represent the function and are not used as a limitation on the number and importance of specific features.

[0062] like Figure 1 As shown, the present application provides an image quality assessment model training method, comprising the following steps:

[0063] S110: Acquire a basic image, and obtain a plurality of reference images captured by a shooting device;

[0064] The shooting device is, for example, a webcam, but the present application is not limited thereto, and the shooting device may also be other devices such as a camera and a mobile phone;

[0065] Here, the basic image refers to a high-quality image collected in advance, which may be a high-quality image captured by a shooting device under relatively good shooting conditions, or a high-quality image collected through a network that is similar to or the same as the scene of the shooting device;

[0066] The reference image is an actual image captured by a camera in an actual application scenario. When the camera captures the reference image, various actual shooting conditions can be simulated, such as shooting a moving object or shooting in a low ambient brightness environment. The reference image can be processed to obtain a degraded resource, and the obtained degraded resource is more in line with the image captured in the actual application scenario.

[0067] Here, the photographing device may be a photographing device with the same usage scenario as the photographing device that finally uses the image quality evaluation model to evaluate the photographed image. By extracting degraded resources from the reference image of the photographing device in the actual application scenario, it is helpful to make the degraded image processed based on the degraded resources more consistent with the actual low-quality image photographed by the photographing device.

[0068] S120: Acquire multiple degraded resources based on the reference image, and add the degraded resources into a degraded resource pool, where each degraded resource corresponds to a degradation category and a degradation degree;

[0069] Degradation resources refer to degradation parameters required to achieve the degradation category and degradation degree when degrading an image. For example, the degradation resources when compressing and degrading an image are compression ratios corresponding to different compression degrees, and the degradation resources when blurring an image include blur kernel parameters for different blurring degrees.

[0070] S130: for each basic image, obtaining one or more degraded resources from the degraded resource pool, processing the basic image to obtain a degraded image, and adding the degraded image and the basic image to the sample image set;

[0071] Using degraded resources to process the basic image means degrading the basic image to obtain a degraded image. Each basic image can obtain multiple degraded images corresponding to different degradation categories and different degradation degrees, thereby greatly enriching the sample image set;

[0072] S140: Obtaining label description evaluations corresponding to sample images in a sample image set and multiple evaluation task types;

[0073] The label description evaluation is the label used by the image quality evaluation model during training, and its content may include quality rating and / or quality evaluation detailed description; the quality rating includes ratings for various dimensions of the image (such as transparency, brightness, clarity, noise, etc.) and / or an overall image rating combined with various dimensions; the quality evaluation detailed description refers to a textual description of the evaluation of various dimensions of the details in the image (such as the transparency of a specific location area of ​​the image is poor, the clarity of a specific location area of ​​the image is high, etc.), which can more accurately show the detail quality of the evaluated image. The quality evaluation detailed description can describe the identified degradation category, degree of degradation, and the impact of the degradation on the image quality. Combined with the image quality rating and quality evaluation detailed description, it can provide users with professional and in-depth image quality analysis; and the label description evaluation can be used for training of a variety of different task types; the label description evaluation is associated with the corresponding sample image, and one sample image can be associated with multiple label description evaluations;

[0074] S150: training an image quality assessment model based on a multimodal large model based on a sample image set and label description evaluation;

[0075] In this embodiment, the image quality assessment model is constructed based on a multimodal large model and undergoes a training and fine-tuning process of sample image sets and label description evaluations. The multimodal large model is a type of artificial intelligence model that can process and fuse information from multiple modalities. When training the image quality assessment model, sample images and quality prompt questions in the sample image set are input into the image quality assessment model to obtain the reasoning description evaluation output by the image quality assessment model. The reasoning description evaluation includes a reasoning quality score and / or a reasoning detail description. The reasoning description evaluation and the label description evaluation are compared to analyze the model evaluation effect. The image quality assessment model is optimized and fine-tuned based on the model evaluation effect. Furthermore, the image quality assessment model is trained based on label description evaluations corresponding to multiple evaluation task types, so that the image quality assessment model has multiple task evaluation capabilities and improves the model generalization ability.

[0076] When training a large multimodal model, a large amount of sample data is required to improve the model accuracy. It is difficult to directly obtain a large amount of degraded data and corresponding degraded labels in real industrial scenes. This application expands the training sample image set by collecting high-quality basic images and degrading the basic images based on the degraded resources corresponding to the real scenes of the shooting equipment, which is conducive to meeting the high demand for the number of sample images in the training of large multimodal models.

[0077] By adopting the image quality assessment model training method of the present application, first, through steps S110 and S120, a degraded resource pool is constructed based on a reference image, and the reference image is an actual image obtained by shooting with a shooting device, so that the degraded resource can be closer to the degradation that may occur in the actual application scenario. Through step S130, the basic image is degraded based on the degraded resource pool to obtain multiple degraded images. Through different degrees of degradation processing of different degradation categories, richer data enhancement means are introduced, so that the sample image set is enhanced with high quality, which can improve the generalization ability of the image quality assessment model when facing a small sample data set in actual application, and improve the evaluation accuracy of the trained image quality assessment model. Then, through steps S140 and S150, the image quality assessment model is trained, and through label description evaluation training of multiple evaluation task types, the image quality assessment model has multiple task evaluation capabilities, and multiple different evaluation tasks are unified by one image quality assessment model. The scope of application of the model is expanded, so that the model can meet the usage requirements in various scenarios.

[0078] Through the analysis of the digital image processing imaging process and application scenarios, this application summarizes and classifies the degradation information that may exist in the images captured by the network camera, which can be roughly divided into the following 10 degradation categories, such as blur, noise, color cast, compression, brightness enhancement / reduction, contrast enhancement / reduction, saturation enhancement / reduction, oversharpening, pixelation, and dead black. According to the 10 degradation categories defined above and setting different degradation levels, a degradation resource pool is constructed by obtaining specific degradation parameters of various degradation levels of various degradation categories. The specific parameters of various degradation categories are obtained from the real low-quality images captured by the shooting device. The 10 degradation categories here are only examples. In different implementations, the degradation categories included in the degradation resource pool can be one or more of the above 10 degradation categories, and can also include other degradation categories other than the above 10 degradation categories. The number of levels of degradation degree of each degradation category can be the same or different.

[0079] In this embodiment, in step S120, obtaining a plurality of degraded resources based on the reference image includes the following steps:

[0080] Based on multiple benchmark images, the benchmark degraded resources of various degradation categories are extracted by using the degraded resource extraction algorithms corresponding to various degradation categories;

[0081] The degraded resource extraction algorithm corresponding to each degradation category varies with the type of the degradation category. The following examples introduce optional implementation methods of the degraded resource extraction algorithm for several degradation categories. The baseline degraded resource corresponding to each degradation category can be obtained through the degraded resource extraction algorithm. Each degradation category corresponds to a baseline degraded resource.

[0082] Based on the baseline degraded resources and multiple degradation coefficients of various degradation categories, the degraded resources with different degradation degrees under the degradation category are obtained respectively. For example, for the i-th degradation category, its baseline degraded resource is represented by Pi, and the degradation category has multiple degradation degrees. For the j-th degradation degree, its degradation coefficient is represented by Kj, and the degradation coefficient is a degradation degree weight. Then the j-th degraded resource of the i-th degradation category can be represented as Pi*Kj.

[0083] Take the shooting device as a network camera as an example. For the noise degradation category, its degradation resource extraction algorithm includes: using a shooting device (such as a network camera) to shoot two consecutive RGB images (considering that the image quality needs to be evaluated in the use scenario of the network camera, only 2D noise is considered). Since the noise in the frequency domain is easier to separate, the two RGB images are converted from the time domain to the frequency domain using fast Fourier transform, and the two images are subtracted in the frequency domain to obtain a residual image, which is used as the baseline degradation resource of the noise degradation category. By setting degradation coefficients with different weights for the baseline noise image (if the coefficient is greater than 1, the noise intensity is greater than the baseline degradation resource, and if the coefficient is less than 1, the noise intensity is less than the baseline degradation resource), noise spectrum images of different intensities are obtained, and the noise spectrum images are respectively used as degradation resources of different degradation degrees under the noise degradation category. When using the noise degradation resource to degrade the image, the different noise spectrum images are multiplied with the high-quality basic image, and then the inverse Fourier transform is performed to the time domain to obtain noise degradation images with different noise degrees.

[0084] For the blur degradation category, considering that the image scene of the network camera generally has image blur caused by lens shake, the blur degradation category can use simulation technology to obtain blur parameters. Specifically, the degradation resource extraction algorithm of the blur degradation category includes: using a shooting device (such as a network camera) to shoot multiple consecutive RGB images (such as 10 images, the shooting scene needs to contain moving objects), and then averaging the multiple RGB images to obtain a fused image. Since the fused image contains moving objects, it will contain motion smears around the moving objects, and the motion smears will contain certain blur information; by constructing multiple filter kernels of different sizes (such as 1x1, 3x3 and 5x5, etc.), the blur information of the moving area in the fused image is extracted. The fuzzy information extraction includes: using filter kernels of different sizes to perform sliding window search on the moving area, and then averaging the parameters of a filter kernel when sliding over different areas as the reference blur parameters of the filter kernel, and different fuzzy parameters can be obtained by using the superposition of parameters of different filter kernels of the same size and the combination of filter kernels with different weights. The fuzzy parameters obtained by the fuzzy information extraction are used as the reference degradation resources of the fuzzy degradation type. Degraded resources can be divided into fuzzy parameters of various degrees according to the degree of blur. Specifically, the fuzzy parameters of the base degraded resources are multiplied by different degradation coefficients to obtain degraded resources of various degradation degrees under the fuzzy degradation category. When the fuzzy degraded resources are used to degrade the base image, different fuzzy degraded images can be obtained by applying the degraded resources of different degradation degrees to the high-quality base image.

[0085] Color cast, brightness enhancement / reduction, contrast enhancement / reduction, and saturation enhancement / reduction all belong to the color dimension. By converting the image into the HSV domain, the H (hue) value, S (saturation) value, and V (brightness) value are disturbed differently to obtain images with different brightness, contrast, and saturation as corresponding degradation resources. For example, for the color cast degradation category, its degradation resource extraction algorithm includes: converting the reference image captured by the shooting device into the HSV domain, perturbing the H value, S value, and V value by a preset ratio / preset value, and obtaining the reference degradation resource of the color cast degradation category. Multiplying the reference degradation resource by various degradation coefficients, the degradation coefficients of various degradation degrees under the color cast degradation category are obtained. For the brightness enhancement / reduction degradation category, its degradation resource extraction algorithm includes: converting the reference image captured by the shooting device into the HSV domain, perturbing the V value by a preset ratio / preset value, and obtaining the reference degradation resource of the brightness enhancement / reduction degradation category. The reference degradation resource is multiplied by various degradation coefficients to obtain degradation coefficients of various degradation degrees under the brightness enhancement / reduction degradation category. For the contrast enhancement / reduction degradation category, its degradation resource extraction algorithm includes: converting the reference image captured by the shooting device to the HSV domain, perturbing the H value, S value, and V value by a preset ratio / preset value, and obtaining the reference degradation resource of the contrast enhancement / reduction degradation category. The reference degradation resource is multiplied by various degradation coefficients to obtain degradation coefficients of various degradation degrees under the contrast enhancement / reduction degradation category. For the saturation enhancement / reduction degradation category, its degradation resource extraction algorithm includes: converting the reference image captured by the shooting device to the HSV domain, perturbing the S value by a preset ratio / preset value, and obtaining the reference degradation resource of the saturation enhancement / reduction degradation category. The reference degradation resource is multiplied by various degradation coefficients to obtain degradation coefficients of various degradation degrees under the saturation enhancement / reduction degradation category.

[0086] The pixelation degradation category is obtained by applying different compression ratios to the image. The degradation resources of the pixelation degradation are different compression ratios. The degradation resource extraction algorithm includes: setting a base compression ratio as the base degradation resource. Multiplying the base degradation resource by multiple degradation coefficients to obtain degradation resources of different degradation degrees under the pixelation degradation category. The compression degradation category is compressed by converting the image into a jpeg image, and a certain compression ratio can be set. The compression ratio corresponds to the degradation resource. The dead black degradation category is a manifestation of a local image, which often occurs when the lighting conditions are insufficient. The degradation resource extraction algorithm includes: by using a shooting device (such as a network camera) to shoot multiple actual scenes under extremely low illumination, multiple dead black images can be obtained, and then by cutting the dead black image into different image blocks, multiple dead black image blocks and the pasting positions of the dead black image blocks in the image are obtained as the base degradation resources of the dead black degradation category. The size and / or position of the dead black image block in the image are multiplied by various degradation coefficients to obtain degradation resources of various degradation degrees. When the dead black degradation resource is used to degrade the basic image, the image block is pasted to the corresponding position of the basic image to simulate the occurrence of the dead black phenomenon.

[0087] When performing sample image enhancement, for each base image, one or more degraded resources are obtained from the degraded resource pool, and the base image is processed to obtain a degraded image, so that a large number of degraded images can be obtained based on a limited number of base images to achieve sample data enhancement. When obtaining degraded resources, degraded resources can be selected according to certain preset rules, or one or more degraded resources can be randomly selected from the degraded resource pool. The order in which degraded resources are added to the base image is arbitrary, and there is no order, and the same degraded resource can be added repeatedly. Each degradation category has different degrees. Therefore, a large number of complex and diverse sample image sets can be generated in the end.

[0088] The evaluation task type includes a reference image evaluation task type (Full-Reference) and a no-reference image task type (No-Reference), and / or the evaluation task type includes a single image evaluation task type and a multiple image evaluation task type. In this embodiment, the evaluation task type includes a single image evaluation task type with a reference image, a multiple image evaluation task type with a reference image, a single image evaluation task type without a reference image, and a multiple image evaluation task type without a reference image as an example for explanation. In other alternative implementations, the evaluation task type may include only some of the task types listed here, or may include other task types not listed here. The specific task type is pre-set by the staff when training the image quality evaluation model.

[0089] Through training, the image quality evaluation model that can perform multiple evaluation task types can evaluate the quality of an image from multiple professional perspectives, and can also compare and evaluate multiple images, evaluate each image separately, and can effectively evaluate task types with or without reference images, and can also compare and evaluate a single image or multiple images, greatly improving the scope of application of the image quality evaluation model. When users need different evaluation task types at different times, one image quality evaluation model can meet the needs of multiple task types, without the need to train a large multimodal model for each task type, and without the need to replace the large multimodal model used every time the demand is switched.

[0090] When degrading a single image, one or more degradation categories can be randomly selected, and the degree of degradation can be determined based on a random number. When degrading multiple images, one or more degradation categories can also be randomly selected. After processing the degradation resource pool, the diversity of samples can be effectively increased, and they are closer to real degraded samples. This method can not only generate degraded images of the same degradation but different degrees, but also generate a batch of degraded images of different degradation categories and random degrees, thereby increasing the complexity of training samples. When degrading, while saving the degraded image, a JSON file is automatically generated, which contains the save path of the high-quality original image and each degraded image, as well as the added degradation category and degradation degree as degradation labels.

[0091] Figure 2 This is a flow chart of downgrading a basic image according to an embodiment of the present application. In step S200, for each basic image, one or more degraded resources are obtained from a degraded resource pool, and the basic image is processed to obtain a degraded image, including the following steps:

[0092] Determine the type of evaluation task for which the generated degraded image is intended;

[0093] For a single image evaluation type (including a single image evaluation type with a reference image and a single image evaluation type without a reference image), for each basic image, one or more degraded resources are randomly obtained from the degraded resource pool, and the basic image is processed to obtain a degraded image for the single image evaluation type;

[0094] For multiple image evaluation types (including multiple image evaluation types with reference images and multiple image evaluation types without reference images), for each basic image, one or more degraded resources are randomly obtained from the degraded resource pool, and the basic image is processed with the same degraded resources at different degradation degrees to obtain multiple degraded images for the multiple image evaluation types, or, the basic image is processed with random degraded resources at random degradation degrees to obtain multiple degraded images for the multiple image evaluation types.

[0095] For example, a random number r is obtained, and it is determined whether the random number is greater than or equal to 0.5. If r≥0.5, one or more degraded resources are randomly obtained from the degraded resource pool, and the base image is processed with different degradation degrees of the same degraded resources to obtain multiple degraded images for multiple image evaluation types. If r<0.5, the base image is processed with random degradation degrees of random degradation resources to obtain multiple degraded images for multiple image evaluation types. The classification of different degradation methods according to the size of r here is only an example. In actual applications, only the processing method of different degradation degrees of the same degraded resources can be selected, or only the processing method of random degradation degrees of random degradation resources can be selected, or other division methods can be adopted, all of which are within the scope of protection of this application. In another embodiment, it can be set that when r≥0.5 and r<0.95, one or more degraded resources are randomly obtained from the degraded resource pool, and the base image is processed with the same degraded resources at different degradation degrees, and when r≥0.95, the base image is not degraded, and the base image is directly added to the sample image set. The degradation category and degradation degree corresponding to the base image can be set to 0 or other default values. The base image can also be used as an image to be evaluated during training, so that the image quality evaluation model is not only evaluated for degraded images, but also for high-quality base images during training. The threshold values ​​of r here are 0.5 and 0.95 for example only, and are not intended to limit the scope of protection of this application.

[0096] In this embodiment, the quality evaluation result includes a quality score and a quality evaluation detail description. The quality score includes a score for each dimension of the image and an overall score for the image combined with each dimension; the quality evaluation detail description refers to a textual description of the evaluation of each dimension of the details in the image. A comprehensive and multi-angle evaluation result of the image can be obtained through the quality score and the quality evaluation detail description. In order to obtain a professional and detailed evaluation text description and taking into account the particularity of the network camera image scene, a set of comprehensive and detailed evaluation dimension templates are determined by analyzing and screening a number of professional image quality subjective evaluation reports. The evaluation dimension template includes six dimensions such as transparency, brightness, clarity, noise, color reproduction and dead black to score and evaluate a single image or multiple images in detail. The six dimensions here are only examples. In other implementations, the evaluation dimension combination may also include only some of the six dimensions, or may also include other dimensions outside the six dimensions, which also fall within the scope of protection of this application. When training the image quality evaluation model, the output reasoning result of the image evaluation template needs to be adjusted according to the evaluation dimension template so that the output reasoning result of the image quality evaluation model can conform to the form of the evaluation dimension template. Specifically, by adding label description evaluation to sample images as an evaluation dimension template, when training the image quality assessment model, the image quality assessment model is fine-tuned based on the comparison results of the output inference results of the image quality assessment model and the label description evaluation, so that the output of the image quality assessment model is close to the label description evaluation.

[0097] In order to obtain more professional label description evaluation and reduce the manual annotation workload of staff, this application also uses a description evaluation generation model to generate label description evaluation for sample images, and the label description evaluation includes a quality score label and a quality evaluation detail description label. The description evaluation generation model can be built based on an open source multimodal large model with a large number of parameters, and in order to ensure the privacy of sample data, the description evaluation generation model can be deployed on a local server. Through a two-stage description label generation process, a professional and accurate label description evaluation is obtained, and there is no need for staff to manually add labels for a large number of sample images, saving the workload of staff. When using the description evaluation generation model to generate label description evaluations, in order to be able to generate professional label description evaluations corresponding to different evaluation task types, second prompt word templates corresponding to different evaluation task types are pre-set.

[0098] In this embodiment, in step S140, obtaining the label description evaluation of the sample image in the sample image set includes the following steps:

[0099] The degradation category and degradation degree corresponding to the sample image are used as the degradation label of each sample image; for the degraded image, the degradation category and degradation degree of the degraded resource used in the degradation processing are its degradation label, and for the basic image, its degradation category and degradation degree are 0 or other preset default values;

[0100] After the degraded image is obtained in step S130, a degradation label can be added to the degraded image according to the category and degree of the added degraded resource. The degradation label includes the degradation category and the degradation degree. Each sample image has a separate degradation label.

[0101] Select multiple sample image groups, and obtain the sorting positions of multiple first sample images in each sample image group; therefore, for each sample image group, each first sample image included therein not only has a separate degradation label, but also has a sorting position in the sample image group; wherein the sorting position is obtained in the following manner: for sample image groups of the same degradation category, sort the first sample images according to the degree of degradation, and determine the sorting position of each first sample image; for sample image groups of different degradation categories, input the first sample images into a trained image scoring model to obtain image scores, and sort the first sample images according to the image scores, wherein the training process of the image scoring model includes: manually annotating and scoring the first sample images in part of the sample image group, and then regressively training an image scoring model based on a convolutional neural network based on the part of the sample image group and the manually annotated scores;

[0102] Pre-constructing second prompt word templates corresponding to various evaluation task types, the second prompt word templates including a quality rating template and a quality evaluation detail description template under the corresponding evaluation task type, for prompting the description evaluation generation module with the format of the label description evaluation;

[0103] Corresponding to the single image evaluation type (including the single image evaluation type with reference image and the single image evaluation type without reference image), the sample image, the degraded label and the corresponding second prompt word template are input into the description evaluation generation model constructed based on the multimodal large model to obtain the label description evaluation; wherein, for the single image evaluation type with reference image, the reference image is also input into the description evaluation generation model;

[0104] For multiple image evaluation types (including multiple image evaluation types with reference images and multiple image evaluation types without reference images), each sample image group, the degradation label, the sorting position and the corresponding second prompt word template of each first sample image are input into the description evaluation generation model constructed based on the multimodal large model to obtain the label description evaluation; among them, for the single image evaluation type with a reference image, the reference image is also input into the description evaluation generation model.

[0105] In the process of generating label description evaluation by using the description evaluation generation model, two stages are used for generation. The first stage of the description label generation process is the machine generation and manual correction stage. Figure 3 It is a flowchart of the description evaluation generation process of an embodiment of the present application. First, a first number of sample images are selected, degraded labels and sorting positions are obtained, and second prompt word templates corresponding to various evaluation task types are designed; the sample images, the second prompt word templates and the degraded labels are input into the description evaluation generation model, and the first number is a small number, such as dozens or dozens, and the description evaluation generation model generates the label description evaluation of the first stage; after obtaining the label description evaluation of the first stage, the label description evaluation is manually scored and modified, and the accuracy of the label description evaluation is ensured by manual correction. Based on the difference between the label description evaluation manually corrected by the manually adjusted data set and the label description evaluation output by the model, the description evaluation generation model is adjusted and optimized, and the model parameters are partially or completely fine-tuned, so as to obtain a description evaluation generation model that can output professional and accurate labels. The second stage of the description label generation process is the machine batch generation stage, in which a second number of sample images are selected, degraded labels and sorting positions are obtained, and the sample images, the second prompt word templates and the degraded labels are input into the description evaluation generation model that has been fine-tuned, and the description evaluation labels of the description evaluation generation model are obtained. The second number is a larger number, which is the number of sample images required for subsequent training of the image quality assessment model.

[0106] like Figure 4 As shown, the image quality assessment model includes a proxy algorithm module and an evaluation large model. In step S150, the image quality assessment model constructed based on the multimodal large model is trained based on the sample image set and the label description evaluation, including the following steps:

[0107] Preset first prompt word templates corresponding to a plurality of evaluation task types; wherein the first prompt word template is used to prompt the image quality evaluation model with the required evaluation task type; the first prompt word template is set in a form recognizable by the multimodal large model, for example, a string or vector containing a specific indicator and a specific code, and a first prompt word template is set for each evaluation task type;

[0108] Inputting the quality evaluation problem and the sample image obtained from the sample image set into the image quality evaluation model, the proxy algorithm module is configured to determine the corresponding evaluation task type and the first prompt word template based on the sample image and the quality evaluation problem, and inputting the sample image and the first prompt word template into the evaluation large model, and the evaluation large model is configured to output the reasoning description evaluation based on the input sample image and the first prompt word template;

[0109] Among them, the image to be evaluated can be a degraded image or a basic image, that is, the image quality assessment model can evaluate both the degraded image and the basic image. Taking the degraded image as an example, if the sample image to be evaluated in the sample image includes the degraded image and the basic image corresponding to the degraded image, the sample image and the corresponding quality assessment problem are input as sample data into the image quality assessment model to train the model's image quality assessment ability for the image with reference image evaluation type; if the sample image only includes the degraded image, the sample image and the corresponding quality assessment problem are input as sample data into the image quality assessment model to train the model's image quality assessment ability for the image without reference image evaluation type; it should be further explained that when it is confirmed that the image quality assessment model training is completed, it means that after the image quality assessment model is trained with sample images and label description evaluation, it can cope with the image quality assessment requirements of various evaluation task types and has the ability to perform image quality assessment for various evaluation task types.

[0110] The image quality assessment model is optimized based on the reasoning description evaluation and the corresponding label description evaluation; both the reasoning description evaluation and the label description evaluation are for the image to be evaluated. During the optimization, some or all parameters in the image quality assessment model are fine-tuned in multiple batches so that the image quality assessment model can output a more accurate reasoning description evaluation with reference to the label description evaluation.

[0111] Among them, the quality evaluation question is a description of the user's quality evaluation requirements. In the actual application of the image quality evaluation model, a question setting page can be provided to obtain the questions entered by the user in the question setting page; the quality evaluation question is, for example, "score this photo", "compare these two images", "take the first image as a reference, evaluate the second image", etc.; when training the image quality evaluation model, a variety of quality evaluation questions corresponding to each evaluation task type can be preset. Among them, the proxy algorithm module is used to determine the evaluation task type that needs to be performed by the evaluation model according to the sample image and the quality evaluation question, and select the corresponding first prompt word template. For example, the sample image includes two images, and the quality evaluation question is "take the first image as a reference, evaluate the second image", which corresponds to the single image evaluation type with a reference image. The sample image includes multiple images, and the quality evaluation question is "score these photos", which corresponds to the multiple image evaluation type without a reference image. After determining the evaluation task type, the corresponding first prompt word template can be called. When training the image quality assessment model, multiple commonly used quality assessment questions are prepared in advance for each evaluation task type, and quality assessment questions are randomly selected during training to cope with actual scenarios where users may encounter various problems when using the image quality assessment model.

[0112] like Figure 4As shown, the evaluation model includes a visual encoder, a visual focus projection layer and a large language model. The visual encoder is, for example, a CLIP (Contrastive Language-Image Pretraining) visual encoder, such as a pre-trained vision Transformer encoder. The large language model is, for example, a 13B language model. The categories of the visual encoder and the large language model here are only examples and are not intended to limit the scope of protection of this application. When training the image quality evaluation model, after the sample images and the first prompt word templates corresponding to various evaluation task types are input into the evaluation model, the visual encoder extracts visual features from the sample images, and the visual focus projection layer performs differential feature attention calculation on the visual features. According to the attention calculation results, the projected visual features are obtained (the attention calculation results are projected to the image size corresponding to the visual features to obtain the projected visual features), and the projected visual features and the first prompt word template are input into the large language model to obtain the reasoning description evaluation output by the large language model. Different from the general multimodal large model, the visual focus projection layer of the image quality evaluation model adds differential feature attention calculation to expand the evaluation when there are reference images and contrast images (multiple images). The difference feature attention module not only realizes the dimensional mapping before the fusion of visual features and language features, but also allows the image quality assessment model to pay attention to the quality differences between images when the visual features of each image are integrated with language features. As a result, the image quality assessment model can not only obtain accurate image quality assessment results for the single image assessment type without reference images, but also conduct in-depth mining of difference features for multiple images in the reference image assessment type and the multiple image assessment type without reference images, so that the image quality assessment model can meet the accurate evaluation requirements of various evaluation task types. By introducing difference features into the visual features of the input large language model, the large language model can effectively capture the key contrast information of the input image. This integration method improves the image quality assessment model's ability to understand the input image.

[0113] In this embodiment, the visual focus projection layer performs difference feature attention calculation on the visual features, including the following steps:

[0114] For the evaluation type with reference image (including the evaluation type with single reference image and the evaluation type with multiple reference images), the query (Q) is obtained according to the visual feature encoding of the reference image in the sample image, the key (K) is obtained according to the visual feature encoding of the image to be evaluated in the sample image, and the value (V) is obtained according to the difference visual feature encoding between the image to be evaluated and the reference image, and the difference feature attention calculation is performed on the three;

[0115] For the single image evaluation type without reference image, query (Q), key (K), and value (V) are obtained according to the visual feature encoding of the sample image, and differential feature attention calculation is performed;

[0116] For the multiple image evaluation type without reference images, select two adjacent sample images, obtain the query (Q) according to the visual feature encoding of the previous sample image, obtain the key (K) according to the visual feature encoding of the next sample image, and obtain the value (V) according to the difference visual feature encoding of the two adjacent sample images, and perform difference feature attention calculation.

[0117] Here, visual feature encoding refers to visual features multiplied by a certain scientific weight. The difference feature attention calculation formula is as follows:

[0118] Q=HW Q

[0119] K=LW K

[0120] V=DW V

[0121]

[0122] Where H is the visual feature of the reference image when there is a reference image or the visual feature of the previous sample image of the adjacent sample image when there is no reference image, L is the visual feature of the image to be evaluated when there is a reference image or the visual feature of the next sample image of the adjacent sample image when there is no reference image, D is the difference visual feature between the reference image and the image to be evaluated when there is a reference image or the difference visual feature between two adjacent sample images when there is no reference image, W Q , W K , W V is the corresponding science department weight, d k is the dimension of K.

[0123] The multimodal large model used in the image quality assessment model can also be other types of large models, including but not limited to InternVL2 and QwenVL models. The process of training the image quality assessment model constructed based on the multimodal large model includes multiple batches, and each batch selects multiple sample images from the sample image set for training the current batch. In each batch, the sample image and the quality assessment problem are input into the image quality assessment model, and the reasoning description evaluation and the label description evaluation output by the image quality assessment model are compared, and the image quality assessment model is adjusted according to the comparison result. When adjusting the image quality assessment model, all or part of the parameters of the model can be adjusted. For example, the parameters of the visual encoder and the visual focus projection layer can be frozen, and only the parameters in the large language model can be fine-tuned. Furthermore, the present application also uses an autoregressive training algorithm to train the image quality assessment model, so that the model can quickly converge to the optimal effect when facing new training tasks.

[0124] Among them, the following steps are used to train the image quality assessment model based on the multimodal large model:

[0125] Perform multiple batches of image quality assessment model training based on sample image sets and label description evaluations;

[0126] After each batch of training is completed, if the training of the image quality assessment model is still not completed (for example, the training is completed based on the comparison results of the inference description evaluation and the label description evaluation, and the training is considered completed when the error between the two is less than a certain error threshold, or the training is completed based on the number of batches, and the training is considered completed when the training batch reaches a certain batch threshold), then continue with the following steps:

[0127] Select a test sample image from the sample image set, input the image quality evaluation model, obtain a test reasoning description evaluation, and determine the corresponding test label description evaluation; the test sample image can be randomly selected from the sample image set;

[0128] The test sample image, the test reasoning description evaluation and the test label description evaluation are input into the result evaluation model to obtain the description evaluation professional comparison result; the scoring evaluation mechanism of the result evaluation model can be set as needed so that the result evaluation model can compare and judge between the reasoning result and the label description evaluation. The result evaluation model can be the image quality evaluation model or another artificial intelligence model trained based on the machine learning model; if the result evaluation model is the image quality evaluation model, the image quality evaluation model uses a different process from the image quality evaluation to perform the result evaluation; when the result evaluation model is implemented by the multimodal large model of the image quality evaluation model itself, the sample image, the first prompt word template and the label description evaluation are input into the image quality evaluation model together, and the image quality evaluation model is adaptively adjusted; the description evaluation professional comparison result may include the probability that the reasoning description evaluation is more professional and / or the probability that the label description evaluation is more professional;

[0129] When it is determined that the test reasoning description evaluation is more professional according to the description evaluation professional comparison result, the test reasoning description evaluation is used as the label description evaluation of the test sample image in the next batch of training process.

[0130] Therefore, by inputting the sample image and the first prompt word template into the image quality assessment model, and by comparing the inference results and label description evaluation output by the model, the adaptive autoregressive improvement training of the model can be achieved. This autoregressive training algorithm not only enables the image quality assessment model to remember the provided label description evaluation, but also fully stimulates the reasoning ability of the multimodal large model and improves its professionalism and accuracy. Through continuous iteration and self-optimization, the image quality assessment model can show a higher level of professionalism and practicality when dealing with complex problems. After the image quality assessment model has been fully fine-tuned and evaluated, it can be used to evaluate the quality of the image to be evaluated in the new image to be processed.

[0131] The following is a specific example of an image quality evaluation method using this embodiment. 5000 high-quality images collected by a network camera are used as basic images. First, the basic image is enhanced and expanded, and any one or more of the defined degraded resources are randomly selected from the degraded resource pool and added to the basic image to generate a degraded image. In this process, a short label is automatically generated as a degraded label (degraded type and degree of degradation of the degraded image). For the sample image group with the same degradation category, the first sample image is sorted according to the degree of degradation to obtain the sorting position of the first sample image. For sample image groups with different degradation categories, the scores of 2000 sample image group sequences are manually annotated for regression training of convolutional neural networks to generate a scoring sorting model, and the scoring sorting model is used to sort multiple first sample images with different degradation categories in the same sample image group to obtain the sorting position of each first sample image. Then, according to the different types of evaluation tasks, the corresponding sample images, the second prompt word template, and the degradation label are used as the description evaluation generation model to generate professional label description evaluation as the label of the training image quality evaluation model. The number of sample images in the final expanded sample image set reaches 100,000. For the four evaluation task types of single or multiple non-reference / reference images, corresponding sample image subsets are set. The image quality model includes a proxy algorithm module and an evaluation model. The evaluation model includes a visual encoder, a visual focus projection layer, and a large language model. Among them, the visual encoder uses an encoder pre-trained based on a multimodal model, the visual focus projection layer introduces a difference feature attention mechanism, and the large language model uses a 13B large language model. The expanded sample data set is used to fine-tune some parameters of the evaluation model. The proxy algorithm module is designed for different evaluation task types, and the evaluation task type is judged according to the input image and quality evaluation problem to select the corresponding first prompt word template and send it to the fine-tuned image quality evaluation model to generate the image quality evaluation results required by the user.

[0132] like Figure 5 As shown, the embodiment of the present application also provides an image quality assessment method, which is implemented based on the image quality assessment model trained by the above-mentioned image quality assessment model training method, and the method includes the following steps:

[0133] S210: Acquire an image to be processed captured by a capturing device;

[0134] The shooting device may be the device for shooting the reference image, but is not limited thereto. The shooting device may be, for example, a network camera, a camera, a mobile phone, etc.; obtaining the image to be processed shot by the shooting device may be obtaining an image shot in real time from the shooting device, or may be obtaining an image shot by the shooting device and stored in a designated storage device;

[0135] S220: Inputting the image to be processed and the quality assessment problem into the image quality assessment model to obtain the reasoning description evaluation output by the image quality assessment model;

[0136] The quality evaluation question is a description of the quality evaluation type requirement. A question setting page can be provided to obtain the question entered by the user in the question setting page. Examples of quality evaluation questions include "rate this photo", "compare these two images", "evaluate the second image based on the first image", etc.

[0137] By using the image quality assessment model obtained by the above-mentioned image quality assessment model training method to perform quality assessment of the image to be processed, a more accurate image quality assessment result can be obtained, and the image quality assessment model can be evaluated for a variety of evaluation task types. The user can input the input image and quality assessment problem corresponding to the required evaluation task type into the image quality assessment model, thereby obtaining the reasoning description evaluation corresponding to the required evaluation task type, which expands the scope of application of image quality assessment. Users only need to set the quality assessment problem to set their own evaluation task type requirements. The quality assessment problem can be expressed using a simple description of a sentence or a paragraph, which is very convenient for users to use and has strong operability. Users can use different quality assessment problems according to different image quality evaluation requirements.

[0138] In this embodiment, in step S220, after the image to be processed and the quality assessment question are input into the image quality assessment model, the proxy algorithm module determines the assessment task type according to the image to be processed and the quality assessment question, and obtains the first prompt word template corresponding to the assessment task type, the image to be processed and the first prompt word template are input into the large evaluation model, the image to be processed is processed by the visual encoder and the visual focus projection layer, and then the output of the visual focus projection layer and the first prompt word template are input into the large language model, and the large language model outputs the reasoning description evaluation of the image to be evaluated in the image to be processed. The reasoning description evaluation includes the quality score and / or quality assessment detailed description of the image to be evaluated in the image to be processed, and combined with the image quality score and the quality assessment detailed description, it can provide users with professional and in-depth image quality analysis.

[0139] In this embodiment, the proxy algorithm module uses the following steps to determine the evaluation task type according to the image to be processed and the quality evaluation problem, and obtain the first prompt word template corresponding to the evaluation task type:

[0140] The selected evaluation task type is determined based on the quality evaluation problem; for example, various quality evaluation problems that may appear corresponding to each evaluation task type can be preset, and an evaluation task type can be corresponding to the quality evaluation problem;

[0141] Determine whether the number of images to be processed meets the image number requirement of the selected evaluation task type; specifically, the single image evaluation type with reference image corresponds to two images to be processed, and one of them is specified as the reference image and one is the image to be evaluated in the quality evaluation problem; the multiple image evaluation type with reference image corresponds to more than three images to be processed, and one of them is specified as the reference image and the other two or more images are to be evaluated; the single image evaluation type without reference image corresponds to a single image to be processed, and the multiple image evaluation type without reference image corresponds to more than two images to be processed;

[0142] If yes, then select the corresponding first prompt word template according to the evaluation task type;

[0143] Otherwise, the user is prompted to re-enter the image to be processed and / or the quality evaluation question; for example, if the quality evaluation question entered by the user is "compare these two images", but only one image to be processed is entered, the user needs to be prompted to modify his or her quality evaluation question or upload an additional image to be processed.

[0144] In another embodiment, if the number of images to be processed cannot meet the image quantity requirement of the selected evaluation task type, the evaluation task type can also be determined based on the number of images to be processed and the determined evaluation task type can be executed, and then a prompt that the number of images does not match the question is returned when the inference description label is returned. For example, if the quality evaluation question input by the user is "Compare these two images", but only one image to be processed is input, the evaluation task type is determined to be a single image evaluation type without a reference image, and the quality evaluation of the single image to be processed is performed. When the quality evaluation question input by the user corresponds to an evaluation type with a reference image but no image is specified as the reference image, the first image is used as the reference image by default.

[0145] like Figure 6 As shown, the embodiment of the present application also provides an image quality evaluation system for implementing the above-mentioned image quality evaluation method, and the system includes:

[0146] The image acquisition module M100 is used to acquire the image to be processed taken by the shooting device;

[0147] The image evaluation module M200 is used to input the image to be processed and the quality evaluation problem into the image quality evaluation model, and obtain the reasoning description evaluation output by the image quality evaluation model.

[0148] The functions of each module in the image quality evaluation system can be implemented by adopting the specific implementation methods of the corresponding steps of the above-mentioned image quality evaluation method, and the technical effects of the above-mentioned image quality evaluation method can be obtained, which will not be traced here.

[0149] An embodiment of the present application also provides an image quality assessment device, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to perform the steps of the image quality assessment method by executing the executable instructions.

[0150] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0151] Refer to the following Figure 7 The electronic device 600 according to this embodiment of the present application is described. Figure 7 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0152] like Figure 7 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0153] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present application described in the above image quality evaluation method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 5 Follow the steps shown in .

[0154] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0155] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0156] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0157] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0158] By adopting the image quality assessment device provided by the present application, the processor executes the image quality assessment method when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned image quality assessment method.

[0159] The embodiment of the present application also provides a computer-readable storage medium for storing a program, which implements the steps of the image quality evaluation method when executed by a processor. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present application described in the above-mentioned image quality evaluation method section of this specification.

[0160] refer to Figure 8 As shown, a program product 800 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0161] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0162] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0163] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0164] By adopting the computer-readable storage medium provided in the present application, the program stored therein implements the steps of the image quality assessment method when executed, thereby obtaining the beneficial effects of the above-mentioned image quality assessment model training method.

[0165] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.

Claims

1. A method for training an image quality assessment model, characterized in that: The steps include: Collecting a basic image and obtaining a plurality of reference images taken by a photographing device; Acquire a plurality of degraded resources based on the reference image, and add the degraded resources into a degraded resource pool, wherein each degraded resource corresponds to a degradation category and a degradation degree; For each of the basic images, one or more degraded resources are obtained from the degraded resource pool, the basic image is processed to obtain a degraded image, and the degraded image and the basic image are added to a sample image set; Obtaining label description evaluations corresponding to sample images in the sample image set and multiple evaluation task types; An image quality assessment model constructed based on a multimodal large model is trained based on the sample image set and the label description evaluation.

2. The image quality assessment model training method according to claim 1, characterized in that: Acquiring a plurality of degraded resources based on the reference image comprises the following steps: Based on the plurality of reference images, using the degraded resource extraction algorithms corresponding to the various degraded categories, extracting reference degraded resources of various degraded categories; Based on the benchmark degraded resources of various degradation categories and a variety of degradation coefficients, degraded resources with different degradation degrees under the degradation category are obtained respectively.

3. The image quality assessment model training method according to claim 1, characterized in that: The image quality assessment model includes a proxy algorithm module and an evaluation large model; the image quality assessment model constructed based on the multimodal large model is trained based on the sample image set and the label description evaluation, including the following steps: Presetting a first prompt word template corresponding to a plurality of evaluation task types; wherein the first prompt word template is used to prompt the image quality evaluation model with a required evaluation task type; Inputting a quality assessment problem and a sample image obtained from the sample image set into the image quality assessment model, the proxy algorithm module is configured to determine a corresponding assessment task type and a first prompt word template based on the sample image and the quality assessment problem, and inputting the sample image and the first prompt word template into the evaluation large model, the evaluation large model is configured to output an inference description evaluation based on the input sample image and the first prompt word template; The image quality assessment model is optimized based on the inference description evaluation and the corresponding label description evaluation.

4. The image quality assessment model training method according to claim 3, characterized in that: The large evaluation model includes a visual encoder, a visual focus projection layer and a large language model; the evaluation task type includes a reference image evaluation task type and a non-reference image task type, and / or the evaluation task type includes a single image evaluation task type and a multiple image evaluation task type; After the sample images and the first prompt word templates corresponding to various evaluation task types are input into the large evaluation model, the visual encoder extracts visual features from the sample images, the visual focus projection layer performs difference feature attention calculation on the visual features, and the projected visual features are obtained according to the attention calculation results. The projected visual features and the first prompt word template are input into the large language model to obtain the reasoning description evaluation output by the large language model.

5. The image quality assessment model training method according to claim 4, characterized in that: The evaluation task types include evaluation task types with reference images, single image evaluation types without reference images, and multiple image evaluation types without reference images; The visual focus projection layer performs difference feature attention calculation on the visual features, including the following steps: For the evaluation type with reference image, the query is obtained according to the visual feature encoding of the reference image in the sample image, the key is obtained according to the visual feature encoding of the image to be evaluated in the sample image, and the value is obtained according to the difference visual feature encoding between the image to be evaluated and the reference image, and the difference feature attention calculation is performed; For the single image evaluation type without a reference image, the query, key and value are obtained according to the visual feature encoding of the sample image, and the difference feature attention calculation is performed; For the multiple image evaluation type without reference images, select two adjacent sample images, obtain the query according to the visual feature encoding of the previous sample image, obtain the key according to the visual feature encoding of the next sample image, and obtain the value according to the difference visual feature encoding of the two adjacent sample images, and perform the difference feature attention calculation.

6. The image quality assessment model training method according to claim 3, characterized in that: The evaluation task types include single image evaluation type and multiple image evaluation type; Obtaining label description evaluations of sample images in the sample image set includes the following steps: Using the degradation category and degradation degree corresponding to the sample image as a degradation label for each sample image; Selecting multiple sample image groups, obtaining the sorting positions of multiple first sample images in each sample image group, wherein, for sample image groups of the same degradation category, sorting the first sample images according to the degree of degradation, and determining the sorting position of each first sample image; for sample image groups of different degradation categories, inputting the first sample images into a trained image scoring model to obtain image scores, and sorting the first sample images according to the image scores; Pre-constructing second prompt word templates corresponding to various evaluation task types, wherein the second prompt word templates are used to indicate the format of the label description evaluation to the description evaluation generation model; Corresponding to the single image evaluation type, the sample image, the degraded label and the corresponding second prompt word template are input into the description evaluation generation model constructed based on the multimodal large model to obtain the label description evaluation; For multiple image evaluation types, each sample image group, the degradation label, sorting position and corresponding second prompt word template of each first sample image are input into the description evaluation generation model constructed based on the multimodal large model to obtain the label description evaluation.

7. The image quality assessment model training method according to claim 1, characterized in that: The following steps are used to train the image quality assessment model based on the multimodal large model: Performing multiple batches of image quality assessment model training based on the sample image set and the label description evaluation; After each batch of training is completed, if the training of the image quality assessment model is still not completed, continue with the following steps: Selecting a test sample image from the sample image set, inputting the image quality evaluation model, obtaining a test reasoning description evaluation, and determining a corresponding test label description evaluation; Inputting the test sample image, the test reasoning description evaluation and the test label description evaluation into a result evaluation model to obtain a description evaluation professional comparison result; When it is determined that the test reasoning description evaluation is more professional according to the description evaluation professional comparison result, the test reasoning description evaluation is used as the label description evaluation of the test sample image in the next batch of training.

8. A method for evaluating image quality, characterized in that: The image quality assessment model trained by the image quality assessment model training method according to any one of claims 1 to 7 is implemented, and the method comprises the following steps: Acquire the image to be processed captured by the capturing device; The image to be processed and the quality assessment problem are input into the image quality assessment model to obtain the reasoning description evaluation output by the image quality assessment model.

9. An image quality evaluation system, characterized in that: For implementing the image quality evaluation method according to claim 8, the system comprises: An image acquisition module, used to acquire the image to be processed taken by the shooting device; The image evaluation module is used to input the image to be processed and the quality evaluation problem into the image quality evaluation model, and obtain the reasoning description evaluation output by the image quality evaluation model.

10. An image quality evaluation device, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to perform the steps of the image quality assessment method described in claim 8 by executing the executable instructions.

11. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the image quality evaluation method according to claim 8 are implemented.

Images