Image multi-factor evaluation method, device and equipment based on aesthetic relativity
Through the multi-factor evaluation method based on aesthetic relativity, image segmentation and aesthetic factor prediction processing are used, combined with the comparison model and the multi-head self-attention mechanism, the problem of poor quality after image enhancement is solved, and the accuracy of image aesthetic evaluation is achieved and simplify user operations.
Patent Information
- Application Number
- CN202410604832.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-15
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-05-15
AI Technical Summary
The poor image quality after image enhancement in the prior art leads to multiple adjustments from users, which increases the difficulty of use, and lacks a reliable image aesthetic evaluation method to evaluate the image enhancement effect.
Using the multi-factor evaluation method based on aesthetic relativity, the image segmentation and aesthetic factor prediction processing are performed by obtaining the image to be evaluated and the target reference image, calculating the comparison results, combining the comparison model and the multi-head self-attention mechanism, the evaluation results are generated. Improve evaluation accuracy.
Multi-dimensional evaluation of image aesthetic factors is realized, the accuracy and accuracy of evaluation results are improved, and the image enhancement process is simplified.
Smart Images

Figure CN118587566B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, device and equipment for multi-factor image evaluation based on aesthetic relativity. Background Art
[0002] In recent years, with the rapid development of AI-generated content, image enhancement has become more and more widely used. However, the quality of images after image enhancement is currently poor, requiring users to make multiple adjustments, which increases the difficulty of use. Therefore, reliable image aesthetics evaluation can help users better enhance images. How to comprehensively and accurately evaluate the aesthetics of images after image enhancement is an urgent problem to be solved. Summary of the invention
[0003] The present application provides a method, device and equipment for multi-factor image evaluation based on aesthetic relativity, so as to improve the effect of image aesthetic evaluation.
[0004] In a first aspect, the present application provides an image multi-factor evaluation method based on aesthetic relativity, comprising:
[0005] Acquire an image to be evaluated and a target reference image, wherein the target reference image and the image to be evaluated have the same subject type;
[0006] Performing image segmentation processing on the image to be evaluated and the target reference image respectively to obtain a sub-image to be evaluated in the image to be evaluated and a sub-target reference image in the target reference image;
[0007] Performing aesthetic factor prediction processing on the sub-image to be evaluated and the sub-target reference image respectively, to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image, wherein the first prediction result and the second prediction result represent the prediction result of at least one aesthetic factor of artifacts, composition, color, noise, exposure, clarity and white balance in the image;
[0008] Obtaining a comparison result of the image to be evaluated and the target reference image according to the sub-image to be evaluated, the sub-target reference image, the first prediction result, and the second prediction result, wherein the comparison result represents a degree of difference between an aesthetic factor of the image to be evaluated and an aesthetic factor of the target reference image;
[0009] An evaluation result of the image to be evaluated is determined based on the comparison result and the target reference score label of the target reference image.
[0010] In the present application, the sub-image to be evaluated and the sub-target reference image are respectively subjected to aesthetic factor prediction processing to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image, including:
[0011] Determine the prediction model of aesthetic factors;
[0012] The sub-image to be evaluated and the sub-target reference image are respectively input into the aesthetic factor prediction model for aesthetic factor prediction processing to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image.
[0013] In the present application, before determining the aesthetic factor prediction model, the method further includes:
[0014] Acquire a first training image;
[0015] Editing the aesthetic factors of the first training image to obtain an edited image and edited data of the edited image;
[0016] Inputting the edited image into the first to-be-trained model for aesthetic factor prediction processing to obtain predicted edited data;
[0017] The first model to be trained is adjusted according to the predicted editing data and the editing data of the edited image to obtain an aesthetic factor prediction model.
[0018] In the present application, a comparison result of the image to be evaluated and the target reference image is obtained according to the sub-image to be evaluated, the sub-target reference image, the first prediction result, and the second prediction result, including:
[0019] Inputting the sub-image to be evaluated and the sub-target reference image into the position marking model for image position marking processing, thereby obtaining a first image position marking result of the sub-image to be evaluated and a second image position marking result of the sub-target reference image;
[0020] Inputting the sub-image to be evaluated and the sub-target reference image into the image embedding model for image embedding processing, thereby obtaining a first embedding result of the sub-image to be evaluated and a second embedding result of the sub-target reference image;
[0021] Integrate the first image position marking result, the second image position marking result, the first embedding result, the second embedding result, the first prediction result, and the second prediction result to obtain an integration result;
[0022] The integrated results are input into the comparison model for comparison processing to obtain the comparison results between the image to be evaluated and the target reference image.
[0023] In this application, the integration result is input into the comparison model for comparison processing to obtain the comparison result of the image to be evaluated and the target reference image, including:
[0024] Inputting the integration result into the first-layer normalization unit in the comparison model to obtain the first normalization result;
[0025] According to the integration result, the first normalization result, the multi-head self-attention unit, the second-layer normalization unit, and the multi-layer perceptron unit, an updated integration result is obtained, wherein the updated integration result is determined according to the second-layer normalization unit, the multi-layer perceptron unit, and the initial updated integration result, and the initial updated integration result is determined according to the integration result, the first normalization result, and the multi-head self-attention unit;
[0026] Input the updated integration result into the third layer normalization unit to obtain the third normalization result;
[0027] The third normalization result is input into the Gaussian error linear unit to obtain a Gaussian error linear output result;
[0028] The Gaussian error linear output result is input into the fourth-layer normalization unit to obtain a comparison result between the image to be evaluated and the target reference image, wherein the learning parameters of the first-layer normalization unit, the second-layer normalization unit, the third-layer normalization unit, and the fourth-layer normalization unit are different.
[0029] In this application, according to the integration result, the first normalization result, the multi-head self-attention unit, the second layer normalization unit, and the multi-layer perceptron unit, an updated integration result is obtained, including:
[0030] Input the first normalized result into the multi-head self-attention unit to obtain the self-attention output result;
[0031] Integrate the self-attention output result and the integration result to obtain the initial updated integration result;
[0032] Input the initial update integration result into the second layer normalization unit to obtain a second normalization result;
[0033] Inputting the second normalized result into the multi-layer perceptron unit to obtain the multi-layer perceptron output result;
[0034] The multi-layer perceptron output result and the initial updated integration result are integrated to obtain the updated integration result.
[0035] In the present application, before inputting the integration result into the comparison model for comparison processing to obtain the comparison result of the image to be evaluated and the target reference image, the method further includes:
[0036] Acquire a model training sample, the model training sample comprising a sub-second training image and a sub-training reference image, wherein the sub-second training image is a sub-image in the second training image, the sub-training reference image is a sub-image in the training reference image, and the subject type of the second training image and the subject type of the training reference image meet preset type requirements;
[0037] Performing aesthetic factor prediction processing on the sub-image in the model training sample to obtain a training prediction result of the model training sample, wherein the training prediction result includes a first training prediction result of the sub-second training image and a second training prediction result of the sub-training reference image;
[0038] Obtaining a training comparison result output by the second to-be-trained model according to the model training samples and the training prediction results;
[0039] According to the training comparison result and the training control result, the second model to be trained is adjusted to obtain a comparison model, wherein the training control result is determined according to the training score label of the second training image in the model training sample and the training reference score label of the training reference image.
[0040] In this application, when there are more than two model training samples, and the training reference scoring labels of the training reference images corresponding to the sub-training reference images in different model training samples are different,
[0041] After performing the prediction processing of the aesthetic factors on the sub-images in the model training samples to obtain the training prediction results of the model training samples, the method further includes:
[0042] Determining a training order of model training samples according to the training reference score labels of the training reference images;
[0043] Determine, according to the training sequence of the model training samples, a target model training sample in the model training samples and a target training prediction result of the target model training sample, wherein the target model training sample is a sample that has not been trained and whose training sequence meets a preset sequence requirement;
[0044] Obtaining a target training comparison result output by the second to-be-trained model according to the target model training sample and the target training prediction result;
[0045] According to the target training comparison result and the target training control result, the second model to be trained is adjusted to obtain an initial comparison model;
[0046] Repeat the steps of determining the target model training samples in the model training samples and the target training prediction results of the target model training samples according to the training order of the model training samples until all the model training samples are samples to be used for training, thereby obtaining a comparison model.
[0047] In a second aspect, the present application provides an image multi-factor evaluation device based on aesthetic relativity, comprising:
[0048] An acquisition module, used for acquiring an image to be evaluated and a target reference image, wherein the target reference image and the image to be evaluated have the same subject type;
[0049] A segmentation module is used to perform image segmentation processing on the image to be evaluated and the target reference image respectively to obtain a sub-image to be evaluated in the image to be evaluated and a sub-target reference image in the target reference image;
[0050] A prediction module is used to perform aesthetic factor prediction processing on the sub-image to be evaluated and the sub-target reference image respectively, to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image, wherein the first prediction result and the second prediction result represent the prediction result of at least one aesthetic factor of artifacts, composition, color, noise, exposure, clarity and white balance in the image;
[0051] a comparison module, configured to obtain a comparison result of the image to be evaluated and the target reference image according to the sub-image to be evaluated, the sub-target reference image, the first prediction result, and the second prediction result, wherein the comparison result represents a degree of difference between the aesthetic factors of the image to be evaluated and the aesthetic factors of the target reference image;
[0052] The determination module is used to determine the evaluation result of the image to be evaluated according to the comparison result and the target reference score label of the target reference image.
[0053] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;
[0054] Memory stores computer-executable instructions;
[0055] The processor executes the computer-executable instructions stored in the memory to implement the method of the present application.
[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-readable storage medium is stored computer-executable instructions, which are used to implement the method provided in the present application when executed by a processor.
[0057] The image multi-factor evaluation method, device and equipment based on aesthetic relativity provided by the present application can better compare the target reference image with the image to be evaluated by determining the target reference image of the same subject type, and can achieve the purpose of comparing multiple target reference images with the image to be evaluated by comparing with the target reference image, so that the evaluation results of the image to be evaluated can be determined in multiple dimensions, and the accuracy of the evaluation results can be improved. By constructing aesthetic factor categories and performing aesthetic factor prediction processing on the target reference image and the image to be evaluated, the method of obtaining the respective aesthetic factor prediction results can quantify the aesthetic factors, so that the computer can identify the degree of difference between the image to be evaluated and the target reference image according to the aesthetic factor prediction results, thereby obtaining an evaluation result with better evaluation effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0059] Figure 1 A schematic diagram of a scenario for multi-factor image evaluation based on the theory of aesthetic relativity provided in an embodiment of the present application;
[0060] Figure 2 A schematic diagram of a flow chart of a multi-factor image evaluation method based on the theory of aesthetic relativity provided in an embodiment of the present application;
[0061] Figure 3 A schematic diagram of the structure of an image multi-factor evaluation device based on the theory of aesthetic relativity provided in an embodiment of the present application;
[0062] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0063] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0065] For ease of understanding, the professional terms and concepts involved in the embodiments of the present application are first introduced:
[0066] Aesthetic relativity: Image aesthetics can be quantified by consistently and unambiguously determining the relative preference between two comparable images at a given time.
[0067] Artifact: refers to the appearance of various forms of images on the image even though the originally photographed object does not exist. After image processing, especially in the synthesized image, it appears as unnatural traces, areas, flaws, etc. that can be seen as image processing. In some embodiments, artifacts can also be called pseudo textures. Image artifacts can be modified by operations such as distortion or vignette.
[0068] Composition: refers to the structural configuration of the artistic image in an image. The composition of an image can be modified by rotating, flipping horizontally, flipping vertically, or cropping the image.
[0069] Color: refers to the color distribution of an image. The color of an image can be modified by adjusting the image's color saturation, color brightness, or color harmony.
[0070] Noise: refers to the unwanted random interference or distortion in an image, usually manifested as random graininess or noise on the pixels. The noise of an image can be modified by adjusting the Gaussian noise, compression noise or impulse noise of the image.
[0071] Exposure: refers to the amount of light received by each pixel in the image. Proper exposure ensures that the image has appropriate brightness and contrast, making the details in the image clearly visible. Overexposure will cause the image to be too bright and lose details and color information; while underexposure will cause the image to be too dark and the details are not clear. Exposure can be modified by adjusting the brightness, contrast or shadows of the image.
[0072] Sharpness: refers to the clarity or resolution of objects or details in an image. A high-definition image means that the objects and details in the image can be clearly displayed without blur or distortion. The sharpness of an image can be modified by performing operations such as motion blur, JPEG compression, pixelation, or Gaussian blur on the image.
[0073] White balance: refers to adjusting the color temperature of white or gray areas in an image to make it look realistic and neutral. The purpose of white balance is to eliminate the effect of the color temperature of the light source on the color of the image and ensure that white objects in the image appear white under different light sources. White balance can be adjusted by adjusting the color temperature, tint, or color filter of the image.
[0074] In recent years, with the rapid development of AI-generated content, the use of image enhancement has become more and more widespread, but the image quality after image enhancement is currently poor, requiring users to make multiple adjustments, which increases the difficulty of use. Therefore, reliable image aesthetics evaluation can help users better perform image enhancement.
[0075] In some embodiments, the image aesthetics assessment may be performed on the image through an image aesthetics assessment model. The image aesthetics assessment model may be an artificial intelligence (AI) model. Before performing image aesthetics assessment on the image, the image aesthetics assessment model needs to be trained, and the image aesthetics assessment is performed on the image using the trained image aesthetics assessment model. The training process of the image aesthetics assessment model includes:
[0076] A training image and a training score label of the training image are obtained, wherein the training score label is usually obtained through manual annotation, that is, the score of the aesthetic factor of the image is manually annotated for the sample image.
[0077] Determine a training reference image according to the training image, wherein the training reference image is consistent with the subject type of the training image, and the training reference image includes a reference score label;
[0078] The model to be trained is trained by using the training images, the training score labels, the training reference images, and the reference score labels to determine the learning parameters of the training score labels in the model to be trained;
[0079] According to the learning parameters of multiple training score labels, a trained image aesthetic assessment model is obtained.
[0080] In this embodiment, by determining the learning parameters corresponding to each score, the actual parameters of the image to be evaluated after preprocessing can be compared with the learning parameters during evaluation, and the score of the image to be evaluated can be determined by determining the similarity of the actual parameters to each learning parameter.
[0081] However, in this embodiment, the image to be evaluated is evaluated by using fixed learning parameters, so that the obtained score result is not accurate enough.
[0082] In response to the above-mentioned problems, the inventors discovered in their research that a target reference image corresponding to the image to be evaluated can be determined based on the image to be evaluated, and the prediction results of the aesthetic factors of the image to be evaluated and the target reference image can be determined. The aesthetic factors of the image are represented by the image editing mode and the editing degree, and the prediction results of the image to be evaluated and the evaluation results of the target reference image are obtained. The two images are compared through the two evaluation results to obtain the degree of difference between the aesthetic factors of the image to be evaluated and the aesthetic factors of the target reference image. Then, the evaluation results of the image to be evaluated are determined based on the degree of difference and the target reference scoring label of the target reference image. By using the above method, multiple target reference images are compared with the image to be evaluated, so that the evaluation results of the image to be evaluated can be determined in multiple dimensions, thereby improving the accuracy of the evaluation results.
[0083] The following introduces the application scenarios of the multi-factor image evaluation method based on aesthetic relativity provided in the embodiments of the present application.
[0084] Figure 1 A schematic diagram of a scenario for multi-factor image evaluation based on aesthetic relativity provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the scene includes an image to be evaluated, a reference image database, an aesthetic factor prediction model, a position marking model, an image embedding model, and a comparison model, wherein the reference image database includes reference images with annotated scores classified by subject type. The target reference image is determined from the reference image database through the image to be evaluated, and then the image to be evaluated and the target reference image are respectively input into the aesthetic factor prediction model, the position marking model, and the image embedding model to obtain the first prediction result and the second prediction result output by the aesthetic factor prediction model, the first image position marking result and the second image position marking result output by the position marking model, and the first embedding result and the second embedding result output by the image embedding model. Then, the first prediction result, the second prediction result, the first image position marking result, the second image position marking result, the first embedding result and the second embedding result are integrated and input into the comparison model to obtain the comparison result output by the comparison model. Finally, according to the comparison result and the score of the target reference image, the evaluation result of the image to be evaluated is obtained.
[0085] Figure 2 A flowchart of a method for multi-factor image evaluation based on aesthetic relativity provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:
[0086] S201 , obtaining an image to be evaluated and a target reference image, wherein the target reference image and the image to be evaluated have the same subject type.
[0087] The image to be evaluated may refer to an image that needs to be evaluated for aesthetic factors.
[0088] The target reference image may refer to a reference image determined from a reference image database. The target reference image may include more than one reference image. The reference image database may include subject types, reference images within subject types, and reference scoring labels for reference images. By presetting the reference image database, reference information is provided for the image to be evaluated.
[0089] The subject type may refer to a category for classifying a reference image. For example, the image may be classified according to the subject, such as animals, plants, buildings, mountain views, sea views, etc. The same subject may also be subdivided. For example, animals may be classified according to their species.
[0090] The method for acquiring a target reference image may include selecting a target reference image of the same subject type from a reference image database according to the subject type of the image to be evaluated.
[0091] In some embodiments, in order to obtain better evaluation results, the same image to be evaluated may be evaluated multiple times, so that the obtained evaluation results are more accurate. The following is explained according to the number of target reference images for each evaluation:
[0092] When there is only one target reference image for each evaluation, the target reference image can be determined by using a binary method according to a preset number of evaluations. Specifically, according to the scores of reference images in the same subject type, a first scoring interval and an intermediate score in the first scoring interval are determined, and a reference image with an intermediate score is selected as the target reference image for the first evaluation. According to the image to be evaluated and the target reference image for the first evaluation, the first evaluation result is obtained. Then, according to the first evaluation result, the second scoring interval in which the first evaluation result is located and the intermediate score of the second scoring interval are determined, wherein the second scoring interval is a scoring interval in the first scoring interval divided according to the intermediate score. The reference image corresponding to the intermediate score of the second scoring interval is determined as the target reference image for the second evaluation, and the second evaluation result is obtained according to the image to be evaluated and the target reference image for the second evaluation. In the above manner, the preset number of evaluations is completed, and the last evaluation result is used as the final evaluation result. Evaluation by this method can make the evaluation result more and more accurate, and improve the effect of model evaluation.
[0093] When there are two target reference images for each evaluation, the score difference of the target reference images can be gradually reduced by a step-by-step method according to the preset number of evaluations. Specifically, according to the scores of the reference images in the same subject type, a first scoring interval is determined, and a reference image is selected from both ends of the first scoring interval as the target reference image for the first evaluation. According to the image to be evaluated and the target reference image of the first evaluation, the first evaluation result is obtained, and then according to the first evaluation result, a second scoring interval containing the first evaluation result is determined, wherein the second scoring interval is smaller than the first evaluation interval, and preferably, the first evaluation result can be the middle score of the second scoring interval. A reference image is selected from both ends of the second scoring interval as the target reference image for the second evaluation, and according to the image to be evaluated and the target reference image of the second evaluation, the second evaluation result is obtained. In the above manner, the preset number of evaluations is completed, and the last evaluation result is used as the final evaluation result. By using the method, the proximity between the image to be evaluated and the two target reference images is located through the two target reference images, so that the obtained evaluation result is more accurate and the effect of model evaluation is improved.
[0094] S202 , performing image segmentation processing on the image to be evaluated and the target reference image respectively to obtain a sub-image to be evaluated in the image to be evaluated and a sub-target reference image in the target reference image.
[0095] The image segmentation process may refer to the process of segmenting an image into multiple sub-images according to an image segmentation strategy. The image segmentation process facilitates the electronic device to recognize the image and improves the effect of the evaluation result.
[0096] S203, performing aesthetic factor prediction processing on the sub-image to be evaluated and the sub-target reference image respectively, to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image, wherein the first prediction result and the second prediction result represent the prediction result of at least one aesthetic factor of artifacts, composition, color, noise, exposure, clarity and white balance in the image.
[0097] The aesthetic factor prediction process may refer to the process of predicting the aesthetic factors in the image.
[0098] Aesthetic factors can refer to features that describe the aesthetics of an image, such as artifacts, composition, color, noise, exposure, sharpness, or white balance.
[0099] The first prediction result and the second prediction result may refer to specific data information of the predicted aesthetic factors. For example, aesthetic factor prediction processing is performed on image A, and a prediction result is obtained that the brightness of image A is adjusted by 0.2 during exposure.
[0100] In the embodiment of the present application, the sub-image to be evaluated and the sub-target reference image are respectively subjected to aesthetic factor prediction processing to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image, including:
[0101] Determine the prediction model of aesthetic factors;
[0102] The sub-image to be evaluated and the sub-target reference image are respectively input into the aesthetic factor prediction model for aesthetic factor prediction processing to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image.
[0103] The aesthetic factor prediction model may refer to a pre-trained mathematical model for predicting aesthetic factors of an image, and the aesthetic factor prediction model may be a neural network model.
[0104] In this embodiment of the present application, before determining the aesthetic factor prediction model, the method further includes:
[0105] Acquire a first training image;
[0106] Editing the aesthetic factors of the first training image to obtain an edited image and edited data of the edited image;
[0107] Inputting the edited image into the first to-be-trained model for aesthetic factor prediction processing to obtain predicted edited data;
[0108] The first model to be trained is adjusted according to the predicted editing data and the editing data of the edited image to obtain an aesthetic factor prediction model.
[0109] The first training image may refer to an original image that has not been edited, so that the electronic device can edit the first training image.
[0110] The editing process may refer to adjusting the aesthetic factors of the first training image. Specifically, it may refer to randomly modifying the first training image according to a preset aesthetic factor modification strategy. The aesthetic factor modification strategy is shown in the following table:
[0111]
[0112]
[0113] The edited image may refer to the first training image after being edited.
[0114] The editing data may refer to editing parameters used by the electronic device when editing the first training image. For example, the electronic device adjusts the brightness of image A by 0.2.
[0115] The first model to be trained may refer to a neural network model.
[0116] The predicted editing data may refer to the predicted data output by the first model to be trained according to the edited image.
[0117] Adjusting the first model to be trained may refer to fitting the predicted editing data and the editing data of the edited image through a loss function to adjust the learning parameters in the first model to be trained so that the predicted editing data output by the first model to be trained meets the fitting requirements, thereby obtaining an aesthetic factor prediction model.
[0118] S204, obtaining a comparison result of the image to be evaluated and the target reference image according to the sub-image to be evaluated, the sub-target reference image, the first prediction result, and the second prediction result, wherein the comparison result represents a degree of difference between the aesthetic factors of the image to be evaluated and the aesthetic factors of the target reference image.
[0119] Among them, the comparison result may refer to the data obtained based on the score of the image to be evaluated and the score of the target reference image. Specifically, when the target reference image for a single evaluation is one, the comparison result may be the ratio of the score of the image to be evaluated to the score of the target reference image; when the target reference images for a single evaluation are two, the comparison result may refer to the degree of closeness of the score of the image to be evaluated to the scores of the two target reference images. For example, the comparison result is 0.44, and the target reference images include A and B. The score of A is 3 points and the score of B is 7 points. Then the comparison result indicates that the degree of closeness to A with a lower distance score is 0.44, and the degree of closeness to B with a higher distance score is 0.56.
[0120] The method for determining the comparison result may include inputting each sub-image to be evaluated and the first prediction result of the image to be evaluated, and each sub-target reference image and the second prediction result of the target reference image into a pre-trained comparison model, and determining the comparison result of the image to be evaluated and the target reference image through the comparison model.
[0121] In the embodiment of the present application, a comparison result of the image to be evaluated and the target reference image is obtained according to the sub-image to be evaluated, the sub-target reference image, the first prediction result, and the second prediction result, including:
[0122] Inputting the sub-image to be evaluated and the sub-target reference image into the position marking model for image position marking processing, thereby obtaining a first image position marking result of the sub-image to be evaluated and a second image position marking result of the sub-target reference image;
[0123] Inputting the sub-image to be evaluated and the sub-target reference image into the image embedding model for image embedding processing, thereby obtaining a first embedding result of the sub-image to be evaluated and a second embedding result of the sub-target reference image;
[0124] Integrate the first image position marking result, the second image position marking result, the first embedding result, the second embedding result, the first prediction result, and the second prediction result to obtain an integration result;
[0125] The integrated results are input into the comparison model for comparison processing to obtain the comparison results between the image to be evaluated and the target reference image.
[0126] Among them, the position marking model can refer to the position marking of each sub-image of the image, which is used for the electronic device to identify the sequence of each sub-image. In an embodiment of the present application, the position marking model can mark the position of each sub-image to be evaluated and each sub-target reference image to obtain a first image position marking result of the sub-image to be evaluated and a second image position marking result of the sub-target reference image.
[0127] An image embedding model may refer to a model that converts image data into a continuous, low-dimensional vector representation. In an embodiment of the present application, the image embedding model may convert the content of each sub-image to be evaluated and each sub-target reference image into an embedded feature to obtain a first embedding result of the sub-image to be evaluated and a second embedding result of the sub-target reference image.
[0128] Integration processing can refer to combining feature information from different levels or sources together to improve the model's ability to understand and represent input data, thereby improving the model's performance and generalization capabilities. Integration processing can be achieved through different methods, including concatenation, addition, multiplication, attention mechanism, etc.
[0129] 1. Concatenation: Connect features from different levels or sources according to certain rules to form a longer feature vector. This method is often used for multi-scale feature fusion or multi-modal information fusion.
[0130] 2. Addition: Adding different features together directly can make it easier for the model to learn the correlation between features, which helps to improve the model's representation ability.
[0131] 3. Multiplication: Multiplying different features element by element can introduce interactive information and help learn the correlation between features.
[0132] 4. Attention Mechanism: By learning attention weights, the importance of different features is dynamically adjusted, so that the model can pay more attention to features that are useful for the current task.
[0133] The integration result may refer to a result of integrating the first image position labeling result, the second image position labeling result, the first embedding result, the second embedding result, the first prediction result, and the second prediction result.
[0134] In the embodiment of the present application, the integration result is input into the comparison model for comparison processing to obtain the comparison result of the image to be evaluated and the target reference image, including:
[0135] Inputting the integration result into the first-layer normalization unit in the comparison model to obtain the first normalization result;
[0136] According to the integration result, the first normalization result, the multi-head self-attention unit, the second-layer normalization unit, and the multi-layer perceptron unit, an updated integration result is obtained, wherein the updated integration result is determined according to the second-layer normalization unit, the multi-layer perceptron unit, and the initial updated integration result, and the initial updated integration result is determined according to the integration result, the first normalization result, and the multi-head self-attention unit;
[0137] Input the updated integration result into the third layer normalization unit to obtain the third normalization result;
[0138] The third normalization result is input into the Gaussian error linear unit to obtain a Gaussian error linear output result;
[0139] The Gaussian error linear output result is input into the fourth-layer normalization unit to obtain a comparison result between the image to be evaluated and the target reference image, wherein the learning parameters of the first-layer normalization unit, the second-layer normalization unit, the third-layer normalization unit, and the fourth-layer normalization unit are different.
[0140] Layer Normalization (LN) can be used to normalize the features of each sample. Specifically, layer normalization normalizes the activation of each neuron so that it has a distribution with a mean of 0 and a variance of 1, thereby keeping the activation of each neuron on a relatively similar scale and improving the stability of training. This normalization method helps to accelerate the training of neural networks, reduce the problems of gradient vanishing and gradient exploding, and improve the generalization performance of the network.
[0141] The calculation formula for layer normalization is: LayerNorm(x)=γ*(x-μ) / (σ2+∈)0.5+β, where x is the input feature vector, μ is the mean of the feature vector, σ is the standard deviation of the feature vector, γ and β are the learned scaling coefficients and translation coefficients, and ∈ is a numerical value used to stabilize the calculation.
[0142] In an embodiment of the present application, the layer normalization unit includes a first layer normalization unit, a second layer normalization unit, a third layer normalization unit, and a fourth layer normalization unit, wherein the first layer normalization unit is used to perform normalization operations on the integration results; the second layer normalization unit is used to perform normalization operations on the initial updated integration results; the third layer normalization unit is used to perform normalization operations on the updated integration results; and the fourth layer normalization unit is used to perform normalization operations on the Gaussian error linear output results.
[0143] The first normalization result may refer to an integration result after normalization processing.
[0144] The second normalized result may refer to an initial updated integration result after normalization processing.
[0145] The third normalization result may refer to an updated integration result after normalization processing.
[0146] The initial updated integration result may refer to the self-attention output result and the integration result after the integration process.
[0147] The self-attention output result may refer to the first normalized result after being processed by the multi-head self-attention unit.
[0148] Multi-Head Self-Attention (MSA) may refer to a unit based on an attention mechanism that computes multiple attention representations in parallel using multiple sets of different query, key, and value mappings, and linearly superimposes them to obtain the final multi-head attention representation. Such a design enables the model to focus on different aspects of the input sequence and learn global dependencies in the input sequence. In an embodiment of the present application, the multi-head self-attention unit may be used to convert the first normalized result into a multi-head attention representation.
[0149] The updated integration result may refer to the multilayer perceptron output result after the integration process and the initial updated integration result.
[0150] The multilayer perceptron output result may refer to a second normalized result after being processed by the multilayer perceptron unit.
[0151] A multi-layer perceptron (MLP) unit can be a feed-forward artificial neural network model, whose structure includes an input layer, one or more hidden layers, and an output layer. In a multi-layer perceptron, the neurons of each layer are fully connected to the neurons of the next layer, but there is no connection between the neurons of the same layer. This network structure allows the model to learn complex patterns and features in the input data, and transforms them through nonlinear activation functions, so that the model can solve nonlinear problems. In an embodiment of the present application, multiple perceptron units are used to process the second normalized result.
[0152] The comparison result may refer to a Gaussian error linear output result after normalization.
[0153] The Gaussian error linear output result may refer to a third normalized result after being processed by the Gaussian error linear unit.
[0154] Gaussian Error Linear Unit (GELU) is an activation function in the field of deep learning. The error linear unit based on Gaussian distribution can show good performance in some tasks and has stronger nonlinear modeling ability than other activation functions. In an embodiment of the present application, the Gaussian Error Linear Unit is used to process the third normalized result.
[0155] Among them, in the embodiment of the present application, according to the integration result, the first normalization result, the multi-head self-attention unit, the second layer normalization unit, and the multi-layer perceptron unit, an updated integration result is obtained, including:
[0156] Input the first normalized result into the multi-head self-attention unit to obtain the self-attention output result;
[0157] Integrate the self-attention output result and the integration result to obtain the initial updated integration result;
[0158] Input the initial update integration result into the second layer normalization unit to obtain a second normalization result;
[0159] Inputting the second normalized result into the multi-layer perceptron unit to obtain the multi-layer perceptron output result;
[0160] The multi-layer perceptron output result and the initial updated integration result are integrated to obtain the updated integration result.
[0161] For example, given an RGB image x, two reference images are selected based on the RGB image to form a triplet, represented as S = (x, R l , R u). Subsequently, the image in S is segmented into image blocks of size 32*32 using an image block strategy. These image blocks are then processed by three models, among which the position labeling model assigns a position information to each image block through a learnable matrix; the image embedding model converts these image blocks into embeddings; and the aesthetic factor prediction model predicts the aesthetic factor of each image block for comparison. Finally, the output of each model will be integrated and input into the decoder to predict the aesthetic quality difference between images, which is then converted into an aesthetic score. The entire framework can be expressed as:
[0162]
[0163] Among them, L diff is the loss function, and y is the control result. is the comparison result, which can be further expressed as:
[0164]
[0165] z0=[class;PEM(p)]+PIM(p)+APM(p),
[0166]
[0167]
[0168] Among them, LN1 is the first layer normalization unit, LN2 is the second layer normalization unit, LN3 is the third layer normalization unit, LN4 is the fourth layer normalization unit, GELU is the Gaussian error linear unit, MSA is the multi-head self-attention unit, MLP is the multi-layer perceptron unit, z is the integration result, is the initial update integration result, is the updated integration result, PEM(p) is the output of the image embedding model, PIM(p) is the output of the position tagging model, APM(p) is the output of the aesthetic factor prediction model, is the first normalization result, is the second normalization result, is the third normalization result, L is the number of iterations, class is the classification label, and the final state of this label is used as the representation before the task output, so that the framework can extract a global representation from the entire input sequence, which is convenient for overall aesthetic judgment.
[0169] In the embodiment of the present application, before inputting the integration result into the comparison model for comparison processing to obtain the comparison result of the image to be evaluated and the target reference image, the method further includes:
[0170] Acquire a model training sample, the model training sample comprising a sub-second training image and a sub-training reference image, wherein the sub-second training image is a sub-image in the second training image, the sub-training reference image is a sub-image in the training reference image, and the subject type of the second training image and the subject type of the training reference image meet preset type requirements;
[0171] Performing aesthetic factor prediction processing on the sub-image in the model training sample to obtain a training prediction result of the model training sample, wherein the training prediction result includes a first training prediction result of the sub-second training image and a second training prediction result of the sub-training reference image;
[0172] Obtaining a training comparison result output by the second to-be-trained model according to the model training samples and the training prediction results;
[0173] According to the training comparison result and the training control result, the second model to be trained is adjusted to obtain a comparison model, wherein the training control result is determined according to the training score label of the second training image in the model training sample and the training reference score label of the training reference image.
[0174] The model training samples may refer to samples used to train the second model to be trained, and the second model to be trained may refer to a neural network model.
[0175] The sub-second training image may refer to the second training image after segmentation processing, and the second training image includes a training score label, and the training score label may refer to a preset score of the second training image.
[0176] The sub-training reference image may refer to a training reference image after segmentation processing, and the training reference image includes a training reference score label, and the training reference score label may refer to a preset score of the training reference image.
[0177] The preset type requirement may mean that the subject type of the second training image is the same as the subject type of the training reference image.
[0178] The training prediction results may refer to model training samples after aesthetic factor prediction processing.
[0179] The training comparison result may refer to the output result obtained by the second model to be trained based on the input training image position marking result, training embedding result, and training prediction result. Among them, the training image position marking result may refer to the result after the model training sample is input into the position marking model for image position marking processing. The training embedding result may refer to the result after the model training sample is input into the image embedding model for image embedding processing.
[0180] The training control result may refer to a result used for comparison with the training comparison result, so that the training comparison result output by the second model to be trained is as close to the training control result as possible, so as to improve the training effect of the second model to be trained.
[0181] The training comparison result may be the ratio of the training score label of the second training image to the training reference score label of the training reference image, or the degree of proximity between the training score label of the second training image and the training reference score label of the training reference image. For example, when there is one training reference image, the score of the training reference image is a, and the score of the second training image is c, then the training comparison result = a / c; when there are two training reference images, the scores of the training reference images are a and b respectively, and the score of the second training image is c, then the training comparison result = (ca) / (ba).
[0182] In the embodiment of the present application, when there are more than two model training samples, and the training reference scoring labels of the training reference images corresponding to the sub-training reference images in different model training samples are different,
[0183] After performing the prediction processing of the aesthetic factors on the sub-images in the model training samples to obtain the training prediction results of the model training samples, the method further includes:
[0184] Determining a training order of model training samples according to the training reference score labels of the training reference images;
[0185] Determine, according to the training sequence of the model training samples, a target model training sample in the model training samples and a target training prediction result of the target model training sample, wherein the target model training sample is a sample that has not been trained and whose training sequence meets a preset sequence requirement;
[0186] Obtaining a target training comparison result output by the second to-be-trained model according to the target model training sample and the target training prediction result;
[0187] According to the target training comparison result and the target training control result, the second model to be trained is adjusted to obtain an initial comparison model;
[0188] Repeat the steps of determining the target model training samples in the model training samples and the target training prediction results of the target model training samples according to the training order of the model training samples until all the model training samples are samples to be used for training, thereby obtaining a comparison model.
[0189] The training order may refer to the order in which the model training samples are input into the second model to be trained for training.
[0190] The preset order requirement may refer to a preset requirement for determining the training order based on the training reference scoring label. In an embodiment of the present application, the closer the training reference scoring label is to the training scoring label, the later the training order of the model training sample is. For example, assuming that the training reference scoring label is 5 points, the training reference scoring label A in the model training sample A is 10 points, and the training reference scoring label B in the model training sample B is 9 points, it is determined that the difference between the training reference scoring label and the training reference scoring label A is 5 points, and the difference between the training reference scoring label and the training reference scoring label B is 4 points. After comparison, the difference between the training reference scoring label and the training reference scoring label A is larger, then the training order of the model training sample A precedes the training order of the model training sample B.
[0191] The target training prediction result may refer to the result of the prediction processing of the aesthetic factors on the target model training sample. Specifically, the target model training sample may be input into the aesthetic factor prediction model to obtain the target training prediction result.
[0192] The target training comparison result may refer to the output result obtained by the second model to be trained based on the input target training image position marking result, target training embedding result, and target training prediction result. Among them, the target training image position marking result may refer to the result after the target model training sample is input into the position marking model for image position marking processing. The target training embedding result may refer to the result after the target model training sample is input into the image embedding model for image embedding processing.
[0193] The target training control result may refer to a training control result corresponding to a target model training sample.
[0194] In this embodiment, the closer the two scoring labels are, the smaller the difference in the aesthetic factors of the two images is, and the smaller the distinguishing features of the two images are. Therefore, by sorting the model training samples, the second model to be trained first learns the model training samples with larger distinguishing features, and then learns the model training samples with smaller distinguishing features. The learning parameters of the second model to be trained are adjusted from large to small, so that the training effect of the obtained comparison model is better.
[0195] S205 . Determine an evaluation result of the image to be evaluated according to the comparison result and the target reference score label of the target reference image.
[0196] The target reference score label may refer to a score of a preset target reference image.
[0197] The evaluation result may refer to the score of the image to be evaluated.
[0198] The method for determining the evaluation result may include determining the evaluation result according to the number of target reference images evaluated in a single evaluation. When the target reference image for a single evaluation is one, the evaluation result may be the product of the comparison result and the score of the target reference image; when the target reference images for a single evaluation are two, the evaluation result may be the sum of the score of the smaller target reference image and the product result, wherein the product result is the product of the comparison result and the score difference, and the score difference is the difference between the scores of the two target reference images. For example, the comparison result is 0.44, the target reference images include A and B, the score of A is 3 points, and the score of B is 7 points, then the evaluation result = 3 + 0.44 * (7-3).
[0199] The embodiment of the present application provides a multi-factor image evaluation method based on aesthetic relativity. By determining a target reference image of the same subject type, the target reference image and the image to be evaluated can be better compared. By performing aesthetic factor prediction processing on the target reference image and the image to be evaluated, the respective aesthetic factor prediction results are obtained, and the characteristics of the aesthetic factors are converted into data representation, so that the computer can identify the degree of difference between the image to be evaluated and the target reference image according to the aesthetic factor prediction results, thereby obtaining an evaluation result with better evaluation effect. By constructing an aesthetic factor prediction model, the aesthetic factors of the image are represented by image editing data, so that the aesthetic factors are easier to understand and the training effect of the comparison model is improved. By sorting the model training samples, the second model to be trained first learns the model training samples with larger distinguishing features, and then learns the model training samples with smaller distinguishing features. The learning parameters of the second model to be trained are adjusted from large to small, so that the training effect of the obtained comparison model is better.
[0200] Figure 3 A schematic diagram of the structure of an image multi-factor evaluation device based on aesthetic relativity provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the device includes: an acquisition module 301, a segmentation module 302, a prediction module 303, a comparison module 304, and a determination module 305, wherein:
[0201] An acquisition module 301 is used to acquire an image to be evaluated and a target reference image, wherein the target reference image and the image to be evaluated have the same subject type;
[0202] The segmentation module 302 is used to perform image segmentation processing on the image to be evaluated and the target reference image respectively to obtain a sub-image to be evaluated in the image to be evaluated and a sub-target reference image in the target reference image;
[0203] The prediction module 303 is used to perform aesthetic factor prediction processing on the sub-image to be evaluated and the sub-target reference image respectively, to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image, wherein the first prediction result and the second prediction result represent the prediction result of at least one aesthetic factor of artifacts, composition, color, noise, exposure, clarity and white balance in the image;
[0204] A comparison module 304 is used to obtain a comparison result of the image to be evaluated and the target reference image according to the sub-image to be evaluated, the sub-target reference image, the first prediction result, and the second prediction result, wherein the comparison result represents the difference between the aesthetic factors of the image to be evaluated and the aesthetic factors of the target reference image;
[0205] The determination module 305 is used to determine the evaluation result of the image to be evaluated according to the comparison result and the target reference score label of the target reference image.
[0206] In this embodiment of the present application, the prediction module 303 is also used for:
[0207] Determine the prediction model of aesthetic factors;
[0208] The sub-image to be evaluated and the sub-target reference image are respectively input into the aesthetic factor prediction model for aesthetic factor prediction processing to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image.
[0209] In this embodiment of the present application, the comparison module 304 is also used for:
[0210] Inputting the sub-image to be evaluated and the sub-target reference image into the position marking model for image position marking processing, thereby obtaining a first image position marking result of the sub-image to be evaluated and a second image position marking result of the sub-target reference image;
[0211] Inputting the sub-image to be evaluated and the sub-target reference image into the image embedding model for image embedding processing, thereby obtaining a first embedding result of the sub-image to be evaluated and a second embedding result of the sub-target reference image;
[0212] Integrate the first image position marking result, the second image position marking result, the first embedding result, the second embedding result, the first prediction result, and the second prediction result to obtain an integration result;
[0213] The integrated results are input into the comparison model for comparison processing to obtain the comparison results between the image to be evaluated and the target reference image.
[0214] In this embodiment of the present application, the comparison module 304 is also used for:
[0215] Inputting the integration result into the first-layer normalization unit in the comparison model to obtain the first normalization result;
[0216] According to the integration result, the first normalization result, the multi-head self-attention unit, the second-layer normalization unit, and the multi-layer perceptron unit, an updated integration result is obtained, wherein the updated integration result is determined according to the second-layer normalization unit, the multi-layer perceptron unit, and the initial updated integration result, and the initial updated integration result is determined according to the integration result, the first normalization result, and the multi-head self-attention unit;
[0217] Input the updated integration result into the third layer normalization unit to obtain the third normalization result;
[0218] The third normalization result is input into the Gaussian error linear unit to obtain a Gaussian error linear output result;
[0219] The Gaussian error linear output result is input into the fourth-layer normalization unit to obtain a comparison result between the image to be evaluated and the target reference image, wherein the learning parameters of the first-layer normalization unit, the second-layer normalization unit, the third-layer normalization unit, and the fourth-layer normalization unit are different.
[0220] In this embodiment of the present application, the comparison module 304 is also used for:
[0221] Input the first normalized result into the multi-head self-attention unit to obtain the self-attention output result;
[0222] Integrate the self-attention output result and the integration result to obtain the initial updated integration result;
[0223] Input the initial update integration result into the second layer normalization unit to obtain a second normalization result;
[0224] Inputting the second normalized result into the multi-layer perceptron unit to obtain the multi-layer perceptron output result;
[0225] The multi-layer perceptron output result and the initial updated integration result are integrated to obtain the updated integration result.
[0226] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 40 includes:
[0227] The electronic device 40 may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a communication component 403 and other components. The processor 401 , the memory 402 and the communication component 403 are connected via a bus 404 .
[0228] In a specific implementation process, at least one processor 401 executes the computer-executable instructions stored in the memory 402, so that at least one processor 401 executes the above-mentioned image multi-factor evaluation method based on the theory of aesthetic relativity.
[0229] The specific implementation process of the processor 401 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0230] In the above Figure 4 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0231] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.
[0232] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.
[0233] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which, when executed by a processor, implements the steps in any of the above-mentioned image multi-factor evaluation methods based on aesthetic relativity.
[0234] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0235] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0236] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the multi-factor image evaluation methods based on aesthetic relativity provided in the embodiment of the present application.
[0237] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0238] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program comprises computer instructions stored in a computer-readable storage medium.
[0239] Since the instructions stored in the storage medium can execute the steps in any one of the multi-factor image evaluation methods based on aesthetic relativity provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the multi-factor image evaluation methods based on aesthetic relativity provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0240] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0241] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A multi-factor image evaluation method based on aesthetic relativity, characterized in that: include: Acquire an image to be evaluated and a target reference image, wherein the target reference image and the image to be evaluated have the same subject type; Performing image segmentation processing on the image to be evaluated and the target reference image respectively to obtain a sub-image to be evaluated in the image to be evaluated and a sub-target reference image in the target reference image; Performing aesthetic factor prediction processing on the sub-image to be evaluated and the sub-target reference image respectively to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image, wherein the first prediction result and the second prediction result represent prediction results of at least one aesthetic factor of artifacts, composition, color, noise, exposure, clarity and white balance in the image; Obtaining a comparison result of the image to be evaluated and the target reference image according to the sub-image to be evaluated, the sub-target reference image, the first prediction result, and the second prediction result, wherein the comparison result represents a degree of difference between an aesthetic factor of the image to be evaluated and an aesthetic factor of the target reference image; Determining an evaluation result of the image to be evaluated according to the comparison result and the target reference score label of the target reference image; The obtaining, according to the sub-image to be evaluated, the sub-target reference image, the first prediction result, and the second prediction result, a comparison result between the image to be evaluated and the target reference image includes: Inputting the sub-image to be evaluated and the sub-target reference image into a position marking model for image position marking processing to obtain a first image position marking result of the sub-image to be evaluated and a second image position marking result of the sub-target reference image; Inputting the sub-image to be evaluated and the sub-target reference image into an image embedding model for image embedding processing, thereby obtaining a first embedding result of the sub-image to be evaluated and a second embedding result of the sub-target reference image; Integrate the first image position marking result, the second image position marking result, the first embedding result, the second embedding result, the first prediction result, and the second prediction result to obtain an integration result; Inputting the integration result into a comparison model for comparison processing to obtain a comparison result between the image to be evaluated and the target reference image; The step of inputting the integration result into a comparison model for comparison processing to obtain a comparison result between the image to be evaluated and the target reference image includes: Inputting the integration result into the first-layer normalization unit in the comparison model to obtain a first normalization result; According to the integration result, the first normalization result, the multi-head self-attention unit, the second layer normalization unit, and the multi-layer perceptron unit, an updated integration result is obtained, wherein the updated integration result is determined according to the second layer normalization unit, the multi-layer perceptron unit, and the initial updated integration result, and the initial updated integration result is determined according to the integration result, the first normalization result, and the multi-head self-attention unit; Inputting the updated integration result into the third layer normalization unit to obtain a third normalization result; Inputting the third normalized result into a Gaussian error linear unit to obtain a Gaussian error linear output result; Inputting the Gaussian error linear output result into a fourth-layer normalization unit to obtain a comparison result between the image to be evaluated and the target reference image, wherein the learning parameters of the first-layer normalization unit, the second-layer normalization unit, the third-layer normalization unit, and the fourth-layer normalization unit are different; The updating integration result is obtained according to the integration result, the first normalization result, the multi-head self-attention unit, the second layer normalization unit, and the multi-layer perceptron unit, including: Input the first normalized result into the multi-head self-attention unit to obtain a self-attention output result; Integrate the self-attention output result and the integration result to obtain an initial updated integration result; Inputting the initial update integration result into the second layer normalization unit to obtain a second normalization result; Inputting the second normalized result into a multi-layer perceptron unit to obtain a multi-layer perceptron output result; The multilayer perceptron output result and the initial updated integration result are integrated to obtain an updated integration result.
2. The method according to claim 1, characterized in that The performing aesthetic factor prediction processing on the sub-image to be evaluated and the sub-target reference image respectively to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image includes: Determine the prediction model of aesthetic factors; The sub-image to be evaluated and the sub-target reference image are respectively input into the aesthetic factor prediction model for aesthetic factor prediction processing to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image.
3. The method according to claim 2, characterized in that Before determining the aesthetic factor prediction model, the method further includes: Acquire a first training image; Editing the aesthetic factors of the first training image to obtain an edited image and editing data of the edited image; Inputting the edited image into a first model to be trained to perform aesthetic factor prediction processing to obtain predicted edited data; The first model to be trained is adjusted according to the predicted editing data and the editing data of the edited image to obtain an aesthetic factor prediction model.
4. The method according to claim 1, characterized in that: Before inputting the integration result into the comparison model for comparison processing to obtain the comparison result of the image to be evaluated and the target reference image, the method further includes: Acquire a model training sample, wherein the model training sample includes a sub-second training image and a sub-training reference image, wherein the sub-second training image is a sub-image in the second training image, the sub-training reference image is a sub-image in the training reference image, and the subject type of the second training image and the subject type of the training reference image meet a preset type requirement; Performing prediction processing of aesthetic factors on the sub-images in the model training samples to obtain training prediction results of the model training samples, wherein the training prediction results include a first training prediction result of the sub-second training image and a second training prediction result of the sub-training reference image; Obtaining a training comparison result output by a second model to be trained according to the model training sample and the training prediction result; According to the training comparison result and the training control result, the second model to be trained is adjusted to obtain a comparison model, wherein the training control result is determined according to the training score label of the second training image and the training reference score label of the training reference image in the model training sample.
5. The method according to claim 4, characterized in that When there are more than two model training samples, and the training reference scoring labels of the training reference images corresponding to the sub-training reference images in different model training samples are different, After performing the prediction process of the aesthetic factors on the sub-images in the model training samples to obtain the training prediction results of the model training samples, the method further includes: Determining a training order of the model training samples according to the training reference score labels of the training reference images; Determine, according to the training sequence of the model training samples, a target model training sample in the model training samples and a target training prediction result of the target model training sample, wherein the target model training sample is a sample that has not been trained and whose training sequence meets a preset sequence requirement; Obtaining a target training comparison result output by a second to-be-trained model according to the target model training sample and the target training prediction result; According to the target training comparison result and the target training control result, the second model to be trained is adjusted to obtain an initial comparison model; Repeat the steps of determining the target model training samples in the model training samples and the target training prediction results of the target model training samples according to the training order of the model training samples, until all the model training samples are samples used for training, and obtain a comparison model.
6. An image multi-factor evaluation device based on aesthetic relativity, characterized in that: include: An acquisition module, used for acquiring an image to be evaluated and a target reference image, wherein the target reference image and the image to be evaluated have the same subject type; A segmentation module, used to perform image segmentation processing on the image to be evaluated and the target reference image respectively, to obtain a sub-image to be evaluated in the image to be evaluated and a sub-target reference image in the target reference image; A prediction module is used to perform aesthetic factor prediction processing on the sub-image to be evaluated and the sub-target reference image respectively, to obtain a first prediction result of the sub-image to be evaluated and a second prediction result of the sub-target reference image, wherein the first prediction result and the second prediction result represent the prediction result of at least one aesthetic factor of artifacts, composition, color, noise, exposure, clarity and white balance in the image; a comparison module, configured to obtain a comparison result of the image to be evaluated and the target reference image according to the sub-image to be evaluated, the sub-target reference image, the first prediction result, and the second prediction result, wherein the comparison result represents a degree of difference between the aesthetic factors of the image to be evaluated and the aesthetic factors of the target reference image; A determination module, configured to determine an evaluation result of the image to be evaluated according to the comparison result and a target reference score label of the target reference image; The comparison module is specifically used for: Inputting the sub-image to be evaluated and the sub-target reference image into a position marking model for image position marking processing to obtain a first image position marking result of the sub-image to be evaluated and a second image position marking result of the sub-target reference image; Inputting the sub-image to be evaluated and the sub-target reference image into an image embedding model for image embedding processing, thereby obtaining a first embedding result of the sub-image to be evaluated and a second embedding result of the sub-target reference image; Integrate the first image position marking result, the second image position marking result, the first embedding result, the second embedding result, the first prediction result, and the second prediction result to obtain an integration result; Inputting the integration result into a comparison model for comparison processing to obtain a comparison result between the image to be evaluated and the target reference image; The comparison module, when used to input the integration result into the comparison model for comparison processing to obtain the comparison result of the image to be evaluated and the target reference image, is specifically used to: Inputting the integration result into the first-layer normalization unit in the comparison model to obtain a first normalization result; According to the integration result, the first normalization result, the multi-head self-attention unit, the second layer normalization unit, and the multi-layer perceptron unit, an updated integration result is obtained, wherein the updated integration result is determined according to the second layer normalization unit, the multi-layer perceptron unit, and the initial updated integration result, and the initial updated integration result is determined according to the integration result, the first normalization result, and the multi-head self-attention unit; Inputting the updated integration result into the third layer normalization unit to obtain a third normalization result; Inputting the third normalized result into a Gaussian error linear unit to obtain a Gaussian error linear output result; Inputting the Gaussian error linear output result into a fourth-layer normalization unit to obtain a comparison result between the image to be evaluated and the target reference image, wherein the learning parameters of the first-layer normalization unit, the second-layer normalization unit, the third-layer normalization unit, and the fourth-layer normalization unit are different; The comparison module, when used to obtain an updated integration result according to the integration result, the first normalization result, the multi-head self-attention unit, the second layer normalization unit, and the multi-layer perceptron unit, is specifically used to: Input the first normalized result into the multi-head self-attention unit to obtain a self-attention output result; Integrate the self-attention output result and the integration result to obtain an initial updated integration result; Inputting the initial update integration result into the second layer normalization unit to obtain a second normalization result; Inputting the second normalized result into a multi-layer perceptron unit to obtain a multi-layer perceptron output result; The multilayer perceptron output result and the initial updated integration result are integrated to obtain an updated integration result.
7. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 5.
