An unsupervised image aesthetic attribute measurement method and system based on attribute understanding

Abstract

The present invention relates to an unsupervised image aesthetic attribute measurement method and system based on attribute understanding. The method comprises the following steps: S1: constructing an image evaluation model; S2: selecting an image set as a training set and inputting it into a global feature extraction module to obtain global features; S3: inputting the training set into an attribute feature extraction module to obtain individual attribute features; S4: concatenating the global features and individual attribute features, inputting them into an attention module, assigning corresponding weights to each attribute feature, and outputting attention features and attribute contribution values; S5: splitting the attention features by attribute, inputting them into an attribute score prediction module, scoring each attribute feature, and outputting scores for each attribute of the image and an overall image score. The method provided by the present invention obtains aesthetic numerical measurements of individual image attributes through attribute feature training, even when only the overall aesthetic evaluation labels of the image dataset are available.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to an unsupervised image aesthetic attribute measurement method and system based on attribute understanding. Background Art

[0002] In recent years, Aesthetic Attribute Evaluation of Images (AAEI) has attracted widespread attention from AI researchers. It not only outputs an overall aesthetic score but also evaluates aesthetic attributes such as lighting, composition, color arrangement, and image structure. Annotating the overall and attribute aesthetic scores of images is a non-trivial task that requires the experience of artists and professional photographers. To date, only a small number of public images are labeled with aesthetic attribute scores. For each image, there are only a handful of non-professional annotators, which limits the performance of the AAEI task. How to evaluate unlabeled aesthetic attributes, such as the AVA dataset, which only has an overall aesthetic score label, is a challenge.

[0003] Some researchers have employed professionals or recruited volunteers to perform the scoring. Despite significant effort, the results have been suboptimal. First, evaluators need to have sufficient aesthetic knowledge. Second, each image needs to be scored by a different evaluator to reduce the influence of individuality. Last but not least, the intensive workload can impair evaluators' judgment, requiring them to have ample time to relax. In short, manual attribute scoring appears infeasible for large-scale attribute scoring.

[0004] At present, there is no method that can effectively solve the contradiction between the scarcity of image aesthetic attribute datasets and the need to perform image aesthetic evaluation based on image aesthetic attributes. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides an unsupervised image aesthetic attribute measurement method and system based on attribute understanding.

[0006] The technical solution of the present invention is: an unsupervised image aesthetic attribute measurement method based on attribute understanding, comprising:

[0007] Step S1: construct an image evaluation model, including: a global feature extraction module, an attribute feature extraction module, an attention module, and an attribute score prediction module;

[0008] Step S2: Select an image set as a training set and input it into a global feature extraction module to obtain global features;

[0009] Step S3: inputting the training set into the attribute feature extraction module to obtain various attribute features;

[0010] Step S4: After the global features and the attribute features are spliced together, they are input into the attention module, and after assigning corresponding weights to each attribute feature, the attention feature and the attribute contribution value are output;

[0011] Step S5: After the attention features are split according to attributes, they are input into the attribute score prediction module, which scores each attribute feature and outputs the attribute scores of the image and the overall image score.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] The method provided by the present invention can obtain aesthetic numerical measurements of various attributes of an image through attribute feature training when only the overall aesthetic evaluation label of an image dataset is available, and can provide a solution to the problem that it is difficult to conduct research on the aesthetic attributes of an image when the aesthetic attribute dataset is scarce. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of an unsupervised image aesthetic attribute measurement method based on attribute understanding in an embodiment of the present invention;

[0015] Figure 2 Schematic diagram of the structure of an image evaluation model according to an embodiment of the present invention;

[0016] Figure 3 Schematic diagram of another image evaluation model according to an embodiment of the present invention;

[0017] Figure 4 This is a schematic diagram of the structure of the illumination attribute module in an embodiment of the present invention;

[0018] Figure 5 Schematic diagram of the composition attribute module structure in an embodiment of the present invention;

[0019] Figure 6 This is a schematic diagram of the structure of the structural attribute module in an embodiment of the present invention;

[0020] Figure 7 Schematic diagram of the depth of field attribute module structure in an embodiment of the present invention;

[0021] Figure 8 This is a schematic diagram of the structure of the face illumination attribute module in an embodiment of the present invention;

[0022] Figure 9 This is a schematic diagram of the structure of the head angle attribute module in an embodiment of the present invention;

[0023] Figure 10 This is a schematic diagram of the structure of a facial appearance attribute module in an embodiment of the present invention;

[0024] Figure 11This is a schematic diagram of the structure of the facial expression attribute module in an embodiment of the present invention;

[0025] Figure 12 This is a schematic diagram of the structure of a human body posture attribute module in an embodiment of the present invention;

[0026] Figure 13 This is a structural block diagram of an unsupervised image aesthetic attribute measurement system based on attribute understanding in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention provides an unsupervised image aesthetic attribute measurement method based on attribute understanding. In the case of only the overall aesthetic evaluation label of the image data set, the aesthetic numerical measurement of each attribute of the image is obtained through attribute feature training.

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementation and in conjunction with the accompanying drawings.

[0029] Example 1

[0030] like Figure 1 As shown, an embodiment of the present invention provides an unsupervised image aesthetic attribute measurement method based on attribute understanding, comprising the following steps:

[0031] Step S1: construct an image evaluation model, including: a global feature extraction module, an attribute feature extraction module, an attention module, and an attribute score prediction module;

[0032] Step S2: Select an image set as a training set and input it into a global feature extraction module to obtain global features;

[0033] Step S3: Input the training set into the attribute feature extraction module to obtain each attribute feature;

[0034] Step S4: After the global features and each attribute feature are spliced together, they are input into the attention module, and after assigning corresponding weights to each attribute feature, the attention feature and attribute contribution value are output;

[0035] Step S5: After the attention features are split according to attributes, they are input into the attribute score prediction module to score each attribute feature and output the attribute scores of the image and the overall image score.

[0036] In one embodiment, the above step S1: constructing an image evaluation model includes: a global feature extraction module, an attribute feature extraction module, an attention module and an attribute score prediction module, specifically including:

[0037] like Figure 2As shown in the figure, a structural diagram of the image evaluation model is shown. The picture is input into the image evaluation model, the global features are extracted through the global feature extraction module, and the corresponding attribute features are extracted according to different attribute feature extraction modules. After the global and attribute features are fused, different weights are assigned through the attention module, and the contribution value of each attribute is output. Finally, the total score of the image and the scores of different attributes are given respectively through the attribute score prediction module.

[0038] In one embodiment, S2: selecting an image set as a training set and inputting the image set into a global feature extraction module to obtain global features, specifically includes:

[0039] The global feature extraction module uses the feature extraction module in the EfficientNet-B4 network, inputs the images in the training set into the feature extraction module of EfficientNet-B4, and uses its output as the global feature.

[0040] The embodiment of the present invention uses the AVA dataset as a training set. AVA is currently the largest image aesthetic dataset, and the AVA dataset provides a number of people who rated each score from 1 to 10 for each image.

[0041] The global feature extraction module of the present invention only uses the feature extraction module in the EfficientNet-B4 network, inputs the AVA data set into the EfficientNet-B4 network, uses the features output by its feature extraction module as the global features of the present invention, and discards the attention module in the EfficientNet-B4 network.

[0042] In one embodiment, the above step S3: inputting the training set into the attribute feature extraction module to obtain various attribute features specifically includes:

[0043] The attribute feature extraction module includes: lighting attribute module, composition attribute module, structure attribute module, depth of field attribute module, human body composition module, human body posture attribute module, head angle attribute module, face lighting attribute, face appearance attribute module, and face expression attribute module; according to the content of the pictures in the training set, the corresponding network model is used to extract each attribute feature.

[0044] In the attribute feature extraction module, you can adjust the appropriate image attribute feature module according to your needs, such as Figure 2 The following only shows the attribute feature modules of ordinary images: lighting attribute module, composition attribute module, structure attribute module, and depth of field attribute module. If the input image is a face image, you can also use Figure 3 The attribute feature modules related to portraits shown are: human body composition module, human body posture attribute module, head angle attribute module, face lighting attribute module, face appearance attribute module, and face expression attribute module.

[0045] Light attribute module: The embodiment of the present invention uses the outdoor-sky light detection model to extract the light characteristics of the input image, such as Figure 4 As shown in Figure 1, the extracted illumination feature is 256×11×11.

[0046] Composition attribute module: The embodiment of the present invention uses the DHT composition line detection model to extract the composition features of the input image, such as Figure 5 As shown in the figure, the extracted composition features are 2048×22×22.

[0047] Structural attribute module: The embodiment of the present invention uses the DETR object detection model to extract the structural features of the input image, such as Figure 6 As shown in Figure 1, the extracted composition features are 2048×11×11.

[0048] Depth of field attribute module: The embodiment of the present invention uses the Monodepth2 depth map estimation model to extract the depth of field features of the input image, such as Figure 7 As shown in FIG, the extracted depth of field feature is 512×11×11.

[0049] To extract facial attribute features, first use a face detection model such as dlib to extract the face image from the input image, and then use the following facial attribute-related attribute modules to extract the corresponding attribute features.

[0050] Face illumination attribute module: The embodiment of the present invention uses the DPR face illumination estimation model to extract the face illumination features of the face image, such as Figure 8 As shown in Figure 1, the extracted facial illumination feature is 9×1×1.

[0051] Head angle attribute module: The embodiment of the present invention uses the FHPE head pose estimation model to extract the head angle features of the face image, such as Figure 9 As shown in the figure, the extracted head angle feature is 3×66.

[0052] Face appearance attribute module: The embodiment of the present invention uses the OpenFace face recognition model to extract the face appearance features of the face image, such as Figure 10 As shown in the figure, the extracted facial appearance features are 36×128.

[0053] Facial expression attribute module: The embodiment of the present invention uses the fer2013 dataset as input to the face recognition model to extract the facial appearance features of the face image, such as Figure 11 As shown in the figure, the extracted facial expression features are 256×1×1.

[0054] Human posture attribute module: The embodiment of the present invention uses the SHN human posture estimation model to extract the posture features of the face image, such as Figure 12As shown in the figure, the human body posture features extracted are 256×256.

[0055] Human body composition attribute module: The embodiment of the present invention uses the MMAN human body segmentation model to extract human body features of facial images, and the extracted human body segmentation features are 256×256×3.

[0056] The above attribute feature extraction modules are merely examples provided in the embodiments of the present invention, and can be added or deleted according to actual needs.

[0057] In one embodiment, the above step S4: after concatenating the global features and the attribute features, inputting them into the attention module, assigning corresponding weights to each attribute feature, and outputting the attention features and attribute contribution values, specifically includes:

[0058] Step S41: Adjust the global features and attribute features to the same number of channels and feature map size and then splice them. Input the spliced features into the attention module, assign corresponding weights to each attribute feature, and then output the attention features.

[0059] Step S42: The attribute contribution value calculation formula (1) is as follows:

[0060]

[0061] Among them, C a is the contribution value of attribute a, n a is the number of weights of attribute a, w a is the weight coefficient of attribute a output by the attention module, total is the number of all weights, w all is the weight coefficient of all attributes.

[0062] The embodiment of the present invention adjusts the global features and each local feature to have the same number of channels and feature map size, such as Figure 2 As shown, the adjusted feature size is 256×11×11, and then the four attribute features and one global feature are spliced. The spliced feature size is 1280×11×11, and then input into the attention module. The embodiment of the present invention uses the ECA channel attention module to assign a weight to each channel to adjust the importance of the channel, output the attention feature, and calculate the contribution value of each attribute.

[0063] In one embodiment, the above step S5: after the attention features are split according to attributes, they are input into the attribute score prediction module, which scores each attribute feature and outputs the attribute scores of the image and the overall image score. Specifically, it includes:

[0064] Step S51: Split the attention feature according to the attribute feature size to obtain the attribute attention feature. Each attribute attention feature passes through the scoring layer and outputs the score P of each attribute of the image at the corresponding level i. ai , and construct the loss function as formula (2):

[0065]

[0066] Among them, the total score of the picture is scored from 1 to 10 points, that is, it is divided into 10 levels, i is the level, is the predicted score of the scoring layer level i, where n is the number of attributes, P ai is the output of attribute a at level i; is the actual score of rating level i, where N i is the number of people who scored level i, N all For all the number of scorers;

[0067] For example, after the scoring layer, the predicted score P of each attribute feature at each scoring level is ai As shown in Table 1:

[0068] Table 1: Prediction scores of various attribute features at various rating levels

[0069]

[0070]

[0071] Calculate P1 = (0.0 + 0.08 + 0.0 + 0.0 + 0.0) / 5 = 0.016, where P1 is the predicted score of all features at level 1, and so on to get P2 ~ P 10 The predicted score is obtained, and then the real score R1~R in the AVA data set is obtained. 10 , then the following Table 2 is formed:

[0072] Table 2: Predicted scores and true scores of all attribute features at each rating level

[0073]

[0074] The loss function according to formula (2) is used to train the image evaluation model until convergence;

[0075] Step S52: Output the total score of the image, as shown in formula (3):

[0076]

[0077] The method provided by the present invention can obtain aesthetic numerical measurements of various attributes of an image through attribute feature training when only the overall aesthetic evaluation label of an image dataset is available, and can provide a solution to the problem that it is difficult to conduct research on the aesthetic attributes of an image when the aesthetic attribute dataset is scarce.

[0078] Example 2

[0079] like Figure 13 As shown, an embodiment of the present invention provides an unsupervised image aesthetic attribute measurement system based on attribute understanding, including the following modules:

[0080] An image evaluation model construction module 61 is used to construct an image evaluation model, including: a global feature extraction module, an attribute feature extraction module, an attention module, and an attribute score prediction module;

[0081] A global feature acquisition module 62 is used to select an image set as a training set and input it into a global feature extraction module to obtain global features;

[0082] The attribute feature acquisition module 63 is used to input the training set into the attribute feature extraction module to obtain various attribute features;

[0083] The attention feature acquisition module 64 is used to combine the global features and the attribute features, input them into the attention module, assign corresponding weights to each attribute feature, and output the attention features and attribute contribution values;

[0084] The scoring module 65 is used to split the attention features according to attributes, input them into the attribute score prediction module, score each attribute feature, and output the attribute scores of the image and the overall image score.

[0085] The above embodiments are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention are intended to be within the scope of the present invention.

Claims

1. An unsupervised image aesthetic attribute measurement method based on attribute understanding, characterized by: include: Step S1: construct an image evaluation model, including: a global feature extraction module, an attribute feature extraction module, an attention module, and an attribute score prediction module; Step S2: Select an image set as a training set and input it into a global feature extraction module to obtain global features; Step S3: inputting the training set into the attribute feature extraction module to obtain various attribute features; Step S4: After the global features and the attribute features are spliced together, they are input into the attention module, and after assigning corresponding weights to each attribute feature, the attention feature and the attribute contribution value are output; Step S5: After the attention features are split according to attributes, they are input into the attribute score prediction module, which scores each attribute feature and outputs the attribute scores of the image and the overall image score, specifically including: Step S51: Split the attention feature according to the attribute feature size to obtain the attribute attention feature, and output the score of each attribute of the image at the corresponding level i through the scoring layer for each attribute of the image. , and construct the loss function as formula (2): (2) Among them, the total score of the picture is scored from 1 to 10 points, that is, it is divided into 10 levels. For level, = For the rating level The predicted score of , where n is the number of attributes, For attributes In grade Output; For the rating level The real score of Rating for grade Number of people, For all the number of scorers; Step S52: Output the total score of the image , as shown in the calculation formula (3): (3)。 2. The unsupervised image aesthetic attribute measurement method based on attribute understanding according to claim 1 is characterized in that: Step S2: selecting an image set as a training set and inputting it into a global feature extraction module to obtain global features, specifically includes: The global feature extraction module uses the feature extraction module in the EfficientNet-B4 network, inputs the images in the training set into the feature extraction module of the EfficientNet-B4, and outputs them as global features.

3. The unsupervised image aesthetic attribute measurement method based on attribute understanding according to claim 1 is characterized in that: Step S3: inputting the training set into the attribute feature extraction module to obtain various attribute features, specifically including: The attribute feature modules include: lighting attribute module, composition attribute module, structure attribute module, depth of field attribute module, human body composition module, human body posture attribute module, head angle attribute module, face lighting attribute, face appearance attribute module, and face expression attribute module; according to the content of the pictures in the training set, the corresponding network models are used to extract each of the attribute features.

4. The unsupervised image aesthetic attribute measurement method based on attribute understanding according to claim 1 is characterized in that: Step S4: After the global features and the attribute features are combined, they are input into the attention module, and after the corresponding weight is assigned to each attribute feature, the attention feature and the attribute contribution value are output. Specifically, the following steps are performed: Step S41: Adjust the global features and the attribute features to have the same number of channels and feature map size, then splice them together. Input the spliced features into the attention module, assign corresponding weights to each attribute feature, and output the attention features and attribute contribution values. Step S42: The attribute contribution value calculation formula is as follows: (1) in, is the contribution value of attribute a, is the number of weights of attribute a, is the weight coefficient of attribute a output by the attention module, and total is the number of all weights. is the weight coefficient of all attributes.

5. An unsupervised image aesthetic attribute measurement system based on attribute understanding, characterized by: Includes the following modules: Construct an image evaluation model module, which is used to construct an image evaluation model, including: a global feature extraction module, an attribute feature extraction module, an attention module, and an attribute score prediction module; The global feature acquisition module is used to select an image set as a training set and input it into the global feature extraction module to obtain global features; An attribute feature acquisition module is used to input the training set into an attribute feature extraction module to obtain various attribute features; An attention feature acquisition module is used to combine the global feature and each of the attribute features, input the combined feature into an attention module, assign a corresponding weight to each of the attribute features, and output the attention feature and attribute contribution value; The scoring module is used to split the attention features according to attributes, input them into the attribute score prediction module, score each attribute feature, and output the attribute scores of each image and the overall image score. Specifically, it includes: Step S51: Split the attention feature according to the attribute feature size to obtain the attribute attention feature, and output the score of each attribute of the image at the corresponding level i through the scoring layer for each attribute of the image. , and construct the loss function as formula (2): (2) Among them, the total score of the picture is scored from 1 to 10 points, that is, it is divided into 10 levels. For level, = For the rating level The predicted score of , where n is the number of attributes, For attributes In grade Output; For the rating level The real score of Rating for grade Number of people, For all the number of scorers; Step S52: Output the total score of the image , as shown in the calculation formula (3): (3)。

Images