Image aesthetic quality evaluation method, device, electronic device and storage medium

Abstract

The present invention relates to the field of image processing technology, and provides a method, device, electronic device, and storage medium for evaluating the aesthetic quality of an image. The method comprises: performing target detection on an initial image to obtain a target image; inputting the target image into an evaluation model to obtain a distribution of predicted values ​​corresponding to multiple aesthetic quality labels output by multiple output nodes of the evaluation model; wherein the evaluation model is trained based on a multi-scale classification loss, the multi-scale classification loss including the prediction loss of each label group formed by classifying the aesthetic quality labels at different granularities; and calculating the mathematical expectation based on the predicted value distribution to obtain an aesthetic quality evaluation result of the target image. The present invention uses the multi-scale classification loss to strongly constrain the scale-invariant relationship between classification scores, making the output distribution more reasonable, and solving the problem of missing the order relationship between categories when using classification to process regression problems, as well as the problem of unreasonable classification output distribution.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image aesthetic quality evaluation method, device, electronic device and storage medium. Background Art

[0002] For online service platforms, such as online travel service platforms, it is necessary to ensure that the images displayed on the webpage have a certain aesthetic quality in order to improve the quality of the webpage and enhance the user's browsing and usage experience.

[0003] Currently, multi-scale spatial pooling features are mainly used to predict the aesthetic quality of images, which has the following problems:

[0004] Scoring problems typically use regression methods, which consider the numerical order of scores. However, classification is currently used to address image aesthetic quality scoring, which loses the order relationship between labels. In classification, categories are independent of each other, and the lack of order relationship can lead to significant misclassification errors.

[0005] 2) Since labels have no order relationship for classification and are artificially discretized into multiple categories, even if the largest category is reasonable and correct, the overall output distribution is still not reasonable; and if the expected score is used, the score is not scale invariant and the output of the sub-category cannot be combined.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0007] In view of this, the present invention provides a method, device, electronic device and storage medium for evaluating the aesthetic quality of images, which strongly constrain the scale-invariant relationship between classification scores through label multi-scale classification loss, making the output distribution more reasonable, and solving the problem of missing sequential relationship between categories when using classification to process regression problems, and the problem of unreasonable distribution of classification output.

[0008] According to one aspect of the present invention, a method for evaluating image aesthetic quality is provided, comprising: performing target detection on an initial image to obtain a target image; inputting the target image into an aesthetic quality evaluation model to obtain a distribution of predicted values ​​corresponding to multiple aesthetic quality labels output by multiple output nodes of the aesthetic quality evaluation model; wherein the aesthetic quality evaluation model is trained based on a multi-scale classification loss, the multi-scale classification loss including prediction losses for each label group formed by classifying the multiple aesthetic quality labels at different granularities; and calculating a mathematical expectation based on the predicted value distribution to obtain an aesthetic quality evaluation result for the target image.

[0009] In some embodiments, the multiple aesthetic quality labels include N+1 aesthetic quality labels with increasing label values; the multiple aesthetic quality labels are classified at different granularities, including classifying the N+1 aesthetic quality labels into multiple classifications with different numbers of groupings, each classification classifying the N+1 aesthetic quality labels in ascending order of the label values ​​based on a current number of groupings.

[0010] In some embodiments, the respective label groups include a fine-grained label group formed by classifying at least the N+1 aesthetic quality labels into groups equal to N+1, and a coarse-grained label group formed by classifying at least the N+1 aesthetic quality labels into groups less than the minimum number of groupings of the fine-grained label group; each fine-grained label group uses a bulldozer distance loss function to calculate the prediction loss of the predicted value distribution of each sample image based on the fine-grained label group relative to the true label of the sample image; each coarse-grained label group uses a focal loss function to calculate the prediction loss of the predicted value distribution of each sample image based on the coarse-grained label group relative to the true label of the sample image; each coarse-grained label group also uses a center loss function to calculate the prediction loss of the features of each sample image in the coarse-grained label group relative to the feature center of all sample images in the coarse-grained label group.

[0011] In some embodiments, N+1=11; the fine-grained label group includes an 11-category label group with a grouping number equal to 11 and a 5-category label group with a grouping number equal to 5; the coarse-grained label group includes a 3-category label group with a grouping number equal to 3 and two 2-category label groups with a grouping number equal to 2, and the classification methods of the two 2-category label groups are different.

[0012] In some embodiments, for a current sample image in a current fine-grained label group, the formula of the bulldozer distance loss function is:

[0013] loss EMD =∑(CDF j (SoftMax(Y))-CDF j (OneHot(label))) 2 ;

[0014] Where Y is the predicted value distribution of the current sample image based on the current fine-grained label group, SoftMax(·) represents the softmax operation; label is the true label of the current sample image, OneHot(·) represents one hot encoding; CDF j (·) represents the jth element in the probability distribution;

[0015] For the current sample image, the formula of the focus loss function is:

[0016]

[0017] Among them, α label is the balance factor, α label Based on the sample image ratio of the current coarse-grained label group in the sample image set, γ is a concentration parameter with a value of 2; M is the number of groups of the current coarse-grained label group, y n is the predicted value corresponding to the mth group in the predicted value distribution of the current sample image based on the current coarse-grained label group, y label is the true label of the current sample image;

[0018] For the n'th coarse-grained label group, the formula of the center loss function is:

[0019]

[0020] Among them, K n’ is the number of sample images of the n'th coarse-grained label group, x k is the feature of the kth sample image of the n'th coarse-grained label group, c n′ is the feature center of all sample images of the n'th coarse-grained label group.

[0021] In some embodiments, the multi-scale classification loss performs a weighted summation of the prediction losses of the respective label groups.

[0022] In some embodiments, performing target detection on the initial image to obtain the target image includes: performing target detection based on the target object on the initial image to obtain a target detection frame; performing geometric expansion on the target detection frame, and intercepting the initial target image from the initial image based on the geometrically expanded target detection frame; and adjusting the size of the initial target image to the input size of the aesthetic quality assessment model to obtain the target image.

[0023] In some embodiments, the aesthetic quality evaluation model is constructed with the efficiency network EfficientNet as the backbone network.

[0024] In some embodiments, calculating the mathematical expectation based on the predicted value distribution includes: performing a SoftMax operation on the predicted value distribution to obtain a probability distribution corresponding to the multiple aesthetic quality labels; and calculating the mathematical expectation of the probability distribution.

[0025] According to one aspect of the present invention, there is provided an image aesthetic quality assessment apparatus, comprising: a target image acquisition module, configured to perform target detection on an initial image to acquire a target image; an aesthetic quality prediction module, configured to input the target image into an aesthetic quality assessment model to obtain a distribution of predicted values ​​corresponding to multiple aesthetic quality labels output by multiple output nodes of the aesthetic quality assessment model; wherein the aesthetic quality assessment model is trained based on a multi-scale classification loss, the multi-scale classification loss comprising prediction losses for each label group formed by classifying the multiple aesthetic quality labels at different granularities; and an assessment result generation module, configured to calculate a mathematical expectation based on the predicted value distribution to obtain an aesthetic quality assessment result for the target image.

[0026] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing executable instructions; wherein when the executable instructions are executed by the processor, the image aesthetic quality evaluation method as described in any of the above embodiments is implemented.

[0027] According to one aspect of the present invention, a computer-readable storage medium is provided for storing a program, wherein when the program is executed by a processor, the method for evaluating image aesthetic quality as described in any of the above embodiments is implemented.

[0028] The beneficial effects of the present invention compared with the prior art include at least:

[0029] The present invention uses a feature fusion network structure and the idea of ​​classification to deal with regression problems. It strongly constrains the scale-invariant relationship between classification scores through label multi-scale classification loss, making the output distribution more reasonable, and solving the problem of missing order relationship between categories when using classification to deal with regression problems, as well as the problem of unreasonable classification output distribution.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0032] Figure 1 A schematic diagram showing the steps of a method for evaluating image aesthetic quality according to an embodiment of the present invention is shown;

[0033] Figure 2A schematic diagram showing the distribution of predicted values ​​based on various tag groups in one embodiment of the present invention;

[0034] Figure 3 A schematic diagram illustrating training an aesthetic quality assessment model based on label multi-scale classification loss in one embodiment of the present invention is shown;

[0035] Figure 4 A schematic diagram of modules of an image aesthetic quality evaluation device according to an embodiment of the present invention is shown;

[0036] Figure 5 A schematic structural diagram of an electronic device in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to make this disclosure thorough and complete and to fully convey the concepts of the example embodiments to those skilled in the art.

[0038] The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0039] In addition, the processes shown in the accompanying drawings are only exemplary and do not necessarily include all steps. For example, some steps can be decomposed, some steps can be combined or partially combined, and the order of actual execution may change according to actual circumstances. The words "first", "second" and similar terms used in the specific description do not indicate any order, quantity or importance, but are only used to distinguish different components. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in different embodiments can be combined with each other.

[0040] Figure 1 The main steps of the image aesthetic quality evaluation method in one embodiment are shown, referring to Figure 1 As shown, the image aesthetic quality evaluation method of this embodiment includes:

[0041] Step S110: performing target detection on the initial image to obtain a target image.

[0042] During object detection, the initial image is subjected to target object-based object detection to obtain a target detection frame. The target object can be a face, a building, a landscape, food, or other objects. For example, in one scenario, an online travel service platform needs to perform aesthetic quality evaluation on landscape images to select high-quality landscape images for display on scenery-related web pages. In another scenario, an online travel service platform needs to perform aesthetic quality evaluation on facial images to select high-quality facial images for display on person-related web pages. Object detection can be implemented using existing methods, and this invention does not limit this.

[0043] After obtaining the target detection frame, it is geometrically expanded to include more useful target object information. Furthermore, an initial target image is captured from the initial image based on the geometrically expanded target detection frame. This initial target image is then resized to the input size of the aesthetic quality assessment model to obtain the target image.

[0044] In step S120, the target image is input into an aesthetic quality assessment model to obtain a distribution of predicted values ​​corresponding to multiple aesthetic quality labels output by multiple output nodes of the aesthetic quality assessment model. The aesthetic quality assessment model is trained based on a multi-scale classification loss, which includes prediction losses for each label group formed by classifying the multiple aesthetic quality labels at different granularities.

[0045] The aesthetic quality evaluation model can be constructed using the EfficientNet backbone network. EfficientNet is an existing network model, so we will not elaborate on it here.

[0046] Each aesthetic quality label has a corresponding label value, and the output layer of the aesthetic quality assessment model has multiple output nodes corresponding to the multiple aesthetic quality labels. For example, in one embodiment, there are N+1 output nodes, each corresponding to the N+1 aesthetic quality labels, each of the N+1 aesthetic quality labels has an integer label value ranging from 0 to N, and each of the N+1 output nodes outputs a distribution of predicted values ​​corresponding to 0 to N.

[0047] During the training process of the aesthetic quality assessment model, a multi-scale classification loss is constructed to express the scale invariance of labels and constrain the accuracy of the original classification. When classifying aesthetic quality labels, the aesthetic quality labels are mainly divided into fine-grained label groups and coarse-grained label groups. To improve the efficiency of multi-scale label classification, the same label group combinations are avoided during classification. When constructing the multi-scale classification loss, the prediction loss of fine-grained label groups can be calculated using EMDLoss (bulldozer distance loss function), while the prediction loss of coarse-grained label groups can be calculated using FocalLoss (focal loss function) and CenterLoss (center loss function). The specific loss function will be explained in detail below in conjunction with the model training process.

[0048] The use of multi-scale classification loss can approximate the sequential relationship between classification categories by adding label scale invariance constraints, while avoiding classification errors with excessive gaps, thereby making the classification output distribution more reasonable.

[0049] Step S130 , calculating the mathematical expectation according to the predicted value distribution to obtain the aesthetic quality evaluation result of the target image.

[0050] When calculating the mathematical expectation based on the predicted value distribution, a SoftMax operation is performed on the predicted value distribution to obtain the probability distribution corresponding to multiple aesthetic quality labels, and then the mathematical expectation of the probability distribution is calculated.

[0051] By post-processing the predicted values ​​of N+1 output nodes and performing a SoftMax operation on the N+1 predicted values ​​representing 0 to N points, the predicted probability distribution is obtained. Based on the predicted probability distribution representing 0 to N points, the mathematical expectation is calculated to obtain the final aesthetic quality evaluation result.

[0052] The above-mentioned image aesthetic quality evaluation method uses a feature fusion network structure and classification ideas to deal with regression problems. It strongly constrains the scale-invariant relationship between classification scores through label multi-scale classification loss, making the output distribution more reasonable and solving the problem of missing order relationship between categories and unreasonable classification output distribution when using classification to deal with regression problems. The use of a label multi-scale-based image aesthetic quality evaluation method can make multi-classification image aesthetic quality evaluation applicable to fewer types of classifications, suitable for simplified image aesthetic quality evaluation, reduce the demand for data volume for a large number of categories, and at the same time use the original fine classification output distribution and indicators for better results.

[0053] The following describes the training process of the aesthetic quality evaluation model, mainly the loss function.

[0054] First, a sample image set is constructed.

[0055] In this embodiment, initial images can be obtained from the database of an online travel service platform. Each initial image is detected using an object detection algorithm (e.g., a face detection algorithm) to obtain M face frames. The M face frames are expanded by 0.4 times in both the horizontal and vertical directions to obtain expanded face frames. Based on the area corresponding to the expanded face frames, facial images are cropped from the corresponding initial images and resized to the model input size (e.g., 224×224).

[0056] For each acquired facial image, a label value corresponding to a specific aesthetic quality label is annotated as the true label of the facial image. In this embodiment, N+1 (e.g., 11) aesthetic quality labels are set, each with an integer label value ranging from 0 to N (i.e., 10). The annotation added to each facial image is a specific label value between 0 and N. Thus, a set of facial images with true labels is obtained as a sample image set. The sample image set contains sample images of each category (i.e., each true label corresponds to each aesthetic quality label).

[0057] Secondly, construct an aesthetic quality evaluation model.

[0058] In this example, EfficientNet was selected as the backbone network to extract facial image features and predict the aesthetic quality score of facial images. Furthermore, the aesthetic quality assessment model was configured with N+1 output nodes, each outputting a distribution of predicted values ​​from 0 to N.

[0059] Again, construct a multi-scale classification loss.

[0060] In this embodiment, as described above, N+1 aesthetic quality labels with increasing label values ​​are included; the aesthetic quality labels are classified at different granularities, specifically including multiple classifications with different numbers of groupings for the N+1 aesthetic quality labels, each classification classifying the N+1 aesthetic quality labels based on a current number of groupings in ascending order of label values.

[0061] Classification typically involves a classification method that groups the initial N+1 aesthetic quality labels into separate groups, as well as other classification methods that aggregate and classify the N+1 aesthetic quality labels at varying granularities. Consequently, the resulting label groups include a fine-grained label group formed by grouping at least the N+1 aesthetic quality labels into groups equal to N+1, and a coarse-grained label group formed by grouping at least the N+1 aesthetic quality labels into groups less than the minimum number of groupings in the fine-grained label group. For each fine-grained label group, the Earthmoving Distance Loss (EMD Loss) function is used to calculate the prediction loss of the distribution of predicted values ​​for each sample image based on the fine-grained label group relative to the true label value of the sample image. For each coarse-grained label group, the Focal Loss function is used to calculate the prediction loss of the distribution of predicted values ​​for each sample image based on the coarse-grained label group relative to the true label of the sample image. Furthermore, the Center Loss function is used to calculate the prediction loss of the features of each sample image in the coarse-grained label group relative to the feature centers of all sample images in the coarse-grained label group.

[0062] In a specific embodiment, N+1=11, the fine-grained label group includes an 11-category label group with a grouping number equal to 11 and a 5-category label group with a grouping number equal to 5; the coarse-grained label group includes a 3-category label group with a grouping number equal to 3 and two 2-category label groups with a grouping number equal to 2, and the two 2-category label groups have different classification methods.

[0063] For example, the 11 aesthetic quality labels are in ascending order of label value: l0 (0 points), l1 (1 point), l2 (2 points), l3 (3 points), l4 (4 points), l5 (5 points), l6 (6 points), l7 (7 points), l8 (8 points), l9 (9 points), l10 (10 points), l11 (10 points), l12 (10 points), l13 (10 points), l14 (10 points), l15 (10 points), l16 (10 points), l17 (10 points), l18 (10 points), l19 (10 points), l20 (10 points), l21 (10 points), l22 (10 points), l23 (10 points), l24 (10 points), l25 (10 points), l26 (10 points), l27 (10 points), l28 (10 points), l29 (10 points), l30 (10 points), l31 (10 points), l32 (10 points), l33 (10 points), l34 (10 points), l35 (10 points), l36 (10 points), l37 (10 points), l38 (10 points), l39 (10 points), l40 (10 points), l41 (10 points), l42 (10 points), l43 (10 points), l44 (10 points), l45 10 (10 points). In the 11-category label group, each aesthetic quality label is grouped separately. The 5 label groups in the 5-category label group are: l0 and l1, l2 and l3, l4 and l5 and l6, l7 and l8, l9 and l 10 The three label groups of the 3-category label group are: l0 and l1 and l2, l4 and l5 and l6, l8 and l9 and l 10 ; l3 and l7 at the group boundary are eliminated and do not participate in the grouping. The two label groups of the first two-class label group (two-class one-label group) are: l0 and l1 and l2 and l3 and l4, l6 and l7 and l8 and l9 and l 10 ; l5, which is at the group boundary, is eliminated and does not participate in the grouping. The two label groups of the second 2-class label group (2-class 2-label group) are: l0 and l1 and l2 and l3 and l4 and l5 and l6, l8 and l9 and l 10; l7 at the group boundary is eliminated and does not participate in the grouping.

[0064] Thus, in this embodiment, two types of fine-grained label groups and three types of coarse-grained label groups are formed. The two types of fine-grained label groups are 11 label groups of the 11-category label group and 5 label groups of the 5-category label group. The three types of coarse-grained label groups are 3 label groups of the 3-category label group, 2 label groups of the 2-category 1 label group, and 2 label groups of the 2-category 2 label group. In other embodiments, aesthetic quality labels can be classified in other ways that combine fine and coarse granularity.

[0065] Based on the above embodiment, Figure 2 The predicted value distribution based on each label group is shown. Figure 2 As shown, the predicted value distribution 210 based on the 11 classification label groups is composed of 11 output nodes (C0, C1, C2, C3, C4, C5, C6, C7, C8, C9, C 10 ) outputs the initial predicted value distribution (each predicted value is represented by an output node C), which contains 11 predicted values ​​corresponding to 11 aesthetic quality labels. Based on the predicted value distribution 220 of the 5-category label group, the initial predicted value distribution is added according to the 5 groups to form 5 predicted values ​​corresponding to the 5 groups, namely: C0+C1, ​​C2+C3, C4+C5+C6, C7+C8, C9+C10. Based on the predicted value distribution 230 of the 3-category label group, the initial predicted value distribution is added according to the 3 groups (excluding C3 and C7 at the group boundary and not included in the loss), forming 3 predicted values ​​corresponding to the 3 groups, namely: C0+C1+C2, C4+C5+C6, C8+C9+C10. Based on the predicted value distribution 240 for the 2-class, 1-label group, the initial predicted value distribution is summed up according to the first two-grouping method (excluding C5 at the group boundary and not included in the loss), forming two predicted values ​​corresponding to the first two-grouping method: C0+C1+C2+C3+C4, and C6+C7+C8+C9+C10. Based on the predicted value distribution 250 for the 2-class, 2-label group, the initial predicted value distribution is summed up according to the second two-grouping method (excluding C7 at the group boundary and not included in the loss), forming two predicted values ​​corresponding to the second two-grouping method: C0+C1+C2+C3+C4+C5+C6, and C8+C9+C10.

[0066] Therefore, to account for the scale invariance of labels, the aforementioned multi-scale classification loss is constructed to constrain the accuracy of the original classification. Based on the original 11-class classification, by aggregating the output results, a 5-class classification, a 3-class classification, and two 2-class classification losses are calculated to constrain the output probability distribution of the 11-class classification. The practical significance of this combination is that, taking the 5-class classification as an example, it is equivalent to classifying the scores into five categories: low, lower-middle, middle, upper-middle, and high. Its main purpose is to avoid excessive errors when misclassifying, essentially simulating the effect of ordered regression.

[0067] Figure 3 The following figure shows the scenario of training the aesthetic quality assessment model based on the label multi-scale classification loss. Figure 3 As shown in the figure, the sample image is input into the Backbone network, and the output node outputs the initial predicted value distribution. The prediction loss of each label group is calculated according to the above five classifications. Among them, Loss11 (i.e., the predicted value distribution of the fine-grained label group based on 11 categories) and Loss5 (i.e., the predicted value distribution of the fine-grained label group based on 5 categories) use EMD Loss to calculate the prediction loss; Loss3 (i.e., the predicted value distribution of the coarse-grained label group based on 3 categories), Loss2_1 (i.e., the predicted value distribution of the coarse-grained label group based on 2 categories 1), and Loss2_2 (i.e., the predicted value distribution of the coarse-grained label group based on 2 categories 2) use FocalLoss to calculate the prediction loss, and Center Loss is added to improve the intra-class aggregation.

[0068] When calculating the loss, for a current sample image in a current fine-grained label group, the formula for EMD Loss is:

[0069] loss EMD =∑(CDF j (SoftMax(Y))-CDF j (OneHot(label))) 2 ;

[0070] Where Y is the predicted value distribution of the current sample image based on the current fine-grained label group, SoftMax(·) represents the softmax operation, which converts the predicted value distribution into a probability distribution; label is the true label of the current sample image, OneHot(·) represents onehot encoding, which can convert the true label into a probability distribution based on the current fine-grained label group; CDF j (·) represents the jth element in the probability distribution.

[0071] For a current sample image in a current coarse-grained label group, the formula of FocalLoss is:

[0072]

[0073] Among them, α label is the balance factor, α label The value is determined based on the proportion of sample images in the current coarse-grained label group in the sample image set. γ is a concentration parameter with a value of 2. M is the number of groups in the current coarse-grained label group, and y m is the predicted value corresponding to the mth group in the predicted value distribution of the current sample image based on the current coarse-grained label group, y label is the true label of the current sample image.

[0074] For the n'th coarse-grained label group, the formula of the center loss function is:

[0075]

[0076] Among them, K n’ is the number of sample images of the n'th coarse-grained label group, x k is the feature of the kth sample image of the n'th coarse-grained label group, c n′ is the feature center of all sample images in the n'th coarse-grained label group. The features of the sample images are output by the feature extraction layer of the aesthetic quality assessment model.

[0077] Finally, the multi-scale classification loss is a weighted sum of the prediction losses of each label group. total =loss 11 +w5loss5+w3loss3+w 2_1 loss 2_1 +w 2_2 loss 2_2 , where w5, w3, w 2_1 and w 2_2 It is a weight factor that can be set as needed to balance the losses of each part.

[0078] Finally, the sample image set is used to iteratively train the aesthetic quality evaluation model based on the constructed multi-scale classification loss until the model converges and the multi-scale classification loss reaches the threshold.

[0079] In summary, the image aesthetic quality evaluation method of the present invention uses a feature fusion network structure and the concept of classification to deal with regression problems. It strongly constrains the scale-invariant relationship between classification scores through label multi-scale classification loss, making the output distribution more reasonable. It solves the problem of missing order relationship between categories when using classification to deal with regression problems and the problem of unreasonable distribution of classification output.

[0080] Embodiments of the present invention also provide an image aesthetic quality assessment device that can be used to implement the image aesthetic quality assessment method described in any of the above embodiments. The features and principles of the image aesthetic quality assessment method described in any of the above embodiments can be applied to the following embodiments of the image aesthetic quality assessment device. In the following embodiments of the image aesthetic quality assessment device, the features and principles of image aesthetic quality assessment that have already been explained will not be repeated.

[0081] Figure 4 FIG. 1 shows the main modules of an image aesthetic quality evaluation device in one embodiment. Figure 4 As shown, the image aesthetic quality assessment device 400 of this embodiment includes: a target image acquisition module 410, which is used to perform target detection on an initial image to obtain a target image; an aesthetic quality prediction module 420, which is used to input the target image into an aesthetic quality assessment model to obtain a distribution of predicted values ​​corresponding to multiple aesthetic quality labels output by multiple output nodes of the aesthetic quality assessment model; wherein the aesthetic quality assessment model is trained based on a multi-scale classification loss, and the multi-scale classification loss includes prediction losses for each label group formed by classifying multiple aesthetic quality labels at different granularities; and an evaluation result generation module 430, which is used to calculate a mathematical expectation based on the predicted value distribution to obtain an aesthetic quality assessment result of the target image.

[0082] Furthermore, the image aesthetic quality evaluation device 400 may also include modules for implementing other process steps of the above-mentioned image aesthetic quality evaluation method embodiments. The specific principles of each module can refer to the description of the above-mentioned image aesthetic quality evaluation method embodiments, and will not be repeated here.

[0083] The image aesthetic quality evaluation device of the present invention can use the feature fusion network structure and the classification concept to deal with regression problems. It strongly constrains the scale-invariant relationship between classification scores through the label multi-scale classification loss, making the output distribution more reasonable, and solving the problem of missing sequential relationship between categories when using classification to deal with regression problems, and the problem of unreasonable classification output distribution.

[0084] An embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores executable instructions, and when the executable instructions are executed by the processor, the image aesthetic quality evaluation method described in any of the above embodiments is implemented.

[0085] The electronic device of the present invention can use the network structure of feature fusion and the concept of classification to deal with regression problems. It strongly constrains the scale-invariant relationship between classification scores through label multi-scale classification loss, making the output distribution more reasonable, and solving the problem of missing sequential relationship between categories when using classification to deal with regression problems, and the problem of unreasonable classification output distribution.

[0086] Figure 5 is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. It should be understood that Figure 5 The modules are merely schematically shown; these modules may be virtual software modules or actual hardware modules; the merging and splitting of these modules and the addition of other modules are all within the protection scope of the present invention.

[0087] like Figure 5 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), and a display unit 640.

[0088] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of the image aesthetic quality evaluation method described in any of the above embodiments. For example, the processing unit 610 can perform the following steps: Figure 1 Steps shown.

[0089] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0090] The storage unit 620 may also include a program / utility 6204 having one or more program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0091] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0092] The electronic device 600 can also communicate with one or more external devices 700, which may be one or more of a keyboard, a pointing device, a Bluetooth device, or the like. These external devices 700 enable a user to interact with the electronic device 600. The electronic device 600 can also communicate with one or more other computing devices, including a router and a modem. This communication can be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0093] Embodiments of the present invention further provide a computer-readable storage medium for storing a program that, when executed, implements the image aesthetic quality assessment method described in any of the above embodiments. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the image aesthetic quality assessment method described in any of the above embodiments.

[0094] When the storage medium of the present invention is executed, it can use the network structure of feature fusion and the idea of ​​classification to process regression problems, and strongly constrain the scale-invariant relationship between classification scores through label multi-scale classification loss, so that the output distribution is more reasonable, and solve the problem of missing order relationship between categories when using classification to process regression problems, and the problem of unreasonable distribution of classification output.

[0095] The program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0096] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media include, but are not limited to, an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0097] The readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0098] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device, such as via the Internet using an Internet service provider.

[0099] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for evaluating image aesthetic quality, characterized in that: include: Perform target detection on the initial image to obtain the target image; Inputting the target image into an aesthetic quality evaluation model to obtain a distribution of predicted values ​​corresponding to a plurality of aesthetic quality labels output by a plurality of output nodes of the aesthetic quality evaluation model; The aesthetic quality evaluation model is trained based on a multi-scale classification loss, wherein the multi-scale classification loss includes prediction losses for each label group formed by classifying the multiple aesthetic quality labels at different granularities. The multiple aesthetic quality labels include N+1 aesthetic quality labels with increasing label values, and the classification of the multiple aesthetic quality labels at different granularities includes classifying the N+1 aesthetic quality labels into multiple classifications with different numbers of groupings, each classification classifying the N+1 aesthetic quality labels in ascending order of the label values ​​based on a current number of groupings. Wherein, each label group includes a fine-grained label group formed by classifying at least the N+1 aesthetic quality labels into groups equal to N+1, and a coarse-grained label group formed by classifying at least the N+1 aesthetic quality labels into groups less than the minimum number of groups of the fine-grained label group; each fine-grained label group uses a bulldozer distance loss function to calculate the prediction loss of the predicted value distribution of each sample image based on the fine-grained label group relative to the true label of the sample image; each coarse-grained label group uses a focal loss function to calculate the prediction loss of the predicted value distribution of each sample image based on the coarse-grained label group relative to the true label of the sample image; each coarse-grained label group also uses a center loss function to calculate the prediction loss of the features of each sample image in the coarse-grained label group relative to the feature center of all sample images in the coarse-grained label group; For a current sample image in a current fine-grained label group, the formula of the bulldozer distance loss function is: loss EMD =∑(CDF j (SoftMax(Y))-CDF j (OneHot(label))) 2 ; Where Y is the predicted value distribution of the current sample image based on the current fine-grained label group, SoftMax·) represents the softmax operation; label is the true label of the current sample image, OneHot(·) represents one hot encoding; CDF j (·) represents the jth element in the probability distribution; For a current sample image in a current coarse-grained label group, the formula of the focal loss function is: Among them, α label is the balance factor, α label Based on the sample image ratio of the current coarse-grained label group in the sample image set, γ is a concentration parameter with a value of 2; M is the number of groups of the current coarse-grained label group, y m is the predicted value corresponding to the mth group in the predicted value distribution of the current sample image based on the current coarse-grained label group, y label is the true label of the current sample image; Among them, for the n'th coarse-grained label group, the formula of the center loss function is: Among them, K n’ is the number of sample images of the n'th coarse-grained label group, x k is the feature of the kth sample image of the n'th coarse-grained label group, c n' is the feature center of all sample images of the n'th coarse-grained label group; wherein the multi-scale classification loss performs a weighted summation of the prediction losses of the respective label groups; and A mathematical expectation is calculated based on the predicted value distribution to obtain an aesthetic quality evaluation result of the target image.

2. The image aesthetic quality evaluation method according to claim 1, wherein: N+1＝11； The fine-grained label group includes an 11-category label group with a grouping number equal to 11 and a 5-category label group with a grouping number equal to 5; The coarse-grained label group includes a three-category label group with a grouping number equal to 3 and two two-category label groups with a grouping number equal to 2, and the two two-category label groups have different classification methods.

3. The image aesthetic quality evaluation method according to claim 1, wherein: The performing target detection on the initial image to obtain the target image includes: Performing target detection based on the target object on the initial image to obtain a target detection frame; Performing geometric expansion on the target detection frame, and intercepting an initial target image from the initial image based on the geometrically expanded target detection frame; The size of the initial target image is adjusted to the input size of the aesthetic quality assessment model to obtain the target image.

4. The image aesthetic quality evaluation method according to claim 1, wherein: The aesthetic quality evaluation model is constructed with the efficiency network EfficientNet as the backbone network.

5. The image aesthetic quality evaluation method according to claim 1, wherein: Calculating a mathematical expectation based on the predicted value distribution includes: Performing a SoftMax operation on the predicted value distribution to obtain a probability distribution corresponding to the multiple aesthetic quality labels; and Compute the mathematical expectation of the probability distribution.

6. An image aesthetic quality evaluation device, characterized in that: include: A target image acquisition module is used to perform target detection on the initial image and acquire the target image; an aesthetic quality prediction module, configured to input the target image into an aesthetic quality assessment model and obtain a distribution of predicted values ​​corresponding to a plurality of aesthetic quality labels output by a plurality of output nodes of the aesthetic quality assessment model; wherein the aesthetic quality assessment model is trained based on a multi-scale classification loss, wherein the multi-scale classification loss includes prediction losses for respective label groups formed by classifying the plurality of aesthetic quality labels at different granularities; the plurality of aesthetic quality labels includes N+1 aesthetic quality labels with increasing label values, and the plurality of aesthetic quality labels are classified at different granularities, including multiple classifications of the N+1 aesthetic quality labels with different numbers of groupings, each classification classifying the N+1 aesthetic quality labels in ascending order of the label values ​​based on a current number of groupings; Wherein, each label group includes a fine-grained label group formed by classifying at least the N+1 aesthetic quality labels into groups equal to N+1, and a coarse-grained label group formed by classifying at least the N+1 aesthetic quality labels into groups less than the minimum number of groups of the fine-grained label group; each fine-grained label group uses a bulldozer distance loss function to calculate the prediction loss of the predicted value distribution of each sample image based on the fine-grained label group relative to the true label of the sample image; each coarse-grained label group uses a focal loss function to calculate the prediction loss of the predicted value distribution of each sample image based on the coarse-grained label group relative to the true label of the sample image; each coarse-grained label group also uses a center loss function to calculate the prediction loss of the features of each sample image in the coarse-grained label group relative to the feature center of all sample images in the coarse-grained label group; For a current sample image in a current fine-grained label group, the formula of the bulldozer distance loss function is: loss EMD =∑(CDF j (SoftMax(Y))-CDF j (OneHot(label))) 2 ; Where Y is the predicted value distribution of the current sample image based on the current fine-grained label group, SoftMax(·) represents the softmax operation; label is the true label of the current sample image, OneHot(·) represents one hot encoding; CDF j (·) represents the jth element in the probability distribution; For a current sample image in a current coarse-grained label group, the formula of the focal loss function is: Among them, α label is the balance factor, α label Based on the sample image ratio of the current coarse-grained label group in the sample image set, γ is a concentration parameter with a value of 2; M is the number of groups of the current coarse-grained label group, y m is the predicted value corresponding to the mth group in the predicted value distribution of the current sample image based on the current coarse-grained label group, y label is the true label of the current sample image; Among them, for the n'th coarse-grained label group, the formula of the center loss function is: Among them, K n’ is the number of sample images of the n'th coarse-grained label group, x k is the feature of the kth sample image of the n'th coarse-grained label group, c n' is the feature center of all sample images of the n'th coarse-grained label group; wherein the multi-scale classification loss performs a weighted summation of the prediction losses of the respective label groups; and An evaluation result generation module is used to calculate the mathematical expectation according to the predicted value distribution to obtain the aesthetic quality evaluation result of the target image.

7. An electronic device, characterized in that: include: a processor; a memory storing executable instructions; Wherein, when the executable instruction is executed by the processor, the image aesthetic quality evaluation method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the image aesthetic quality evaluation method according to any one of claims 1 to 5 is implemented.

Images