Image aesthetic analysis method and device, computer equipment and storage medium

Abstract

The invention discloses an image aesthetics analysis method and device, computer equipment and a storage medium, relates to the technical field of artificial intelligence, is applied to an image aesthetics analysis scene related to financial services, and comprises the following steps: obtaining a target image, performing feature extraction on the target image by using a feature extractor, and generating an image data structure; performing feature encoding on the graph data structure by using an encoder to generate feature representation; performing image convolution processing on the feature representation by using an image convolution block based on an attention mechanism to generate a first feature; performing feature processing on the target image by using a convolutional network to generate a second feature; performing fusion processing on the first feature and the second feature to generate a fused feature; and based on the fusion features, generating an aesthetic evaluation result of the target image by using a decoder. According to the method, the problem that in the prior art, a current image aesthetics evaluation method based on deep learning is insufficient in accuracy and reliability can be solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and is applied to image aesthetic analysis scenarios related to financial services, and specifically to an image aesthetic analysis method, device, computer equipment, and non-volatile computer-readable storage medium. Background Art

[0002] At present, with the development of digital photography and image processing technology, image aesthetics assessment has become an important research direction in the field of computer vision. The goal of image aesthetics assessment is to simulate human subjective evaluation of image beauty through computational methods, so as to automatically evaluate the aesthetic quality of images. The aesthetic quality of images is usually affected by many factors, including but not limited to color and contrast, content layout, visual balance, exposure level and clarity.

[0003] In recent years, deep learning-based image aesthetic assessment methods have become mainstream due to their powerful automatic feature learning capabilities. Deep neural networks (DNNs), especially deep convolutional neural networks (CNNs), have shown excellent performance in image feature extraction. However, these deep learning-based image aesthetic assessment methods have some significant limitations. First, neural networks usually require the size of the input image to be fixed, which means that the image must be cropped, scaled, or padded before it is input into the network to adapt to the network's input size requirements; these preprocessing steps may cause distortion and deformation of the image composition, thereby affecting the neural network's learning of compositional features. Second, although traditional convolutional neural networks perform well in capturing local features, they are deficient in handling long-range dependencies; in image aesthetic assessment, long-range visual elements and compositional relationships are crucial for the judgment of overall beauty, and this limitation of traditional CNNs limits its performance. That is, in the prior art, current deep learning-based image aesthetic assessment methods are still deficient in accuracy and reliability.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide an image aesthetic analysis method, apparatus, computer equipment and non-volatile computer-readable storage medium that can be used in financial technology or other related fields, aiming to solve the problem in the prior art that the current image aesthetic evaluation method based on deep learning is still insufficient in accuracy and reliability.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An image aesthetic analysis method, comprising:

[0008] Acquire a target image, extract features from the target image using a feature extractor, and generate a graph data structure;

[0009] Using an encoder to perform feature encoding on the graph data structure to generate a feature representation;

[0010] Performing graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature;

[0011] Performing feature processing on the target image using a convolutional network to generate a second feature;

[0012] fusing the first feature with the second feature to generate a fused feature;

[0013] Based on the fused features, a decoder is used to generate an aesthetic evaluation result of the target image.

[0014] In a further technical solution, the image aesthetic analysis method, wherein the step of acquiring a target image and extracting features of the target image using a feature extractor to generate a graph data structure, comprises:

[0015] Pre-constructing an image aesthetic analysis interface, and obtaining a target image input by a target user through the image aesthetic analysis interface;

[0016] Preprocessing the target image to generate a processed image;

[0017] The processed image is input into a feature extractor built based on ImageNet for feature extraction to generate a graph data structure.

[0018] In a further technical solution, the image aesthetic analysis method, wherein the step of using an encoder to perform feature encoding on the graph data structure to generate a feature representation, comprises:

[0019] Inputting the graph data structure into an encoder for feature encoding to generate an initial result;

[0020] Perform feature optimization on the initial result to obtain feature representation.

[0021] In a further technical solution, the image aesthetic analysis method, wherein the step of performing graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature, comprises:

[0022] Input the feature representation into the graph convolution block based on the attention mechanism for spatial reasoning and information transfer to generate initial features;

[0023] The initial feature is enhanced to obtain a first feature.

[0024] In a further technical solution, the image aesthetic analysis method, wherein the step of performing feature processing on the target image using a convolutional network to generate a second feature, comprises:

[0025] Inputting the target image into a pre-designed convolutional network for feature processing, identifying and extracting multiple key visual features in the target image;

[0026] The key visual features are integrated to obtain a second feature.

[0027] In a further technical solution, the image aesthetic analysis method, wherein the step of fusing the first feature with the second feature to generate a fused feature, includes:

[0028] Performing an adjustment operation on the first feature and the second feature to ensure that the two are consistent in format and dimension;

[0029] The first feature and the second feature are fused using a preset fusion strategy to generate a fused feature.

[0030] In a further technical solution, the image aesthetic analysis method, wherein the step of generating the aesthetic evaluation result of the target image by using a decoder based on the fusion feature, comprises:

[0031] Inputting the fused features into a decoder to generate an aesthetic evaluation result of the target image;

[0032] The aesthetic evaluation result is output or displayed.

[0033] An image aesthetic analysis device, comprising:

[0034] A feature extraction module is used to obtain a target image, extract features from the target image using a feature extractor, and generate a graph data structure;

[0035] A feature encoding module, used for performing feature encoding on the graph data structure using an encoder to generate a feature representation;

[0036] A graph convolution processing module, used to perform graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature;

[0037] A feature processing module, used for performing feature processing on the target image using a convolutional network to generate a second feature;

[0038] A fusion processing module, used for fusing the first feature and the second feature to generate a fusion feature;

[0039] The result generation module is used to generate an aesthetic evaluation result of the target image based on the fusion feature using a decoder.

[0040] A computer device, wherein the computer device comprises at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, any of the above-mentioned image aesthetic analysis methods can be implemented.

[0043] A non-volatile computer-readable storage medium stores a computer program, wherein the computer program, when executed by at least one processor, can implement any of the above-mentioned image aesthetic analysis methods.

[0044] Compared with the prior art, the present invention provides an image aesthetic analysis method, device, computer equipment and non-volatile computer-readable storage medium, wherein the method includes: obtaining a target image, extracting features from the target image using a feature extractor to generate a graph data structure; encoding features from the graph data structure using an encoder to generate a feature representation; performing graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature; performing feature processing on the target image using a convolution network to generate a second feature; fusing the first feature with the second feature to generate a fused feature; and generating an aesthetic evaluation result of the target image using a decoder based on the fused feature. In this way, the method of the present invention can solve the problem in the prior art that the current image aesthetic evaluation method based on deep learning is still insufficient in accuracy and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a flow chart of an image aesthetic analysis method provided by an embodiment of the present invention.

[0047] Figure 2A schematic diagram of a program module of an image aesthetics analysis device provided by an embodiment of the present invention.

[0048] Figure 3 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] In the description of the present invention, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps therein is not limited and can be appropriately adjusted as needed.

[0051] Various non-limiting embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0052] See also Figure 1 The embodiment of the present invention provides an image aesthetic analysis method, wherein the method specifically comprises the following steps:

[0053] S100, acquiring a target image, and using a feature extractor to extract features from the target image to generate a graph data structure;

[0054] S200, using an encoder to perform feature encoding on the graph data structure to generate a feature representation;

[0055] S300, performing graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature;

[0056] S400, performing feature processing on the target image using a convolutional network to generate a second feature;

[0057] S500, fusing the first feature and the second feature to generate a fused feature;

[0058] S600: Based on the fusion features, generate an aesthetic evaluation result of the target image using a decoder.

[0059] Furthermore, the image aesthetic analysis method, wherein the step S100, obtaining a target image, extracting features from the target image using a feature extractor, and generating a graph data structure, specifically comprises the following steps:

[0060] Pre-constructing an image aesthetic analysis interface, and obtaining a target image input by a target user through the image aesthetic analysis interface;

[0061] Preprocessing the target image to generate a processed image;

[0062] The processed image is input into a feature extractor built based on ImageNet for feature extraction to generate a graph data structure.

[0063] In the specific implementation, in this embodiment, an image aesthetic analysis interface is pre-built, and a target image input by a target user is obtained through the image aesthetic analysis interface; the target image is pre-processed to generate a processed image; the processed image is input into a feature extractor built based on ImageNet to extract features and generate a graph data structure; wherein the specific implementation process of each of the above steps is roughly as follows:

[0064] 1. Build the User Interface

[0065] Design and implement a user-friendly image aesthetic analysis interface that should contain necessary functions such as an upload button, an image display area, a submit button, and an area for displaying analysis results; the interface can be built using front-end technologies such as HTML, CSS, and JavaScript, and ensure that it works properly on different devices and browsers;

[0066] 2. Image upload and acquisition

[0067] Integrate image upload functionality into the image aesthetic analysis interface, allowing target users to select the image files they want to analyze by clicking an upload button; use JavaScript and backend languages ​​(such as Python, Java, etc.) to handle file uploads and ensure that image files are correctly received and stored on the server;

[0068] 3. Image Preprocessing

[0069] Perform a series of preprocessing operations on the target image uploaded by the target user to prepare for subsequent feature extraction; the preprocessing steps may include adjusting the image size to meet the input requirements of the feature extractor, rotating and cropping the image to maintain the aspect ratio of the image, color correction to ensure color consistency, and possible denoising to improve image quality;

[0070] 4. Integration of feature extractors

[0071] Integrate a convolutional neural network (CNN) pre-trained on ImageNet as a feature extractor; this feature extractor can extract rich visual features from images; use a deep learning framework to load the pre-trained model and ensure that the model can receive pre-processed images (processed images) as input; ImageNet is a large visualization database used for visual object recognition software research;

[0072] 5. Generate graph data structure

[0073] The preprocessed target image (processed image) is input into the feature extractor to extract the features of the target image; these features may include local features (such as edges, textures) and global features (such as the overall layout of the image) of the target image; then, the extracted features are converted into a graph data structure, in which nodes represent different regions or features of the target image, and edges represent the relationship between these regions or features; this step can be implemented using a graph database or a custom data structure;

[0074] Through the above process, the program can extract the corresponding graph data structure from the target image uploaded by the target user.

[0075] Furthermore, the image aesthetic analysis method, wherein the step S200, using an encoder to perform feature encoding on the graph data structure to generate a feature representation, specifically comprises the steps of:

[0076] Inputting the graph data structure into an encoder for feature encoding to generate an initial result;

[0077] Perform feature optimization on the initial result to obtain feature representation.

[0078] In the specific implementation, in this embodiment, the graph data structure is input into the encoder for feature encoding to generate an initial result; the initial result is feature optimized to obtain a feature representation; wherein the specific implementation process of each of the above steps is roughly as follows:

[0079] 1. Input graph data structure to encoder

[0080] The obtained graph data structure is input into the encoder; the encoder can be a neural network designed specifically for graph structured data, such as a graph convolutional network (GCN), a graph attention network (GAT), or other graph neural network (GNN) architectures; the goal of the encoder is to encode the node features in the graph data structure into a more compact and abstract feature representation;

[0081] 2. Feature encoding generates initial results

[0082] The encoder processes the graph data structure through a series of graph convolutional layers or other graph neural network layers, each of which may include nonlinear activation functions and possible regularization techniques; these layers gradually extract and integrate the features of the nodes to generate the encoded representation of each node; finally, these encoded representations are integrated into the initial result;

[0083] 3. Feature Optimization

[0084] Initial results may need further refinement to increase their expressiveness and discrimination; this can be achieved in a variety of ways, including:

[0085] Feature selection: Select the most informative features from the initial results and remove redundant or irrelevant features;

[0086] Feature transformation: Applying techniques such as principal component analysis (PCA) or autoencoders for feature dimensionality reduction to reduce the dimensionality of features in the initial results and extract the most important information;

[0087] Regularization: Use techniques such as L1 or L2 regularization to reduce overfitting and improve the generalization ability of features in the initial results;

[0088] 5. Generate final feature representation

[0089] The optimized initial results are integrated into the final feature representation; the feature representation should be able to capture the key aesthetic attributes of the target image and provide useful information for subsequent aesthetic evaluation or other analysis tasks; the feature representation can be a fixed-length vector or a more complex data structure such as a graph or a tree;

[0090] Through the above process, the graph data structure can be effectively converted into useful feature representations to support image aesthetic evaluation or other related tasks.

[0091] Furthermore, the image aesthetic analysis method, wherein the step S300, using a graph convolution block based on an attention mechanism to perform graph convolution processing on the feature representation to generate a first feature, specifically comprises the steps of:

[0092] Input the feature representation into the graph convolution block based on the attention mechanism for spatial reasoning and information transfer to generate initial features;

[0093] The initial feature is enhanced to obtain a first feature.

[0094] In the specific implementation, in this embodiment, the feature representation is input into the graph convolution block based on the attention mechanism for spatial reasoning and information transfer to generate initial features; the initial features are feature enhanced to obtain the first features; wherein the specific implementation process of the above steps is roughly as follows:

[0095] 1. Constructing a graph convolution block based on the attention mechanism

[0096] Design a graph convolutional network module that includes an attention mechanism; this mechanism allows dynamically adjusting the attention paid to different nodes in the target image (i.e., different regions in the target image); this is usually achieved by learning the importance weights of each node to other nodes, which are learned during training;

[0097] 2. Input of feature representation

[0098] The feature representation obtained from the previous step is input into the graph convolution block based on the attention mechanism; the feature representation may include local features, texture, color, etc. of the target image;

[0099] 3. Perform spatial reasoning and information transfer

[0100] In the graph convolution block, the features of each node are updated through the spatial reasoning and information transfer process enhanced by the attention mechanism; this process involves combining the features of each node with the features of its neighboring nodes, while adjusting the contribution of each neighbor according to the attention weight; in this way, the new feature representation of each node not only contains its own information, but also incorporates the contextual information of the surrounding environment;

[0101] Of course, information transmission can be implemented based on the self-attention mechanism. Specifically, the introduction of the self-attention mechanism enables the information transmission process to process information from different neighbors more flexibly and differently. This mechanism is particularly suitable for scenarios where neighbor nodes contribute differently to the current node state update. For example, in the target image, the features of a certain area may have a greater impact on the overall aesthetic quality of the target image than the features of other areas. In summary, the self-attention mechanism of this embodiment is mainly used in the information transmission process to enhance the ability to identify and process the importance of different neighbor nodes. The information transmission method based on the self-attention mechanism can more effectively capture the key aesthetic features in the target image and generate more accurate aesthetic evaluation results of the target image.

[0102] 4. Generate initial features

[0103] After the attention-weighted graph convolution operation, each node will generate a set of new feature representations, which are called initial features. These initial features contain enhanced image region information and are more conducive to capturing the global and local aesthetic properties of the target image.

[0104] 5. Feature Enhancement

[0105] Enhance the initial features to further improve their expressiveness and discriminability; this may include applying nonlinear activation functions, normalizing the features, using batch normalization to reduce internal covariate shift, or other feature transformation techniques;

[0106] 6. Get the first feature

[0107] The enhanced initial features are integrated and optimized to form the final first features; the first features are the basis for aesthetic evaluation or other analysis tasks of the target image, and they should be able to effectively reflect the key visual and aesthetic attributes of the image;

[0108] Through the above process, the graph convolution block based on the attention mechanism can be effectively used to process the feature representation and generate the first feature that can accurately reflect the aesthetic quality of the target image.

[0109] Furthermore, the image aesthetic analysis method, wherein the step S400, using a convolutional network to perform feature processing on the target image to generate a second feature, specifically comprises the steps of:

[0110] Inputting the target image into a pre-designed convolutional network for feature processing, identifying and extracting multiple key visual features in the target image;

[0111] The key visual features are integrated to obtain a second feature.

[0112] In the specific implementation, in this embodiment, the target image is input into a pre-designed convolutional network for feature processing, and multiple key visual features in the target image are identified and extracted; each of the key visual features is integrated to obtain a second feature; wherein the specific implementation process of each of the above steps is roughly as follows:

[0113] 1. Design and select convolutional network architecture

[0114] Design or select a suitable convolutional neural network (CNN) architecture based on the task requirements; this network should be able to process various image features, such as edges, textures, shapes, etc.; commonly used network architectures include AlexNet, VGG, ResNet, etc.; when selecting or designing a network, the complexity of the task and the limitations of computing resources should be taken into account;

[0115] 2. Feature processing

[0116] The target image is input into a pre-designed convolutional network; the network extracts the features of the target image through multiple convolutional layers and pooling layers; in the convolutional layer, the filter (convolution kernel) slides on the target image to capture local features; the pooling layer is used to reduce the spatial dimension of the features while retaining important information;

[0117] 3. Identification and extraction of key visual features

[0118] At different levels of the convolutional network, information that can represent the key visual features of the target image is extracted; the lower levels may contain basic features such as edges and textures, while the higher levels may contain more complex semantic information; these key visual features can be identified and understood through techniques such as activation mapping and feature visualization;

[0119] 4. Feature Integration

[0120] Integrate the key visual features extracted from different layers of the network; this may involve operations such as feature fusion, feature selection, or feature transformation to generate a second feature that integrates multiple aspects of information; feature fusion can be achieved through splicing, weighted averaging, or more complex fusion strategies;

[0121] Through the above process, key visual features can be effectively extracted and integrated from the target image to obtain the second feature, providing support for further analysis and application of the target image.

[0122] Furthermore, the image aesthetic analysis method, wherein the step S500, fusing the first feature and the second feature to generate a fused feature, specifically comprises the steps of:

[0123] Performing an adjustment operation on the first feature and the second feature to ensure that the two are consistent in format and dimension;

[0124] The first feature and the second feature are fused using a preset fusion strategy to generate a fused feature.

[0125] In the specific implementation, in this embodiment, the first feature and the second feature are adjusted to ensure that the two are consistent in format and dimension; the first feature and the second feature are fused using a preset fusion strategy to generate a fused feature; wherein the specific implementation process of each of the above steps is roughly as follows:

[0126] 1. Feature format and dimensionality analysis

[0127] Before fusion, you first need to analyze the format and dimensions of the first and second features; this includes understanding their shapes, data types, and whether any conversions are required to ensure they can be merged; for example, if the features are multidimensional arrays, you may need to check whether they have the same number of dimensions and compatible sizes;

[0128] 2. Perform adjustment operations

[0129] Based on the results of the analysis, make necessary adjustments to the features; this may include:

[0130] Scaling features: If the value ranges of two sets of features are very different, you may need to normalize or standardize them so that they are on the same scale;

[0131] Adjust dimensions: If the dimensions of the features do not match, you may need to reshape them to have the same dimensions through interpolation, pooling, or convolution operations;

[0132] Feature selection: In some cases, it may be necessary to select the most relevant subset from each set of features to reduce redundancy and improve fusion effect;

[0133] 3. Design integration strategy

[0134] Determine how to fuse the adjusted features together; fusion strategies can be simple, such as direct concatenation, or more complex, such as using weighted averages or specific neural network layers (such as fully connected layers or convolutional layers) to learn how to best combine features;

[0135] 4. Implement fusion operations

[0136] Merge the first feature and the second feature into a fused feature according to the designed fusion strategy; this may involve writing corresponding code to implement feature merging, or using existing layers and functions in the deep learning framework;

[0137] When using a neural network for fusion, it may be necessary to train the network to learn the best way to fuse features; this usually involves defining a loss function and adjusting the network parameters through backpropagation and optimization algorithms;

[0138] Through the above process, it can be ensured that the first features and second features from different sources can be effectively integrated, providing a richer and more powerful data foundation for subsequent image analysis tasks.

[0139] Furthermore, the image aesthetic analysis method, wherein the step S600, based on the fusion feature, using a decoder to generate an aesthetic evaluation result of the target image, specifically comprises the steps of:

[0140] Inputting the fused features into a decoder to generate an aesthetic evaluation result of the target image;

[0141] The aesthetic evaluation result is output or displayed.

[0142] In specific implementation, in this embodiment, the fusion feature is input into a decoder to generate an aesthetic evaluation result of the target image; the aesthetic evaluation result is output or displayed; wherein the specific implementation process of each of the above steps is roughly as follows:

[0143] 1. Prepare the decoder model

[0144] In this process, you need to prepare or design a decoder model that can process the fused features and output aesthetic evaluation results; the decoder can be a fully connected neural network, a regression model, or any machine learning model suitable for such tasks; make sure the decoder has been properly trained to learn from the fused features and predict the aesthetic quality of the image;

[0145] 2. Input fusion features

[0146] Input the fused features generated in the previous step into the decoder; this usually involves formatting the feature data into the input format expected by the decoder model; data preprocessing, such as normalization or resizing the data, may be required to ensure that the features are compatible with the decoder's input layer;

[0147] 3. Generate aesthetic evaluation results

[0148] The decoder processes the fused features to generate an aesthetic evaluation result of the target image; this result is usually a numerical score that represents the aesthetic quality of the image; the decoder may contain multiple layers and activation functions to process complex features and output a continuous score;

[0149] 4. Post-processing scoring (optional)

[0150] In some cases, the raw output of the decoder may require further post-processing; for example, it may be necessary to normalize the ratings to a specific range (such as 0 to 100) or apply some form of calibration to ensure that the ratings are consistent with the standards of human raters;

[0151] 5. Output or display results

[0152] Present the aesthetic assessment results to the target users in an intuitive way; this may include displaying the scores on the image aesthetic analysis interface, generating detailed reports, or using visualization tools to present the results; ensure that the results are presented in a way that is clear and easy to understand for the target users;

[0153] Of course, the aesthetic evaluation results can also be output to a designated location as needed;

[0154] Through the above process, it can be ensured that the decoder can effectively process the fused features, generate accurate aesthetic evaluation results, and output or display the aesthetic evaluation results to the target users in a useful and intuitive manner.

[0155] From the above method embodiments, it can be seen that the image aesthetic analysis method provided by the present invention includes: obtaining a target image, using a feature extractor to extract features from the target image to generate a graph data structure; using an encoder to perform feature encoding on the graph data structure to generate a feature representation; using a graph convolution block based on an attention mechanism to perform graph convolution processing on the feature representation to generate a first feature; using a convolutional network to perform feature processing on the target image to generate a second feature; fusing the first feature with the second feature to generate a fused feature; based on the fused feature, using a decoder to generate an aesthetic evaluation result of the target image. In this way, the method of the present invention can solve the problem in the prior art that the current image aesthetic evaluation method based on deep learning is still insufficient in accuracy and reliability.

[0156] Specifically, the image aesthetic analysis method provided by the embodiment of the present invention can at least achieve the following beneficial technical effects:

[0157] 1. The method of the present invention introduces a graph convolutional network, which can solve the problem that ordinary convolutional networks cannot consider the impact of the horizontal and vertical ratios of images and the spatial layout on the aesthetic quality of images;

[0158] 2. The method of the present invention adopts the attention mechanism, which can adaptively assign weights to each area of ​​the image, and can obtain higher evaluation accuracy;

[0159] 3. The method of the present invention fuses the first feature obtained after graph convolution with the second feature obtained by the traditional convolutional network, which can more comprehensively capture the aesthetic attributes of the image, thereby improving the effect and reliability of image aesthetic evaluation.

[0160] Of course, the image aesthetic analysis method of the present invention can be applied in a variety of ways in the field of financial services. The following are some specific application scenarios:

[0161] 1. Personalized financial service recommendations: By analyzing the aesthetic features of pictures shared by customers on social media, financial institutions can better understand customers’ personal tastes and preferences, thereby providing personalized financial products and services;

[0162] 2. Investment decision support: In the field of art and collectibles investment, image aesthetic analysis can help investors evaluate the visual appeal and potential value of artworks as a reference for investment decisions;

[0163] 3. Risk management: During the credit approval process, image aesthetic analysis can be used to assess the appearance and maintenance condition of collateral (such as real estate, cars, etc.) as a factor in assessing its value and risk;

[0164] 4. Customer service experience optimization: Financial institutions can use image aesthetic analysis to optimize the visual effects of their websites, mobile applications and promotional materials, and improve the aesthetics of the user interface and user experience;

[0165] These application scenarios demonstrate the potential of the image aesthetic analysis method of the present invention in the field of financial services and are worthy of subsequent promotion and application.

[0166] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0167] Based on the above method embodiment, please refer to Figure 2 Another embodiment of the present invention further provides an image aesthetic analysis device, wherein the device specifically comprises:

[0168] A feature extraction module 11 is used to obtain a target image, extract features from the target image using a feature extractor, and generate a graph data structure;

[0169] A feature encoding module 12, used to perform feature encoding on the graph data structure using an encoder to generate a feature representation;

[0170] A graph convolution processing module 13, configured to perform graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature;

[0171] A feature processing module 14 is used to perform feature processing on the target image using a convolutional network to generate a second feature;

[0172] A fusion processing module 15, used for fusing the first feature and the second feature to generate a fusion feature;

[0173] The result generation module 16 is used to generate an aesthetic evaluation result of the target image based on the fusion feature using a decoder.

[0174] Furthermore, in the image aesthetics analysis device, the feature extraction module 11 specifically includes:

[0175] An image acquisition unit, used to pre-build an image aesthetic analysis interface, and acquire a target image input by a target user through the image aesthetic analysis interface;

[0176] A preprocessing unit, used for preprocessing the target image to generate a processed image;

[0177] The feature extraction unit is used to input the processed image into a feature extractor built based on ImageNet to extract features and generate a graph data structure.

[0178] Furthermore, in the image aesthetics analysis device, the feature encoding module 12 specifically includes:

[0179] A feature encoding unit, used for inputting the graph data structure into an encoder for feature encoding to generate an initial result;

[0180] The feature optimization unit is used to perform feature optimization on the initial result to obtain a feature representation.

[0181] Furthermore, in the image aesthetics analysis device, the graph convolution processing module 13 specifically includes:

[0182] A reasoning and transfer unit, used for inputting the feature representation into a graph convolution block based on an attention mechanism for spatial reasoning and information transfer to generate initial features;

[0183] The feature enhancement unit is used to perform feature enhancement on the initial feature to obtain a first feature.

[0184] Furthermore, in the image aesthetics analysis device, the feature processing module 14 specifically includes:

[0185] A feature processing unit, used to input the target image into a pre-designed convolutional network for feature processing, and identify and extract multiple key visual features in the target image;

[0186] The feature integration unit is used to integrate the key visual features to obtain a second feature.

[0187] Furthermore, in the image aesthetics analysis device, the fusion processing module 15 specifically includes:

[0188] an adjustment unit, configured to perform an adjustment operation on the first feature and the second feature to ensure that the two are consistent in format and dimension;

[0189] The fusion unit is used to fuse the first feature with the second feature by adopting a preset fusion strategy to generate a fusion feature.

[0190] Furthermore, in the image aesthetics analysis device, the result generation module 16 specifically includes:

[0191] A feature input unit, used for inputting the fused feature into a decoder to generate an aesthetic evaluation result of the target image;

[0192] An output or display unit is used to output or display the aesthetic evaluation result.

[0193] The specific implementation process and effects of the device embodiment of the present invention can be found in the above method embodiment, which will not be described in detail here.

[0194] It can be seen from the above device embodiments that the image aesthetic analysis device provided by the present invention includes: a feature extraction module for acquiring a target image, using a feature extractor to extract features from the target image, and generating a graph data structure; a feature encoding module for using an encoder to perform feature encoding on the graph data structure and generate a feature representation; a graph convolution processing module for performing graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature; a feature processing module for performing feature processing on the target image using a convolution network to generate a second feature; a fusion processing module for performing fusion processing on the first feature and the second feature to generate a fusion feature; and a result generation module for generating an aesthetic evaluation result of the target image based on the fusion feature using a decoder. In this way, the device of the present invention can solve the problem in the prior art that the current image aesthetic evaluation device based on deep learning is still insufficient in accuracy and reliability.

[0195] Specifically, the image aesthetics analysis device provided by the embodiment of the present invention can at least achieve the following beneficial technical effects:

[0196] 1. The device of the present invention introduces a graph convolutional network, which can solve the problem that ordinary convolutional networks cannot consider the impact of the horizontal and vertical ratios of images and the spatial layout on the aesthetic quality of images;

[0197] 2. The device of the present invention adopts an attention mechanism, which can adaptively assign weights to various regions of the image, thereby achieving higher evaluation accuracy;

[0198] 3. The device of the present invention fuses the first feature obtained after graph convolution with the second feature obtained by the traditional convolutional network, which can capture the aesthetic attributes of the image more comprehensively, thereby improving the effect and reliability of image aesthetic evaluation.

[0199] Of course, the image aesthetics analysis device of the present invention can be used in a variety of applications in the field of financial services. The following are some specific application scenarios:

[0200] 1. Personalized financial service recommendations: By analyzing the aesthetic features of pictures shared by customers on social media, financial institutions can better understand customers’ personal tastes and preferences, thereby providing personalized financial products and services;

[0201] 2. Investment decision support: In the field of art and collectibles investment, image aesthetic analysis can help investors evaluate the visual appeal and potential value of artworks as a reference for investment decisions;

[0202] 3. Risk management: During the credit approval process, image aesthetic analysis can be used to assess the appearance and maintenance condition of collateral (such as real estate, cars, etc.) as a factor in assessing its value and risk;

[0203] 4. Customer service experience optimization: Financial institutions can use image aesthetic analysis to optimize the visual effects of their websites, mobile applications and promotional materials, and improve the aesthetics of the user interface and user experience;

[0204] These application scenarios demonstrate the potential of the image aesthetic analysis device of the present invention in the field of financial services and are worthy of subsequent promotion and application.

[0205] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides a computer device, wherein the computer device 10 comprises:

[0206] memory 120 and one or more processors 110, Figure 3 A processor 110 is used as an example for the description. The processor 110 and the memory 120 may be connected via a communication bus or other means. Figure 3 The example of connecting via a communication bus is taken as an example.

[0207] The processor 110 is used to complete various control logics of the computer device 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP, or any other such configuration.

[0208] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the computer program corresponding to the image aesthetics analysis method in any of the above method embodiments of the present invention. The processor 110 executes various functional applications and data processing of the computer device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the image aesthetics analysis method in any of the above method embodiments.

[0209] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device or at least one function; the data storage area may store data created according to the use of the computer device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the computer device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0210] One or more units are stored in the memory 120, and when executed by one or more processors 110, the image aesthetic analysis method in any of the above method embodiments can be implemented, for example, the above described method can be implemented. Figure 1 The method comprises steps S100 to S600.

[0211] Those skilled in the art will understand that Figure 3 The hardware structure diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0212] Based on the above method embodiments, another embodiment of the present invention further provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the image aesthetic analysis method in any of the above method embodiments can be implemented, for example, the image aesthetic analysis method described above can be implemented. Figure 1 The method comprises steps S100 to S600.

[0213] As an example, the non-volatile storage medium can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable ROM (EEPROM), or a flash memory. Volatile memory can include a random access memory (RAM) as an external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synch link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0214] Based on the above method embodiments, another embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a processor, the image aesthetic analysis method in any of the above method embodiments can be implemented, for example, the above described Figure 1 The method comprises steps S100 to S600.

[0215] The embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0216] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can exist in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0217] Conditional language such as "can," "can," "might," or "may," unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey, among other things, that a particular embodiment can include (while other embodiments do not) a particular feature, element, and / or operation. Thus, such conditional language is also generally intended to imply that a feature, element, and / or operation is required for one or more embodiments anyway or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular embodiment.

[0218] It should be noted that if software tools or components other than those of the company appear in the embodiments of the present application, they are only used for example and do not represent actual use. The content described in this article, in this specification and in the drawings includes examples that can provide an image aesthetic analysis method, apparatus, computer device and non-volatile computer-readable storage medium. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it can be recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is obvious that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure, but all these various modifications should belong to the scope of protection of the claims attached to the present invention. In addition, or in the alternative, other embodiments of the present disclosure may be obvious from consideration of the present specification and the drawings and from the practice of the present disclosure as presented herein. It is intended that the examples set forth in the present specification and the drawings are considered to be illustrative and not restrictive in all respects. Although specific terms are used herein, they are used in a general and descriptive sense and are not used for limiting purposes.

Claims

1. A method for image aesthetic analysis, characterized in that: include: Acquire a target image, extract features from the target image using a feature extractor, and generate a graph data structure; Using an encoder to perform feature encoding on the graph data structure to generate a feature representation; Performing graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature; Performing feature processing on the target image using a convolutional network to generate a second feature; fusing the first feature with the second feature to generate a fused feature; Based on the fused features, a decoder is used to generate an aesthetic evaluation result of the target image.

2. The image aesthetic analysis method according to claim 1, characterized in that: The step of acquiring a target image, extracting features from the target image using a feature extractor, and generating a graph data structure includes: Pre-constructing an image aesthetic analysis interface, and obtaining a target image input by a target user through the image aesthetic analysis interface; Preprocessing the target image to generate a processed image; The processed image is input into a feature extractor built based on ImageNet for feature extraction to generate a graph data structure.

3. The image aesthetic analysis method according to claim 1, characterized in that: The step of using an encoder to perform feature encoding on the graph data structure to generate a feature representation includes: Inputting the graph data structure into an encoder for feature encoding to generate an initial result; Perform feature optimization on the initial result to obtain feature representation.

4. The image aesthetic analysis method according to claim 1, characterized in that: The step of performing graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature includes: Input the feature representation into the graph convolution block based on the attention mechanism for spatial reasoning and information transfer to generate initial features; The initial feature is enhanced to obtain a first feature.

5. The image aesthetic analysis method according to claim 1, characterized in that: The process of performing feature processing on the target image by using a convolutional network to generate a second feature includes: Inputting the target image into a pre-designed convolutional network for feature processing, identifying and extracting multiple key visual features in the target image; The key visual features are integrated to obtain a second feature.

6. The image aesthetic analysis method according to claim 1, characterized in that: The fusing the first feature and the second feature to generate a fused feature includes: Performing an adjustment operation on the first feature and the second feature to ensure that the two are consistent in format and dimension; The first feature and the second feature are fused using a preset fusion strategy to generate a fused feature.

7. The image aesthetic analysis method according to claim 1, characterized in that: The step of generating an aesthetic evaluation result of the target image by using a decoder based on the fusion feature comprises: Inputting the fused features into a decoder to generate an aesthetic evaluation result of the target image; The aesthetic evaluation result is output or displayed.

8. An image aesthetic analysis device, characterized in that: include: A feature extraction module is used to obtain a target image, extract features from the target image using a feature extractor, and generate a graph data structure; A feature encoding module, used for performing feature encoding on the graph data structure using an encoder to generate a feature representation; A graph convolution processing module, used to perform graph convolution processing on the feature representation using a graph convolution block based on an attention mechanism to generate a first feature; A feature processing module, used for performing feature processing on the target image using a convolutional network to generate a second feature; A fusion processing module, used for fusing the first feature and the second feature to generate a fusion feature; The result generation module is used to generate an aesthetic evaluation result of the target image based on the fusion feature using a decoder.

9. A computer device, characterized in that: The computer device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the image aesthetic analysis method according to any one of claims 1 to 7 can be implemented.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the image aesthetic analysis method according to any one of claims 1 to 7 can be implemented.

Images