A series photo selection method and system based on multi-view aesthetics

Abstract

The disclosure provides a series of photo selection method and system based on multi-view aesthetics, relates to the field of aesthetic quality evaluation, and the method comprises the following steps: acquiring image data; performing multi-view feature extraction on the image data, extracting artificial features and deep features of the series of images, obtaining node relationships between the extracted multi-views, constructing a multi-view aesthetic graph structure based on the node relationships; inputting the constructed multi-view aesthetic graph structure into a graph convolutional neural network, performing graph convolution, and refining feature maps through message passing between nodes; introducing an adaptive weight attention mechanism to compress and excite the input multi-view features, obtain the correlation between the multi-view features and highlight the common features; migrating each feature parameter obtained above to a twin network, inputting a series of images into the twin network, passing through three fully connected layers, obtaining the comparison result of the images, and obtaining the final series of image selection results.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of aesthetic quality evaluation, and in particular to a series photo selection method and system based on multi-view aesthetics. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] Series photo selection is closely related to people's life, and its influence reaches every aspect, such as micro-difference detection, aesthetic preference learning, multi-view integration, etc. Accurate series photo selection can assist in quickly recommending photos, making it convenient for people to publish the best photos on social media platforms; in addition, in the field of aesthetic quality evaluation, by providing data such as aesthetic quality scores of photos, images can be reasonably and efficiently screened; at the same time, it also provides great convenience for our daily life.

[0004] Current series photo selection methods mainly include methods based on manual features and methods based on deep convolutional features. The method based on manual features is inevitably affected by the fuzziness of certain photography or artistic rules, and since the input image is reshaped by the deep learning model to adapt to the model, the original aesthetics of the image may be destroyed. In addition, the traditional method based on deep convolutional features only extracts deep convolutional features, and often gives similar scores to similar images, which is not enough for ranking. Generally speaking, these deep methods ignore the saturation and color histogram, which is not conducive to reflecting subtle aesthetic changes and cannot better distinguish images. SUMMARY

[0005] In order to solve the above problems, the present disclosure proposes a series photo selection method and system based on multi-view aesthetics, which uses multi-view features, combines graph neural networks to construct the relationship between multi-view features, and proposes a twin network to select the best one from a group of almost identical photos, thereby improving the accuracy of series photo selection.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions:

[0007] A series photo selection method based on multi-view aesthetics, comprising:

[0008] Obtaining image data and performing preprocessing;

[0009] Performing multi-view feature extraction on the image data, extracting manual features and deep features of the series images, constructing a multi-view aesthetic graph structure based on the node relationship between the extracted multi-views;

[0010] The constructed multi-view aesthetic graph structure is input into a graph convolutional neural network, graph convolution is performed, and a feature map is refined through message passing between nodes; an attention mechanism with adaptive weights is introduced, multi-view features are compressed and excited, the correlation between the multi-view features is obtained and common features are highlighted, each feature parameter obtained is migrated to a twin network, a series of images is input into the twin network, three fully connected layers are passed, a comparison result of the images is obtained, and a final series of image selection results are obtained.

[0011] According to some embodiments, the present disclosure adopts the technical scheme as follows:

[0012] A series of photo selection system based on multi-view aesthetics, comprising:

[0013] An image acquisition module is configured to acquire image data and perform preprocessing.

[0014] A feature extraction module is configured to perform multi-view feature extraction on the image data, extract artificial features and deep features of the series of images, obtain node relationships between the multi-view features, and construct a multi-view aesthetic graph structure based on the node relationships.

[0015] An image selection module is configured to input the constructed multi-view aesthetic graph structure into a graph convolutional neural network, perform graph convolution, refine a feature map through message passing between nodes, introduce an attention mechanism with adaptive weights, compress and excite input multi-view features, obtain the correlation between the multi-view features and highlight common features, migrate each feature parameter obtained to a twin network, input a series of images into the twin network, pass three fully connected layers, obtain a comparison result of the images, and obtain a final series of image selection results.

[0016] According to some embodiments, the present disclosure adopts the technical scheme as follows:

[0017] A non-transitory computer-readable storage medium is configured to store computer instructions, which are executed by a processor to implement the series of photo selection method based on multi-view aesthetics.

[0018] According to some embodiments, the present disclosure adopts the technical scheme as follows:

[0019] An electronic device comprises a processor, a memory, and a computer program; the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the series of photo selection method based on multi-view aesthetics.

[0020] Compared with the prior art, the beneficial effects of the present disclosure are:

[0021] The present disclosure proposes a series photo selection method based on multi-view aesthetics, which fully considers the combination of multiple depths and artificial feature views, and can effectively reflect subtle aesthetic changes. In order to enhance the multi-view correlation, a multi-view graph learning network is first proposed to construct the spatial relationship between feature views, and a more comprehensive image representation is obtained. Secondly, in order to enhance the consistency information between views, different views are also merged into latent representations through self-attention units with adaptive weights. Finally, a twin network is proposed to select the best photo from a series of almost identical photos, further improving the prediction ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of the present disclosure, are intended to provide further understanding of the present disclosure, and the schematic embodiments of the present disclosure and the description thereof are intended to explain the present disclosure, and do not constitute improper limitations on the present disclosure.

[0023] Figure 1 The flowchart of the series photo selection method based on multi-view aesthetics provided by the embodiments of the present disclosure is shown in the figure.

[0024] Figure 2 The model framework diagram of the series photo selection method based on multi-view aesthetics provided by the embodiments of the present disclosure is shown in the figure.

[0025] Figure 3 The schematic diagram of the twin network provided by the embodiments of the present disclosure is shown in the figure. DETAILED DESCRIPTION

[0026] The present disclosure will be further described below in combination with the drawings and embodiments.

[0027] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present disclosure belongs.

[0028] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0029] Embodiment 1

[0030] In an embodiment of the present disclosure, a series photo selection method based on multi-view aesthetics is provided, comprising:

[0031] Step one: obtain image data and pre-process;

[0032] Step two: multi-view feature extraction is performed on the image data, artificial features and deep features of the series of images are extracted, node relationships between the extracted multi-views are obtained, and a multi-view co-occurrence graph structure is constructed based on the node relationships;

[0033] Step three: input the constructed multi-view co-occurrence graph structure into a graph convolutional neural network, perform graph convolution, and refine the feature map through message passing between nodes; introduce an adaptive weight attention mechanism to compress and excite the input multi-view features, obtain the correlation between the multi-view features and highlight the common features; migrate each feature parameter obtained above to a twin network, input a series of images into the twin network, pass through three fully connected layers, obtain the comparison result of the images, and obtain the final series image selection result.

[0034] As an embodiment, the image data includes a plurality of series of different photos and co-occurrence comparison relationships between photos in the same series, and includes the following data:

[0035]

[0036] wherein D represents a data set, respectively represent the ith image and the jth image of the kth series, represents the co-occurrence comparison relationship between the ith image and the jth image of the kth series, represents is better is better, otherwise

[0037] Then, the image is pre-processed, including image standardization processing.

[0038] The image standardization used is as follows:

[0039]

[0040] wherein x s represents the result of image standardization, x represents the original image (understood as a certain image in the data set), μ represents the mean of the image, and ξ represents the standard deviation of the image.

[0041] As an embodiment, the present disclosure uses multi-view features to construct relationships between multi-view features combined with a graph neural network. Finally, a twin network is proposed to select the best one from a set of almost identical photos, which improves the accuracy of series photo selection. The specific method is as follows:

[0042] Given an image x, first, a feature extraction unit is established to extract multiple views. After inputting the image x into this unit, the view matrix V = [v 1 , v 2 ,..., v l ] ∈ R l×C , where l and C represent the number of feature views and feature dimensions. These models include but are not limited to: artificial feature extractors, deep learning feature extractors, etc.

[0043] On the multi-view feature extraction unit, for the image, its corresponding artificial features, deep features need to be extracted. For this purpose, the corresponding artificial feature extractor F m , deep feature extractor F d are designed. At the same time, in order to model the relationship between views, a graph G related to aesthetics is constructed based on node representation.

[0044] In the multi-view aesthetic graph structure, nodes are associated with each other. First, a fully connected undirected graph is constructed in the coordinate space, which includes nodes, edges connecting nodes, and adjacent matrices describing edge weights. The deep feature view is taken as the center node of the graph, and the correlation between multi-views is used to obtain the weight of the edge. That is:

[0045] Considering that l nodes are associated with node representation V, first, a fully connected undirected graph G = (V, E, A) is constructed in the coordinate space. Here, G is composed of its nodes V, V is the set of edges connecting nodes E and the adjacent matrix A describing the edge weight. Since the features obtained by deep learning are much more effective in aesthetic evaluation than hand-designed features and other general features, the deep feature view is taken as the center node of the graph. Then the correlation between views is calculated to obtain the weight of the edge.

[0046] The content correlation between multi-views is calculated using cosine similarity, and the similarity matrix is calculated to obtain the affinity matrix, also known as the similarity matrix, which is used to organize the mutual similarity between a set of data points.

[0047] Specifically, the cosine similarity is used to measure the content relationship between views. After calculating the pairwise similarity matrix, the affinity matrix A ∈ R l×l , each row of the matrix is normalized using the softmax function:

[0048]

[0049] Where exp(·) is the exponential function. A threshold λ is set to determine which edges are retained to exclude the negative effects caused by excessive feature differences.

[0050] As Figure 2In the graph convolutional neural network, for the input multi-view graph, the feature map can be refined through message passing between nodes by performing graph convolution. The two-layer graph convolutional network is used to encode the interaction information, and the latent graph interaction embedding node is constructed.

[0051] Assuming that the two-layer graph convolutional network is used to encode the interaction information, the latent graph interaction embedding F G ∈R n×d The node can be constructed as follows:

[0052]

[0053]

[0054] Where, and are the hierarchical trainable weights of the control node encoder. V is the node feature vector v i is the matrix, and σ is the ReLU activation function, is the adjacency matrix of the undirected graph G, and I is the identity matrix. is a diagonal matrix defined as

[0055] The view matrix is converted into a multi-view matrix, and in the adaptive weight attention mechanism, the conversion process of the multi-view feature mapping is established. Given the input multi-view feature, first, the feature map is squeezed in parallel along the spatial axis using global average and maximum pooling, then the features are summed element by element, and the attention matrix is obtained using the Sigmoid operation. Finally, the multi-view feature and the channel attention matrix are combined to obtain the channel refined feature.

[0056] The self-attention mechanism can focus on important information in a large amount of information and ignore unimportant information. The role of this mechanism is to aggregate the feature map, or in other words, to aggregate the feature map.

[0057] On the adaptive weight self-attention mechanism, for the input multi-view feature, the correlation between the multi-view features can be obtained and the important common features can be highlighted by compressing and exciting the features.

[0058] After processing, the view matrix V = [v 1 , v 2 ,..., v l ] ∈ R l×C is converted into a feature F ∈ R l×C . The self-attention unit can map the multi-view feature F ∈ R l×C to a feature map F G ∈ R M×CThe conversion process of the feature dimension is realized by the mapping matrix. The feature of l*C dimension is mapped to the feature of M*C dimension. The purpose is to ensure the uniformity of the dimension, otherwise it cannot be calculated.

[0059] Given the input feature F, the self-attention unit first uses the global average pool and the maximum pool to squeeze the feature map along the spatial axis in parallel, and then sums the features element by element. Finally, the Sigmoid operation is used to obtain the F attention matrix.

[0060] a = δ (W2 (W1 (F Avg +F Max ))),

[0061] Where a ∈ R 1×C , δ (·) represents the Sigmoid nonlinear function. Finally, the multi-view feature F is combined with the channel attention matrix a to obtain the channel refined feature F S .

[0062]

[0063] Where represents element-wise multiplication. In this way, F S can be regarded as an interactive feature selector for enhancing the common information between features for the next quality evaluation.

[0064] As an embodiment, the specific method of model training is as follows:

[0065] The final series photo selection model contains an artificial feature extractor F m , a deep feature extractor F d , a graph constructor, a graph convolutional neural network, and a self-attention mechanism. The parameters of the deep feature extractor F d and the graph convolutional neural network are migrated to the final series photo selection model, and then the whole model is trained.

[0066] The twin network structure is as follows: Figure 3 Given a pair of images (x i , x j ) of the same series, the network MVNet designed in the present disclosure can obtain the feature representation i of the image x j and the feature representation of the image x i respectively. After three fully connected layers, the comparison result f (x j , x ) of the image pair can be obtained.

[0067] Where the loss function selects the designed error loss function as follows:

[0068]

[0069] The optimizer selects a stochastic gradient descent optimizer, and in the final test phase, the output of the model is de-normalized to obtain the final series of photo selection results.

[0070] Embodiment 2

[0071] In an embodiment of the present disclosure, a series of photo selection system based on multi-view aesthetics is provided, comprising:

[0072] An image acquisition module is configured to acquire image data and perform preprocessing;

[0073] A feature extraction module is configured to perform multi-view feature extraction on the image data, extract artificial features and deep features of the series of images, acquire node relationships between the multi-views based on the node relationships, and construct a multi-view aesthetic graph structure based on the node relationships;

[0074] An image selection module is configured to input the constructed multi-view aesthetic graph structure into a graph convolutional neural network, perform graph convolution, refine feature maps through message passing between nodes, introduce an adaptive weight attention mechanism to compress and excite the input multi-view features, acquire correlations between the multi-view features and highlight common features, migrate each feature parameter obtained above to a twin network, input a series of images into the twin network, pass through three fully connected layers to obtain a comparison result of the images, and obtain a final series of image selection results.

[0075] Embodiment 3

[0076] In an embodiment of the present disclosure, a non-transitory computer readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the series of photo selection method based on multi-view aesthetics.

[0077] Embodiment 4

[0078] In an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the series of photo selection method based on multi-view aesthetics.

[0079] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 2. The flowchart and / or block diagram in the drawings Figure 1 2. Means for performing the function specified by the flowchart block or blocks in 2. Steps for performing the function specified by the flowchart block or blocks in

[0080] 2. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 2. The flowchart and / or block diagram in the drawings Figure 1 2. Steps for performing the function specified by the flowchart block or blocks in 2. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0081] 2. The flowchart and / or block diagram in the drawings​​​​​​

Claims

1. A method for selecting a series of photos based on multi-view aesthetics, the method comprising: The method comprises the following steps: obtaining image data and performing preprocessing; extracting artificial features and depth features of the series of images from the image data, obtaining node relationships between the multiple views based on the extracted features, and constructing a multi-view aesthetic graph structure based on the node relationships; in the multi-view aesthetic graph structure, nodes are associated with each other, a fully connected graph is first constructed in a coordinate space, which includes nodes, edges connecting the nodes, and an adjacency matrix describing the edge weights, the depth feature view is taken as the center node of the graph, and the correlation between the multiple views is used to obtain the edge weights; inputting the constructed multi-view aesthetic graph structure into a graph convolutional neural network, performing graph convolution, refining the feature map through message passing between nodes, introducing an adaptive weight attention mechanism to compress and excite the input multi-view features, obtaining the correlation between the multi-view features and highlighting the common features, and migrating the obtained feature parameters to a twin network, inputting a series of images into the twin network, passing through three fully connected layers to obtain the comparison result of the images, and obtaining the final series image selection result.

2. The method of claim 1, wherein the series of photos is selected based on a multi-view aesthetic. Obtaining image data, including multiple series of different photos and aesthetic comparison relationships between photos in the same series.

3. The method of claim 1, wherein the series of photos is selected based on a multi-view aesthetic. The preprocessing process is a standardization process for images.

4. The method of claim 1, wherein the series of photos is selected based on a multi-view aesthetic. The content correlation between multiple views is calculated using cosine similarity, a similarity matrix is calculated, and an affinity matrix is obtained.

5. The method of claim 1, wherein the series of photos is selected based on a multi-view aesthetic. In the graph convolutional neural network, for the input multi-view aesthetic graph, the feature map can be refined through message passing between nodes by performing graph convolution, a two-layer graph convolutional network is used to encode the interaction information, and a latent graph interaction embedding node is constructed.

6. The method of claim 1, wherein the series of photos is selected based on a multi-view aesthetic. The view matrix is converted into a multi-view matrix, and in the adaptive weight attention mechanism, a conversion process of multi-view feature mapping is established, given the input multi-view features, first use global average and maximum pool to squeeze the feature map along the spatial axis in parallel, then sum the features element by element, use Sigmoid operation to obtain the attention matrix, and finally combine the multi-view features with the channel attention matrix to obtain the channel refined features.

7. A multi-view aesthetic based series photo selection system, characterized by, The method comprises the following steps: An image acquisition module is used to obtain image data and perform preprocessing; A feature extraction module is used to extract artificial features and depth features of the series of images from the image data, obtain node relationships between the multiple views based on the extracted features, and construct a multi-view aesthetic graph structure based on the node relationships; in the multi-view aesthetic graph structure, nodes are associated with each other, a fully connected graph is first constructed in a coordinate space, which includes nodes, edges connecting the nodes, and an adjacency matrix describing the edge weights, the depth feature view is taken as the center node of the graph, and the correlation between the multiple views is used to obtain the edge weights; The image selection module is used for inputting the constructed multi-view aesthetic graph structure into a graph convolutional neural network, performing graph convolution, refining a feature map through message passing between nodes, introducing an attention mechanism with adaptive weights, compressing and exciting the input multi-view features, obtaining the correlation between the multi-view features and highlighting common features, migrating the obtained feature parameters to a twin network, inputting a series of images into the twin network, obtaining a comparison result of the images through three fully connected layers, and obtaining a final series of image selection results.

8. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium is used to store computer instructions, and the computer instructions are executed by a processor to implement the series photo selection method based on multi-view aesthetics according to any one of claims 1-6.

9. An electronic device, comprising: The series photo selection method based on multi-view aesthetics comprises: A processor, a memory and a computer program; wherein the processor is connected with the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the series photo selection method based on multi-view aesthetics according to any one of claims 1-6.

Images