An image color aesthetic evaluation method based on multi-scale feature fusion

Abstract

The present application relates to the technical field of image color aesthetic evaluation, and particularly to an image color aesthetic evaluation method based on multi-scale feature fusion, comprising the following steps: S1, converting image data of an input image into sequence data of the input image; S2, performing n times of feature extraction operations on the sequence data of the input image to obtain n multi-scale features, performing vector addition to obtain aggregated features F; S3, performing feature weighted fusion operation on the aggregated features F to obtain calibrated features; S4, using the calibrated features to perform image color aesthetic evaluation on the input image. In summary, color information in the image can be more comprehensively captured. Different color details can be given different degrees of attention according to learnable weight distribution, ensuring that key color information is given priority. This enables the model to have a more comprehensive understanding of color information in the entire image, thereby enhancing its overall perceptual ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image color aesthetic evaluation, and particularly to an image color aesthetic evaluation method based on multi-scale feature fusion. BACKGROUND

[0002] With the advancement of digital photography equipment and image processing technology, image aesthetics, especially the rendering of colors, has gradually become an important issue in image processing and evaluation. Existing photography equipment and image signal processors (ISPs) can compensate for insufficient lighting and other issues through algorithms to improve the visual effect of images. However, these adjustments often result in deviations in color reproduction, causing significant color differences in the same scene on different devices. Therefore, the rendering of colors not only affects image quality but also directly relates to the aesthetics of images, which has driven the development of the emerging field of image color aesthetic assessment (ICAA). ICAA is specifically designed to assess the aesthetic quality of image color or color preference, which involves evaluating key factors such as color harmony, color combination, and their overall impact on color perception. In the ICAA task, an important challenge is to establish a global perception of complex color combinations in images, so as to extract more comprehensive color aesthetic representations to guide the model to make accurate evaluations consistent with human subjective perspectives.

[0003] Traditional color aesthetic evaluation methods usually rely on simple pixel statistics or color histograms to evaluate color quality, but these methods have some limitations. Pixel statistics can calculate the distribution characteristics of each pixel color in the image, but cannot capture the overall aesthetic effect of color combinations. Similarly, color histograms reflect the color features of the image, but cannot express the subtle relationships and matching effects between different colors. These traditional methods are difficult to effectively capture human color aesthetic preferences, because color aesthetics often involve multi-dimensional perceptual experiences, not only including the presentation of colors themselves, but also the matching and global combination between colors. This multi-level perceptual experience requires more advanced models to achieve.

[0004] As disclosed in the Chinese patent document CN117315438A, a method, device and equipment for image color aesthetic evaluation based on interest points are disclosed, which include: dividing the original image color aesthetic evaluation dataset into a training set and a test set; performing data preprocessing on the training set to obtain a target training set, and performing data preprocessing on the test set to obtain a target test set; constructing an image color aesthetic evaluation model based on interest point allocation; training the image color aesthetic evaluation model based on interest point allocation according to the target training set to obtain a primary evaluation model, and adjusting the parameters of the primary evaluation model according to the target test set to obtain a target evaluation model; obtaining image color aesthetic data to be processed; inputting the image color aesthetic data to be processed into the target evaluation model to obtain an image color aesthetic evaluation result.

[0005] In the prior art method, there is a Transformer-based image color aesthetic evaluation method. This method adaptively allocates interest points using Delegate Transformer, and learns human color space segmentation behavior through a dedicated module to improve the accuracy of color aesthetic evaluation. However, this method mainly focuses on key color information in the image, resulting in some color information being ignored, thereby affecting the comprehensiveness and accuracy of the evaluation.

[0006] To overcome the limitations of traditional methods, it is necessary to optimize and improve image color aesthetic assessment (ICAA). SUMMARY

[0007] The present application is aimed at providing a technical solution to solve the above problems.

[0008] The present application provides an image color aesthetic evaluation method based on multi-scale feature fusion, comprising the following steps:

[0009] S1, converting the image data of the input image into sequence data of the input image;

[0010] S2, performing n times of feature extraction operation on the sequence data of the input image to obtain n multi-scale features, and performing vector addition to obtain aggregated features F;

[0011] S3, performing feature weighted fusion operation on the aggregated features F to obtain calibrated features ;

[0012] S4, using the calibrated features to perform image color aesthetic evaluation on the input image.

[0013] As a further scheme of the present application: in step S3, the feature weighted fusion operation is performed on the aggregated features F, specifically comprising the following steps:

[0014] S3.1, performing feature weighting operation on the aggregated feature F, so as to generate attention weight W;

[0015] S3.2, using the attention weight W to calibrate the multi-scale feature;

[0016] S3.3, performing fusion operation on the calibrated multi-scale feature, so as to obtain the calibration feature .

[0017] As a further scheme of the present application: in step S3.1, the feature weighting operation is performed on the aggregated feature F, specifically including the following steps: using global average pooling to calculate and process the aggregated feature F, so as to generate global feature descriptor; then mapping the global feature descriptor through a multi-layer perceptron, and converting the mapped data through a Softmax function, so as to generate the attention weight W.

[0018] As a further scheme of the present application: in step S3.2, the calibration operation is performed on the multi-scale feature, specifically including the following steps: using the corresponding nth attention weight W n to perform scalar multiplication on the nth multi-scale feature F n , so as to obtain the nth calibration feature .

[0019] As a further scheme of the present application: in step S3.3, the fusion operation is performed on the calibrated multi-scale feature, specifically including the following steps: performing vector addition on n calibration features , so as to obtain the calibration feature .

[0020] As a further scheme of the present application: in step S4, using the calibration feature to perform image color aesthetic evaluation on the input image, specifically including the following steps: using global average pooling and multi-layer perceptron to learn the mapping relationship between the calibration feature and the aesthetic score, and then using a regressor to convert the feature mapping relationship into the final aesthetic score.

[0021] As a further scheme of the present application: in step S2, the feature extraction operation is performed on the sequence data of the input image, specifically including the following steps: using the VMamba block to perform continuous data processing operation on the sequence data of the input image, and after multiple data processing, performing a feature extraction operation.

[0022] As a further scheme of the present application: in step S2, the sequence data of the input image is subjected to a feature extraction operation, specifically including the following steps: the sequence data of the input image is subjected to a data processing operation of every 4 VMamba blocks, that is, a feature extraction operation is performed once, and a multi-scale feature is generated, and after n times of feature extraction operation, n multi-scale features (F1, F2, …, Fn) are sequentially obtained. n ].

[0023] As a further scheme of the present application: in step S2, the sequence data of the input image is subjected to a data processing operation of every 4 VMamba blocks, that is, a feature extraction operation is performed once, and a multi-scale feature is generated, and after n times of feature extraction operation, n multi-scale features (F1, F2, …, Fn) are sequentially obtained.

[0024] S2.1, the sequence data is preprocessed using normalized data preprocessing;

[0025] S2.2, the preprocessed sequence data is processed using linear projection and activation function to obtain first data;

[0026] S2.3, the preprocessed sequence data is processed using linear projection, depth separable convolution and state space model in sequence to obtain second data;

[0027] S2.4, the first data and the second data are multiplied and combined element by element, then linearly projected, and then residual connected and combined with the sequence data without preprocessing.

[0028] As a further scheme of the present application: in steps S2.3 and S2.4, further including the following steps: the preprocessed sequence data is processed using linear projection, forward convolution and forward state space model to obtain third data; the preprocessed sequence data is processed using linear projection, backward convolution and backward state space model to obtain fourth data; the first data is multiplied and combined with the third data and the fourth data element by element, and the results of the two are subjected to vector addition processing, then linearly projected, and then residual connected and combined with the sequence data without preprocessing.

[0029] Compared with the prior art, the present application has the following advantages:

[0030] The present application proposes an image color aesthetic evaluation method based on multi-scale feature fusion, aiming to accurately and efficiently better perceive the color information in the image. The method includes 3 key components, the first component is based on the feature extraction network of Vision Mamba (VMamba), which is the core module for extracting multi-scale color features. The second component is the feature weighted fusion module (WFFM), which dynamically allocates learnable weights to multi-scale features to provide more accurate representation for aesthetic evaluation. The third component is a regressor that uses recalibrated features to predict aesthetics for aesthetic evaluation of image color.

[0031] The image color aesthetic evaluation method of the present application can more comprehensively capture color information in the image. Different color details can be given different degrees of attention according to the learnable weight distribution, ensuring that key color information is given priority. This method enables the model to have a more comprehensive understanding of color information in the entire image, thereby enhancing its overall perceptual ability.

[0032] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor under the premise of the drawings.

[0034] Figure 1 is a schematic block diagram of the overall process of the present application;

[0035] Figure 2 is a schematic block diagram of the feature weighted fusion operation of the present application;

[0036] Figure 3 is a schematic block diagram of the feature extraction operation of the present application;

[0037] Figure 4 is another schematic block diagram of the feature extraction operation of the present application;

[0038] Figure 5 is a schematic diagram of the model framework of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0040] Please refer to Figures 1 to 5 In the embodiments of the present application, an image color aesthetic evaluation method based on multi-scale feature fusion comprises the following steps:

[0041] S1, converting image data of an input image into sequence data of the input image;

[0042] S2, performing n times of feature extraction operations on the sequence data of the input image to obtain n multi-scale features, and performing vector addition to obtain aggregated features F;

[0043] S3, performing feature weighted fusion operation on the aggregated features F to obtain calibrated features .

[0044] S4, using the calibrated features to perform image color aesthetic evaluation on the input image.

[0045] Specifically, to overcome the limitations of traditional image color aesthetic evaluation methods, the present application explores the emerging basic model-state space model (Mamba) in the context of image color aesthetic assessment (ICAA). The Mamba model has strong ability to process complex sequence data, has a global receptive field, and has linear computational complexity when processing long sequence data, making it particularly suitable for capturing multi-level and global color information in images.

[0046] Therefore, the present application proposes an image color aesthetic evaluation method based on the Mamba model and multi-scale feature fusion, aiming to accurately and efficiently better aesthetically perceive color information in images. The model diagram of the technical solution embodiment of the present application is shown in Figure 5 The model architecture is composed of 3 key components. The first component is a feature extraction network based on Vision Mamba (VMamba), which is the core module for extracting multi-scale color features. The second component is a feature weighted fusion module (WFFM), which dynamically allocates learnable weights to multi-scale features to provide more accurate feature representation for aesthetic evaluation. The third component is a regressor which uses recalibrated features to predict aesthetics and perform aesthetic evaluation on image color.

[0047] In addition, it should be noted that the model referred to in the present application is an image data processing model made according to the image color aesthetic evaluation method provided by the present application.

[0048] As shown in Figure 1 Preferably, in step S1, the image data of the input image is converted into sequence data of the input image, specifically including the following steps: dividing the input image into a plurality of fixed-size image blocks, each image block being subjected to feature extraction by a convolutional layer to be converted into a fixed-dimension feature vector, linearly projecting the extracted feature vector to obtain a vector representation of the image block, and combining the vector representations of the plurality of image blocks to form the sequence data of the input image.

[0049] Specifically, the image data of the input image can be converted using Patch Embedding to obtain the sequence data of the input image. Patch Embedding is a process of dividing an image into a plurality of small blocks (patches) and converting each small block into a fixed-length vector representation. Its basic principles and processes include:

[0050] ‌Divide the image‌: First, divide the input image into a plurality of fixed-size patches. For example, if the image size is 224x224 pixels, the image can be divided into 16x16 pixel patches.

[0051] ‌Feature extraction‌: Feature extraction is performed on each patch by a convolutional layer, and the size and step of the convolutional kernel are usually the same as the size of the patch. In this way, each patch is converted into a fixed-dimension feature vector.

[0052] ‌Linear projection‌: Linear projection is performed on the extracted feature vector to obtain the final embedding representation. This step usually involves a fully connected layer that maps the dimension of the feature vector to a fixed dimension.

[0053] Secondly, Patch Embedding can convert image data into sequence data, so that the model made according to the method provided by the present application can directly process sequence data, thereby utilizing its powerful context understanding ability to process image data. In addition, Patch Embedding can also improve the generalization ability of the model, so that it can better process images of different input sizes and resolutions.

[0054] In another embodiment, preferably, in step S1, the position information of each image block in the input image is encoded to form a position vector, and the position vector is used for vector addition with the vector representation of the image block to form sequence data with position information.

[0055] Specifically, in the model, although each element in the sequence data can be processed in parallel, the relative position information between the elements cannot be directly perceived. Therefore, position embedding can be introduced to encode the position information into a vector and perform vector addition with the embedding vector of each element, thereby preserving the order information of each element and allowing the model to learn the relative importance of elements at different positions. Position Embedding is a prior art in the model for representing the position information of each element or Token in the sequence.

[0056] As shown in Figure 1 and Figure 3 , preferably, in step S2, a feature extraction operation is performed on the sequence data of the input image, specifically including the following steps: using a VMamba block to perform a continuous data processing operation on the sequence data of the input image, and after multiple data processing, performing a feature extraction operation. For example, the following scheme can be used: the sequence data of the input image is processed by the data processing operation of the continuous 4 VMamba blocks, that is, a feature extraction operation is performed once, and a multi-scale feature is generated. Continue the above steps until n feature extraction operations are performed, and n multi-scale features (F1, F2, …, Fn) are obtained in turn. n )。

[0057] Specifically, the present application uses a bidirectional Vision Mamba (VMamba) block as the core of the visual encoder for feature extraction, and can use multiple data processing to perform feature extraction at different scales to form n multi-scale features. Vision Mamba is a deep learning model for visual tasks, commonly used in tasks such as object detection and image classification in the field of computer vision. Its design concept is to improve processing speed and accuracy while reducing the demand for computing resources.

[0058] Secondly, multi-scale feature refers to extracting image features at different scales to capture different size structure and texture information, and obtaining different levels of information by changing the scale of the image. Features at different scales can provide different detailed information about the image, thereby more comprehensively and accurately describing the image. Multi-scale features can capture texture and structural features at different scales to improve the accuracy of image data.

[0059] As shown in Figure 3 , preferably, in step S2, the data processing operation on the sequence data of the input image using the VMamba block includes the following steps:

[0060] S2.1, pre-processing the sequence data using normalized data preprocessing;

[0061] S2.2, processing the pre-processed sequence data using linear projection and activation function to obtain first data;

[0062] S2.3, sequentially using linear projection, depth separable convolution and state space model to process the pre-processed sequence data to obtain second data;

[0063] S2.4, element-wise multiplication and merging of the first data and the second data, then linear projection, and then residual connection and merging with the sequence data without preprocessing.

[0064] Specifically, for each image block label in the input image sequence data, first, normalization data preprocessing (Normalization) is performed, and then it is split into two branches. The first branch processes the data through a linear layer and an activation function (SiLU); the second branch uses a linear layer, a depth separable convolution, and a state space model to process the data. The data results of the two branches are then combined by element-wise multiplication, and then combined by linear layer and residual connection. This design enables the VMamba block to effectively model the global visual context and enhance the spatial information perception ability.

[0065] Secondly, normalization (Normalization) is a data preprocessing technique aimed at scaling the numerical range of data to a specific scale, usually to eliminate the dimensional differences between different features, making the data more suitable for subsequent analysis and processing. It can significantly improve the training efficiency and performance of the model, providing strong support for the optimization and generalization ability of deep learning models.

[0066] Thirdly, the activation function (SiLU) combines the advantages of ReLU and Sigmoid functions, with smooth and non-monotonic characteristics, providing better numerical stability and gradient propagation. The state space model (State Space Model, SSM) is a statistical model used to describe time series data, mainly used to process dynamic systems and time-varying processes. It is a model used for sequence prediction tasks, which can solve the problem of traditional Transformer model's insensitivity to discrete deep latent features, thereby establishing a stronger logic for time series data.

[0067] In addition, the depth separable convolution refers to a Convld, i.e., a Linear Deformable Convolution (LDConv), which is an improved convolution operation aiming to solve the limitations of standard convolution and deformable convolution. Although deformable convolution solves the problem of fixed sampling, the number of parameters increases quadratically with the size of the convolution kernel, which is low in computational efficiency. LDConv overcomes these problems by giving the convolution kernel an arbitrary number of parameters and an arbitrary sampling shape, so that the number of parameters increases linearly. LDConv can replace the traditional convolution operation to improve network performance.

[0068] As shown in Figure 4 Preferably, in steps S2.3 and S2.4, the following steps are further included: using linear projection, forward convolution and forward state space model to process the pre-processed sequence data to obtain third data; using linear projection, backward convolution and backward state space model to process the pre-processed sequence data to obtain fourth data; using the first data to perform element-wise multiplication and merging with the third data and the fourth data respectively, and performing vector addition processing on the results of the two, and then performing linear projection, and then performing residual connection merging with the sequence data without preprocessing.

[0069] Specifically, forward convolution (forward convld) and backward convolution (backward convld) can also be introduced in the process of depth separable convolution. Forward convolution refers to the convolution operation of input data (such as sequence data of input images) with convolution kernels (filters) in the forward propagation, so as to generate feature maps (feature maps). Thus, the features of the input data are extracted through forward convolution, so that they can be processed by the subsequent layers of the network. Backward convolution is used in the backward propagation phase to calculate the gradient of the loss function and update the model parameters (such as convolution kernels). The calculation of backward convolution can be regarded as the "inverse operation" of the forward convolution process, and its purpose is to pass the gradient from the output feature map back to the input feature map and the convolution kernel. Thus, the optimized gradient is obtained through backward convolution, which is used to update the parameters and improve the learning of the model.

[0070] Secondly, Forward SSM (forward state space model) and Backward SSM (backward state space model) can also be introduced in the process of state space model. Forward state space model is mainly used to infer the current state or output based on historical data (previous state) from the current time. It usually processes new observations by merging them into previous states in a recursive manner. Backward state space model is used to derive the state from the future to the past under the condition that the complete observation sequence is known. It is usually used to improve the existing estimates.

[0071] In addition, in the state space model, an activation function (SiLU) and a 2D-Selective-Scan (SS2D) module can also be included. The SS2D module is to selectively scan specific regions of the input image instead of uniformly processing the entire image. It can more effectively utilize computing resources, focusing on areas that are more likely to contain targets or important features, thereby improving the efficiency and accuracy of processing the regions of interest in the image. For example, the SS2D module can expand image patches in four different directions: from the top left corner to the bottom right corner, from the bottom right corner to the top left corner, from the top right corner to the bottom left corner, and from the bottom left corner to the top right corner, creating four different sequences.

[0072] As shown in Figure 2 , preferably, in step S3, the aggregated features F are subjected to a feature weighted fusion operation, which specifically includes the following steps:

[0073] S3.1, performing a feature weighting operation on the aggregated features F to generate attention weights W;

[0074] S3.2, using the attention weights W to calibrate the multi-scale features;

[0075] S3.3, performing a fusion operation on the calibrated multi-scale features to obtain calibrated features .

[0076] Specifically, when predicting quality, the present application does not rely on a set of fully connected layers behind the last layer of features, but instead utilizes a set of rich multi-scale features to construct a global representation of color information. In order to fuse these multi-scale features, the present application carefully designs a WFFM module to assign learnable weights to color features of different scales (as shown in Figure 5 ). In this way, the model can fully perceive and integrate the advantages of multi-scale color features.

[0077] Secondly, for example, given multi-scale features , the element-wise sum of the features is calculated to obtain an aggregated feature , and the calculation formula # (1) is defined as:

[0078] wherein is the multi-scale feature.

[0079] In another embodiment, in step S3.1, the feature weighting operation is performed on the aggregated features F, specifically including the following steps: using global average pooling (GAP) to calculate the channel statistics of the aggregated features F across the spatial dimension, thereby generating a global feature descriptor; then mapping the global feature descriptor through a multi-layer perceptron (MLP), and converting the mapped data through a Softmax function, thereby generating attention weights W. In step S3.2, the calibration operation is performed on the multi-scale features, specifically including the following steps: performing scalar multiplication on the nth multi-scale feature F n using the corresponding nth attention weight W n to obtain the nth calibrated feature . In step S3.3, the fusion operation is performed on the calibrated multi-scale features, specifically including the following steps: performing vector addition on the n calibrated features to obtain the calibrated feature .

[0080] Specifically, the aggregated features F generate a global feature descriptor by combining information from the multi-scale stream. The global feature descriptor is a vector used to describe the entire image or feature map, which can capture the overall features of the input data. For example, the output channel number of the multi-layer perceptron (MLP) can be defined as n×c, where n is the number of features aggregated in the aggregated features F, and c is the number of channels. Through the Softmax function, the attention of each feature on the channel can be learned, thereby calibrating the features.

[0081] Secondly, for example, given multi-scale features , the calculation formula of the attention weight W is . Then the dynamically changing weight W can be used to adaptively recalibrate the multi-scale feature map.

[0082] Thirdly, the overall process of feature aggregation and recalibration is defined by formulas # (2) and # (3) as follows:

[0083] ,

[0084] wherein, is the multi-scale feature, is the attention weight.

[0085] In summary, the WFFM module forms a comprehensive perception of color information in the image by aggregating and dynamically selecting key global features.

[0086] In addition, (GAP) and (MLP) in the feature weighting fusion module (WFFM) are used to learn the weight of the feature calibration. Global average pooling (GAP) is a pooling operation applied after the last convolutional layer of a convolutional neural network, which takes the average value of each feature map as the overall representation of the feature. For example, for each feature map, GAP calculates the average of all elements to generate a single scalar value. Therefore, for a convolutional layer with multiple feature maps, GAP outputs a vector with the same dimension as the number of feature maps, which can then be used for classification or other tasks.

[0087] Multilayer perceptron (MLP) is a feedforward artificial neural network model that maps multiple input data sets to a single output data set. It is composed of multiple layers (input layer, one or more hidden layers and output layer), each containing multiple neurons (nodes) connected by weights. It can handle classification, regression and other tasks. With reasonable design and training, MLP can achieve good results in many tasks.

[0088] ‌Softmax function‌ is a mathematical function that converts a set of arbitrary real numbers into real numbers representing a probability distribution. Softmax function is a commonly used activation function that converts a set of arbitrary real numbers (such as the output of a neural network) into a probability distribution. Softmax function maps the results to (0, 1) through normalization operation, and the sum of all outputs is 1, so it is suitable for representing the probability of multiple categories. It is mainly used to convert the output of a neural network into an interpretable probability distribution. It emphasizes larger values and suppresses smaller values, allowing the model to better distinguish between categories.

[0089] In another embodiment, in step S4, the calibrated features are used to learn the mapping relationship between the input image and the aesthetic score The image color aesthetic evaluation of the input image includes the following steps: using global average pooling (GAP) and multilayer perceptron (MLP) to learn the calibration features The mapping relationship between the aesthetic score, and then using a regressor Convert the feature mapping relationship to the final aesthetic score.

[0090] Specifically, the recalibrated features are then used by a regressor to predict the color aesthetic score, defined as Here, (GAP), (MLP) are the mapping relationships between the learned calibrated features and the aesthetic scores, and the regressor (Regressor) Q is the last component of the model architecture, which mainly functions to map the extracted and fused features into the final aesthetic scores.

[0091] The application first adopts an emerging base model, namely the State Space Model (SSM) Mamba, as the backbone for extracting global features of images. This model has linear complexity in processing long sequence data and plays a significant role in the ICAA field. Through the application of this model, global information can be effectively extracted from input images, providing strong support for subsequent color aesthetic evaluation.

[0092] Subsequently, a WFFM module is also proposed, which can assign learnable weights to multi-scale color features. The introduction of this mechanism enhances the model's ability to prioritize relevant features, thus more effectively representing color details in image aesthetic evaluation. In this way, the method of the application can more accurately capture key color information in images, providing more accurate and comprehensive results for aesthetic evaluation.

[0093] The image color aesthetic evaluation method based on multi-scale feature fusion provided by the application can achieve the following technical effects:

[0094] (1) Global perception and fine-grained attention: Compared with the method based on Transformer, the application combines the Mamba model and multi-scale feature fusion technology to achieve global perception of complex color combinations. Through dynamic weight distribution, it ensures that key color information is finely focused, thereby improving the comprehensiveness and accuracy of color aesthetic evaluation.

[0095] (2) Spatial information perception and performance optimization: By using bidirectional Vision Mamba blocks for global visual context modeling, the application effectively enhances the model's ability to perceive spatial information, improving overall performance in the field of image color aesthetic evaluation. This comprehensive design makes the application have a significant advantage over existing technologies in dealing with complex color combinations and improving overall aesthetic perception.

[0096] In summary, the application has more comprehensive and accurate feature extraction capabilities in image color aesthetic evaluation. Compared with existing technologies, it can more effectively deal with complex color combinations and significantly improve the overall aesthetic perception of image colors.

[0097] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

Claims

1. An image color aesthetic evaluation method based on multi-scale feature fusion, characterized in that, Includes the following steps: S1, converts the image data of the input image into the sequence data of the input image; S2, perform n feature extraction operations on the sequence data of the input image to obtain n multi-scale color features, and add the n multi-scale color features into vectors to obtain the aggregated color feature F; S3, perform a feature weighting and fusion operation on the aggregated color feature F: highlight key color information through weight allocation, thereby obtaining the calibrated color feature. ; S4, using calibrated color features Perform image color aesthetic evaluation on the input image; In step S2, feature extraction is performed on the sequence data of the input image, specifically including the following steps: using bidirectional VMamba blocks to perform continuous data processing on the sequence data of the input image, and performing a multi-scale color feature extraction operation after multiple data processing operations; In step S3, a feature weighting and fusion operation is performed on the aggregated color feature F, specifically including the following steps: S3.1, Perform feature weighting operation on aggregated color feature F: use global average pooling to calculate the channel statistics of aggregated color feature F to generate a global feature descriptor; map the global feature descriptor through a multilayer perceptron, and then transform the mapped data through the Softmax function to generate attention weight W; S3.2, use attention weights W to calibrate multi-scale color features; S3.3, perform a fusion operation on the calibrated multi-scale color features to obtain the calibrated color features. ; In step S4, the calibration color feature is used. The image color aesthetic evaluation of the input image includes the following steps: learning calibration color features using global average pooling and a multilayer perceptron. The mapping relationship with aesthetic scores was then established, and a regressor was used. The feature mapping relationship is converted into the final aesthetic score.

2. The image color aesthetic evaluation method based on multi-scale feature fusion according to claim 1, characterized in that, In step S3.2, a calibration operation is performed on the multi-scale color features, specifically including the following steps: For the nth multi-scale color feature Fn, a scalar multiplication is performed using the corresponding nth attention weight Wn to obtain the nth calibrated color feature. .

3. The image color aesthetic evaluation method based on multi-scale feature fusion according to claim 1, characterized in that, In step S3.3, a fusion operation is performed on the calibrated multi-scale color features, specifically including the following steps: fusing the n calibrated color features... By performing vector addition, the calibration color features are obtained. .

4. The image color aesthetic evaluation method based on multi-scale feature fusion according to claim 1, characterized in that, In step S2, feature extraction is performed on the sequence data of the input image, specifically including the following steps: after the sequence data of the input image undergoes data processing operations of 4 VMamba blocks, a multi-scale color feature extraction operation is performed, and a multi-scale color feature is generated. After n feature extraction operations, n multi-scale color features (F1, F2, ..., Fn) are obtained in sequence.

5. The image color aesthetic evaluation method based on multi-scale feature fusion according to claim 1, characterized in that, In step S2, the data processing operations performed on the sequence data of the input image using VMamba blocks specifically include the following steps: S2.1, Preprocess the sequence data using normalized data preprocessing; S2.2, use linear projection and activation functions to process the preprocessed sequence data to obtain the first data; S2.3, the preprocessed sequence data is processed sequentially using linear projection, depthwise separable convolution, and state-space model to obtain the second data; S2.4, perform element-wise multiplication and merging of the first data and the second data, then perform linear projection, and finally perform residual connection and merging with the unprocessed sequence data.

6. The image color aesthetic evaluation method based on multi-scale feature fusion according to claim 5, characterized in that, Steps S2.3 and S2.4 further include the following steps: processing the preprocessed sequence data using linear projection, forward convolution, and forward state space model to obtain the third data; processing the preprocessed sequence data using linear projection, backward convolution, and backward state space model to obtain the fourth data; performing element-wise multiplication and merging of the first data with the third and fourth data respectively, and performing vector addition on the results of the two, then performing linear projection, and then performing residual connection and merging with the unprocessed sequence data.

Images