Multi-modal cultural product value evaluation method

Abstract

The invention discloses a multi-modal cultural product value evaluation method. The method is specifically implemented according to the following steps: step 1, collecting and preprocessing cultural product data; 2, performing feature coding on the text features and the image features processed in the step 1 to obtain a text multi-modal feature vector and an image multi-modal feature vector; 3, preliminarily predicting the price of the cultural product; and step 4, sending the result of preliminary prediction of the price of the cultural product obtained in the step 3 into the constructed weighted voting mechanism to obtain a final prediction result of the price of the cultural product. According to the method, machine learning and a voting mechanism are combined, the accuracy of cultural product value prediction is improved, meanwhile, manual evaluation is avoided, and the evaluation efficiency is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data analysis and prediction, and specifically relates to a multimodal cultural product value assessment method. Background Art

[0002] The industry characteristics such as high added value, strong plasticity, low energy consumption and pollution have led to a significant increase in the attention paid to the cultural industry in recent years. With the continuous expansion of the cultural industry, the transaction of cultural products has assumed the new mission of promoting cultural exchanges and economic and trade cooperation, and has become an important driving force for the rapid development of the cultural industry. However, the cultural product transaction process still faces some challenges. First, the cultural product value assessment process is greatly influenced by the subjective influence of the evaluator, and the credibility of the cultural product value assessment is lacking; second, cultural product supply companies have light asset characteristics and it is difficult to provide tangible value guarantees. The above factors have resulted in an imperfect cultural product value assessment mechanism, which is also the main factor affecting the activity and transaction volume of cultural product transactions, hindering the healthy development of cultural industry transactions. Therefore, how to make a more reasonable value assessment of the transaction value of cultural products is an urgent problem to be solved.

[0003] The traditional cultural value assessment process usually relies on professional manual appraisal to conduct comprehensive analysis and evaluation of cultural products from multiple dimensions and perspectives. However, the above evaluation process is easily affected by personal subjective factors, redundant market information and current trends, which makes the results of cultural product value assessment less credible and the evaluation efficiency low. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal cultural product value assessment method to solve the problems of low credibility and low assessment efficiency of existing cultural product value assessment results.

[0005] The technical solution adopted by the present invention is a multimodal cultural product value assessment method, which is specifically implemented according to the following steps:

[0006] Step 1, cultural product data collection and preprocessing;

[0007] Step 2, performing feature encoding on the text features and image features processed in step 1 to obtain a text multimodal feature vector and an image multimodal feature vector;

[0008] Step 3, make a preliminary forecast of the price of cultural products;

[0009] Step 4, sending the preliminary prediction results of the cultural product prices obtained in step 3 to the constructed weighted voting mechanism to obtain the final prediction results of the cultural product prices.

[0010] The present invention is also characterized in that:

[0011] In step 1, the cultural product data includes the work description, form, major category classification, minor category classification, material, creation year, auction time, size, auction price, auction company, work image information, author name, and author dynasty;

[0012] The description of the work is a text feature; the creation year, auction time, form, material, author's name, author's dynasty, size, auction price, and auction company are numerical features; the image information of the work is an image feature; the major category classification and minor category classification of the work are category features.

[0013] The major categories of works include Chinese calligraphy and painting, and Western painting; the minor categories of works include painting, printmaking, calligraphy, sketching, oil painting, and gouache.

[0014] In step 1, the preprocessing of cultural product data is as follows:

[0015] The numerical features and categorical features are numerically encoded, the missing values ​​of cultural product data are filled, the outliers of indicators are eliminated, and the auction prices are converted at exchange rates, using RMB as the unified transaction currency unit to obtain the processed cultural product data set.

[0016] The specific process of step 2 is:

[0017] Step 2.1, use the Vision Transformer model to encode the image features and obtain a 512-dimensional image multimodal feature vector;

[0018] In step 2.2, the RoBERTa model is used to encode the text features. The encoding method is byte pair encoding. The vocabulary size is 50K, and a 512-dimensional text multimodal feature vector is obtained.

[0019] The specific process of step 3 is:

[0020] Step 3.1, sequentially concatenate the numerical features, the category features, the image multimodal feature vectors, and the text multimodal feature vectors to obtain concatenated features;

[0021] Step 3.2, divide the concatenated features obtained in step 3.1 into a training set and a test set;

[0022] Step 3.3, input the training set into the XGBoost model for training to obtain a trained XGBoost model, input the test set into the trained XGBoost model to obtain the preliminary prediction result of the cultural product price I;

[0023] Step 3.4, input the training set into the LightGBM model for training to obtain the trained LightGBM model, input the test set into the trained LightGBM model to obtain the preliminary prediction result of cultural product prices II;

[0024] Step 3.5, input the training set into the CatBoost model for training to obtain the trained CatBoost model, and input the test set into the trained CatBoost model to obtain the preliminary prediction result of the cultural product price III.

[0025] The specific process of step 4 is:

[0026] Step 4.1, the numpy library randomly generates the np.random.rand() function, generates a random weight for the XGBoost model, LightGBM model, and CatBoost model respectively, and normalizes the current weight of each model to obtain the normalized weight of each model;

[0027] Step 4.2, multiply the result I of the preliminary prediction of cultural product prices by the normalized weight of the XGBoost model, multiply the result II of the preliminary prediction of cultural product prices by the normalized weight of the LightGBM model, and multiply the result III of the preliminary prediction of cultural product prices by the normalized weight of the CatBoost model, and obtain the weighted prediction results of the three models respectively;

[0028] Step 4.3, add the weighted prediction results of the three models to the initialized list respectively, sum up all the weighted prediction results in the list, and obtain the final prediction result of the cultural product price.

[0029] The beneficial effects of the present invention are:

[0030] (1) The multimodal cultural product value assessment method of the present invention effectively integrates the visual information of cultural products and the textual information of various assessment indicators, and utilizes the complementarity between multimodal cultural product data to more comprehensively evaluate the value of cultural products, thereby greatly improving the credibility of the assessment results;

[0031] (2) The multimodal cultural product value assessment method of the present invention uses a machine learning model to assess the value of cultural products, and combines a voting mechanism to further improve the accuracy of cultural product value prediction. At the same time, it avoids manual assessment and greatly improves the assessment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the multimodal cultural product value assessment method of the present invention;

[0033] Figure 2The figures are comparison diagrams of the prediction results of the method of the present invention and a single model. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Example 1

[0036] The multimodal cultural product value assessment method of the present invention is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0037] Step 1, cultural product data collection and preprocessing;

[0038] Cultural product data include work description, form, major work classification, minor work classification, material, creation year, auction time, size, auction price, auction company, work image information, author name, and author dynasty;

[0039] The major categories of works include Chinese calligraphy and painting, Western painting;

[0040] Subcategories of works include painting, printmaking, calligraphy, drawing, oil painting, and gouache;

[0041] Among them, the description of the work is the text feature; the creation year, auction time, author dynasty, size, auction price, and auction company are the numerical features; the image information of the work is the image feature; the shape, material, author's name, major category classification of the work, and minor category classification of the work are the category features;

[0042] The preprocessing of cultural product data is as follows:

[0043] The numerical features and categorical features are numerically encoded, where the numerical features include material, shape, author's name, author's dynasty, and auction company. Then the missing values ​​of the cultural product data are filled, the outliers of the indicators are eliminated, and the auction prices are converted at the exchange rate, with RMB as the unified transaction currency unit, to obtain the processed cultural product data set.

[0044] Step 2, performing feature encoding on the text features and image features processed in step 1 to obtain a text multimodal feature vector and an image multimodal feature vector;

[0045] The specific process is:

[0046] Step 2.1: Use the Vision Transformer model to encode the image features and obtain a 512-dimensional image multimodal feature vector;

[0047] The Vision Transformer model divides the original image into several sub-images in the form of a grid, first processes it through a convolutional layer to extract preliminary features of the image, and then flattens the image in the height and width directions after the convolutional layer, converting the two-dimensional image data into a one-dimensional data sequence. The flattened image data is divided into small blocks, each of which is then converted into a vector. Each vector represents a local feature in the original image. For each image, the Vision Transformer model generates a 512-dimensional image multimodal feature vector after normalization.

[0048] Step 2.2: Use the RoBERTa model (Robustly optimized BERT approach) to encode text features. The encoding method is Byte Pair Encoding (BPE). The vocabulary size is 50K, and a 512-dimensional text multimodal feature vector is obtained.

[0049] The RoBERTa model is a lightweight version of the BERT model in the comparative language-image pre-training model in the Chinese context. It divides the input text into a series of sub-word units with bytes as the basic unit. The vocabulary size is 50K. It iteratively analyzes the text data to find the most common character pairs and merge them into a single symbol. This process is repeated until the predetermined vocabulary size is reached or no more common character pairs appear. Finally, after normalization, a 512-dimensional text multimodal feature vector containing deep semantic information of the text is output.

[0050] Step 3, make a preliminary forecast of the price of cultural products;

[0051] The specific process is:

[0052] Step 3.1, sequentially concatenate the numerical features, the category features, the image multimodal feature vectors, and the text multimodal feature vectors to obtain concatenated features;

[0053] Step 3.2, divide the concatenated features obtained in step 3.1 into a training set and a test set, with a division ratio of 8:2;

[0054] Step 3.3, input the training set into the XGBoost model (Extreme Gradient Boosting) for training to obtain the trained XGBoost model, input the test set into the trained XGBoost model, and obtain the preliminary prediction result of the cultural product price I;

[0055] The objective function of the XGBoost model is:

[0056]

[0057] In formula (1), a represents a constant; represents the predicted value of the model iterated to the tth tree; f t (x i ) represents the score of the newly added tth tree; represents the model prediction value after the optimization of the previous t-1 trees; x i represents the i-th sample feature vector; n is the total number of data points; y i Represents the true value; Ω represents the model complexity function term;

[0058] If the loss function is a general function, the minimum value is solved by the Tate expansion method; by recursively calling the above tree building method, a large number of regression tree structures can be obtained, and the target is used to search for the optimal tree structure;

[0059] in,

[0060]

[0061] In formula (2), f k The function representing the kth tree, f k ∈F(i=1,2,…n), where F represents the set of all regression trees, expressed as:

[0062] F={f(x)=w q(x)}(q:R m →{1,2,…T},w∈R T )(3)

[0063] In formula (3), q(x) represents the mapping function of the sample data corresponding to a leaf node on the tree; w q(x) The score of all leaf nodes in the regression tree, that is, the weight value corresponding to the input x, R m means that each element is a vector containing m real numbers, R T Each element is a vector containing T real numbers;

[0064]

[0065] In formula (4), γ and λ are XGBoost custom parameters for limiting the number of leaf nodes and controlling the size of node scores, respectively; T represents the number of leaf nodes in the tree, and w j is the value of the jth parameter in the model;

[0066] The preliminary prediction result I of the cultural product price is saved and evaluated using the mean square error (MSE) and mean absolute error (MAE). MSE is the mean square error that measures the gap between the predicted value and the true value. The closer the value is to 0, the smaller the error.

[0067]

[0068] In the formula, Expressed as the predicted value, y i represents the true value, and m represents the number of samples.

[0069] MAE is the mean absolute error, which is mainly used to measure the accuracy of the true value and the predicted result. The closer the value is to 0, the better the effect.

[0070]

[0071] In the formula, Expressed as the predicted value, y i represents the true value, and m represents the number of samples.

[0072] Step 3.4, input the training set into the LightGBM model (Light Gradient Boosting Machine) for training to obtain a trained LightGBM model, input the test set into the trained LightGBM model, and obtain the preliminary prediction result of cultural product prices II;

[0073] The residual value of the current decision tree is approximated by using the negative gradient of the loss function and used to fit a new decision tree. In each iteration, the model remains unchanged and a new function is added to the model to continuously reduce the difference between the predicted value and the measured value;

[0074] The objective function of the LightGBM model is:

[0075]

[0076] In formula (5), i is the number of data points; x i is the feature of the data point, y i is the measured value, is the predicted value of the K-th learning, L is the error function term, Ω(f K ) is the regularization term of the Kth tree, c K-1 is the regularization term and of the previous tree K-1, f K is the function of the Kth tree;

[0077] The result of the quadratic Taylor expansion of the objective function is:

[0078]

[0079] In formula (6), g i is the first-order derivative of the loss function of the ith sample, h i is the second-order derivative of the i-th sample loss function;

[0080] LightGBM uses the histogram method to discretize continuous floating-point feature values ​​into integers. These discrete values ​​are used to construct a histogram with a width of k. When traversing the data, the discretized values ​​are used as cumulative statistics in the histogram. After traversing the data once, the histogram accumulates the required statistics and finds the optimal segmentation point based on the discrete values ​​of the histogram.

[0081] When training the model, LightGBM uses unilateral gradient and random sampling methods to retain instances with larger gradients and randomly sample instances with smaller gradients, so that accurate estimates can be obtained using less data. LightGBM also uses mutually exclusive feature merging technology to merge mutually exclusive features within a certain conflict ratio, achieving the effect of dimensionality reduction from the perspective of reducing features, thereby preventing information loss.

[0082] Similarly, MSE and MAE were used for model evaluation;

[0083] Step 3.5, input the training set into the CatBoost model (Category Boosting) for training to obtain the trained CatBoost model, input the test set into the trained CatBoost model, and obtain the preliminary prediction result of the cultural product price III;

[0084] For categorical features, the CatBoost model proposes statistics based on ordered target variables. This method relies on the sorting principle and introduces "artificial time" to randomly sort the data.

[0085] Then the expression of the target variable statistics of the CatBoost model is:

[0086] D k ={x j :σ(j)<σ(k)}(7)

[0087]

[0088] In equations (7) and (8), p is the added prior term; β is a weight coefficient greater than 0, σ represents random permutation; x i,k is the encoding result of the k-th category feature of the i-th sample; x j,k is the k-th category feature value of the j-th sample; Dk Represents the node of the decision tree; x j represents the jth sample or data point in the data set; σ(j) is the value of the jth sample under the sorting criteria; σ(k) is the threshold of the kth node;

[0089] Similarly, MSE and MAE were used for model evaluation.

[0090] Step 4, sending the preliminary prediction result of the cultural product price obtained in step 3 to the constructed weighted voting mechanism to obtain the final prediction result of the cultural product price;

[0091] The specific process is:

[0092] In step 4.1, the weighted voting mechanism is randomly generated by the numpy library (Numeric Python) using the np.random.rand() function to generate a random weight for the XGBoost model, the LightGBM model, and the CatBoost model, respectively. The current weight of each model is normalized to obtain the normalized weight of each model, where the normalized weight of the XGBoost model is 0.3865, the normalized weight of the LightGBM model is 0.3225, and the normalized weight of the CatBoost model is 0.2910.

[0093] Step 4.2, multiply the result I of the preliminary prediction of cultural product prices by the normalized weight of the XGBoost model, multiply the result II of the preliminary prediction of cultural product prices by the normalized weight of the LightGBM model, and multiply the result III of the preliminary prediction of cultural product prices by the normalized weight of the CatBoost model, and obtain the weighted prediction results of the three models respectively;

[0094] Step 4.3, add the weighted prediction results of the three models to an initialized empty list "predictions" list, sum up all the weighted prediction results in the list, and get the final prediction result of the cultural product price.

[0095] Example 2

[0096] The error rate evaluation index of the method of the present invention, the XGBoost regression prediction method, the LightGBM regression prediction method, and the CatBoost regression prediction method are compared respectively. The comparison results are as follows: Figure 2 As shown, the prediction error rate of the method of the present invention is the lowest, which is only 16.82%, which is significantly better than the other three methods. This shows that the method of the present invention can more accurately estimate the value of cultural products and provide a new possibility for the value estimation of cultural products.

[0097] Example 3

[0098] The method of the present invention is oriented to cultural product data of multiple modes, and uses multimodal representation learning technology to achieve effective feature extraction of multi-angle cultural product data, and uses feature fusion technology to achieve the complementarity of multimodal cultural product features. Finally, a machine learning model is used to perform regression training on complementary and redundant multimodal cultural product features to achieve objective and reliable cultural product value evaluation. The method of the present invention is different from the traditional manual evaluation method. Based on a large amount of real and reliable cultural product value data, it uses a machine learning-based method to predict the value of cultural products, ensuring that the evaluation process is more objective and fair than manual evaluation. In addition, the intelligent evaluation method has greatly improved the speed and efficiency of cultural product value evaluation.

Claims

1. A multimodal cultural product value assessment method, characterized in that: Follow the steps below to implement it: Step 1, cultural product data collection and preprocessing; Step 2, performing feature encoding on the text features and image features processed in step 1 to obtain a text multimodal feature vector and an image multimodal feature vector; Step 3, make a preliminary forecast of the price of cultural products; Step 4, sending the preliminary prediction results of the cultural product prices obtained in step 3 to the constructed weighted voting mechanism to obtain the final prediction results of the cultural product prices.

2. The multimodal cultural product value assessment method according to claim 1, characterized in that: In step 1, the cultural product data includes the work description, form, major category classification, minor category classification, material, creation year, auction time, size, auction price, auction company, work image information, author name, and author dynasty; The description of the work is a text feature; the creation year, auction time, form, material, author's name, author's dynasty, size, auction price, and auction company are numerical features; the image information of the work is an image feature; the major category classification and minor category classification of the work are category features.

3. The multimodal cultural product value assessment method according to claim 2, characterized in that: The major categories of works include Chinese calligraphy and painting, and Western painting; the minor categories of works include painting, printmaking, calligraphy, sketching, oil painting, and gouache.

4. The multimodal cultural product value assessment method according to claim 1, characterized in that: In step 1, the preprocessing of cultural product data is as follows: The numerical features and categorical features are numerically encoded, the missing values ​​of cultural product data are filled, the outliers of indicators are eliminated, and the auction prices are converted at exchange rates, using RMB as the unified transaction currency unit to obtain the processed cultural product data set.

5. The multimodal cultural product value assessment method according to claim 2, characterized in that: The specific process of step 2 is: Step 2.1, use the Vision Transformer model to encode the image features and obtain a 512-dimensional image multimodal feature vector; In step 2.2, the RoBERTa model is used to encode the text features. The encoding method is byte pair encoding. The vocabulary size is 50K, and a 512-dimensional text multimodal feature vector is obtained.

6. The multimodal cultural product value assessment method according to claim 5, characterized in that: The specific process of step 3 is: Step 3.1, sequentially concatenate the numerical features, the category features, the image multimodal feature vectors, and the text multimodal feature vectors to obtain concatenated features; Step 3.2, divide the concatenated features obtained in step 3.1 into a training set and a test set; Step 3.3, input the training set into the XGBoost model for training to obtain a trained XGBoost model, input the test set into the trained XGBoost model to obtain the preliminary prediction result of the cultural product price I; Step 3.4, input the training set into the LightGBM model for training to obtain the trained LightGBM model, input the test set into the trained LightGBM model to obtain the preliminary prediction result of cultural product prices II; Step 3.5, input the training set into the CatBoost model for training to obtain the trained CatBoost model, and input the test set into the trained CatBoost model to obtain the preliminary prediction result of the cultural product price III.

7. The multimodal cultural product value assessment method according to claim 6, characterized in that: The specific process of step 4 is: Step 4.1, the numpy library randomly generates the np.random.rand() function, generates a random weight for the XGBoost model, LightGBM model, and CatBoost model respectively, and normalizes the current weight of each model to obtain the normalized weight of each model; Step 4.2, multiply the result I of the preliminary prediction of cultural product prices by the normalized weight of the XGBoost model, multiply the result II of the preliminary prediction of cultural product prices by the normalized weight of the LightGBM model, and multiply the result III of the preliminary prediction of cultural product prices by the normalized weight of the CatBoost model, and obtain the weighted prediction results of the three models respectively; Step 4.3, add the weighted prediction results of the three models to the initialized list respectively, sum up all the weighted prediction results in the list, and obtain the final prediction result of the cultural product price.

Images