A method for reproducing the price index of ceramic raw materials

Abstract

This invention relates to the field of price index reproduction technology, and more particularly to a method for reproducing ceramic raw material price indices. The method includes: acquiring structured data of the basic attributes of ceramic raw materials from a database on an industrial internet platform; importing this data into a pre-trained ceramic raw material classification model for standard-compliant classification to obtain various ceramic raw material datasets; acquiring historical transaction data, aggregated logistics information data, and basic information of end customers from the industrial internet platform's database; performing initial processing on the above data and various ceramic raw material datasets to obtain various commodity datasets; further integrating or removing duplicate or discrete data to obtain non-duplicate commodity datasets; performing aggregation calculations on a weekly basis to obtain continuous weekly indices for various commodities; transmitting the continuous weekly indices of various commodities to a price index database for updating, and updating and publishing continuous weekly index views for various commodities.

Description

Technical Field

[0001] This invention relates to the field of price index reproduction technology, and in particular to a method for reproducing the price index of ceramic raw materials. Background Technology

[0002] With my country's economic development, the industrial internet has gradually grown stronger. Data collection in traditional industries, such as transportation and transactions, has broken through previous bottlenecks. As the accumulated data continues to grow, its scale upgrades, and its applications deepen, the value of big data is becoming increasingly undeniable.

[0003] Because the ceramic raw material trading industry is a traditional and closed sector, product pricing is determined through negotiation between buyers and sellers, relying heavily on word-of-mouth. This has resulted in the lack of electronic, information-based, and online methods for collecting and storing industry transaction and transportation information. Consequently, price indices and other technologies are absent from the ceramic raw material trading industry. Therefore, how to effectively utilize transaction data from the ceramic industry to create price indices has become a crucial and urgent issue that the industry needs to address. Summary of the Invention

[0004] The purpose of this invention is to propose a method for reproducing the price index of ceramic raw materials. This method combines big data thinking and technology, and by using data generated in the transportation and transaction processes of an industrial internet platform, it can accurately obtain continuous weekly indices for various commodities after multiple processing steps.

[0005] To achieve this objective, the present invention adopts the following technical solution:

[0006] A method for reproducing a ceramic raw material price index includes the following steps:

[0007] S1. Obtain structured data of the basic attributes of ceramic raw materials from the database of the industrial internet platform, import the structured data of the basic attributes of ceramic raw materials into the pre-trained ceramic raw material classification model for classification in accordance with the standards, and obtain various ceramic raw material datasets.

[0008] S2. Obtain historical transaction data, logistics information aggregation data, and basic information of end customers from the database of the industrial internet platform. Perform initial processing on the above data and various ceramic raw material datasets to obtain various commodity datasets.

[0009] S3. Integrate or remove duplicate or discrete data in various commodity datasets to obtain non-duplicate commodity datasets.

[0010] S4. Aggregate and calculate the data of various non-repeating commodity datasets in "week" as the time unit to obtain the continuous weekly index of various commodities;

[0011] S5. Transfer the continuous weekly index of various commodities to the price index database for updating, and update and publish the continuous weekly index view of various commodities.

[0012] Preferably, in S1, the ceramic raw material classification model specifically includes the following pre-training steps:

[0013] S11. Data source acquisition: Obtain structured data of the basic attributes of historical ceramic raw materials from the database of the industrial internet platform. The structured data of the basic attributes of ceramic raw materials includes commodity, raw material composition and morphological information.

[0014] S12. Logically associate the datasets according to the dataset filtering logic, and perform data topology on the data sources based on the logical associations to obtain the datasets; wherein, the dataset filtering logic is the commodity category, commodity sub-category and commodity secondary sub-category, commodity, raw material composition and morphology information;

[0015] S13. Perform data cleaning on the dataset to obtain a cleaned basic dataset;

[0016] S14. Perform data feature calculations on the basic dataset to obtain a dataset with completed feature engineering;

[0017] S15. Divide the dataset with completed feature engineering into a training set and a test set;

[0018] S16. Based on the XGBoost machine learning model, a ceramic raw material classification model is trained and established using the training set, and then tested using the test set; wherein, the function of the ceramic raw material classification model is to input the raw material composition and morphology information, and output the predicted commodity category, commodity sub-category, and commodity secondary sub-category;

[0019] S17. Evaluate the effectiveness of the ceramic raw material classification model based on the prediction error scores of the training and test sets. If the prediction error score reaches the predicted value, the ceramic raw material classification model is deemed usable. If the prediction error score does not reach the predicted value, the ceramic raw material classification model is deemed unusable. Adjust the model parameters of the ceramic raw material classification model and re-execute S15-S16.

[0020] Preferably, in S13, the data cleaning includes format content cleaning and outlier cleaning.

[0021] Preferably, in S14, the data feature calculation includes digitally standardizing the commodity, raw material composition and morphology information, and calculating data features.

[0022] Preferably, in S17, the prediction error scores of the training set and the test set include the model score R2_SCORE, and the formula for calculating R2_SCORE is as follows:

[0023]

[0024] Where R² is the prediction error score; y is the actual value; Let y(i) be the predicted value of the i-th value; y(i) be the actual value of the i-th value. Let y be the average value.

[0025] Preferably, in S17, the model parameters of the ceramic raw material classification model include the number of tree model iterations n_estimators, the tree depth max_depth, and the loss function threshold gamma when splitting nodes.

[0026] Preferably, the historical transaction data of the industrial internet platform includes the following fields: supplier, goods user, order number, transaction time, and unit price;

[0027] The aggregated logistics information data includes the following fields: transportation time, unit, and quantity;

[0028] The basic information of the end customer includes the following fields: end customer name, goods user unit, and industry.

[0029] Preferably, in S2, the initial processing of historical transaction data, aggregated logistics information data, basic information of end customers, and various ceramic raw material datasets from the industrial internet platform to obtain various commodity datasets specifically includes the following steps:

[0030] S21. Historical transaction data, aggregated logistics information data, and basic information of end customers are associated and merged with various ceramic raw material datasets through association conditions to obtain various commodity datasets; the association conditions include order number, commodity name, and end customer name.

[0031] S22. Conduct exploratory analysis on various commodity datasets, and after analysis, unify the units, remove missing values ​​and outliers to obtain the correct commodity datasets.

[0032] Preferably, in S3, the process of integrating or removing duplicate or discrete data from various product datasets to obtain non-duplicate product datasets specifically includes the following steps:

[0033] S31. Count the number of orders for each type of product and remove products with fewer than 3 orders.

[0034] S32. Find data that has two or more identical rows, remove duplicate data, and obtain a dataset of each product that is unique.

[0035] Preferably, in S4, the data of various non-repeating commodity datasets are aggregated and calculated using "week" as the time unit to obtain continuous weekly indices for various commodities; specifically, this includes the following steps:

[0036] S41. For non-repeating data sets of various products, change the delivery date within the same week to Monday of that week.

[0037] S42. Calculate the weekly index for various commodities:

[0038] S421. Count the quantity n of each type of commodity, obtain the price information of each commodity i, and divide the price at time t by the initial price of commodity i to obtain the price change R of commodity i at time t. ti ;

[0039] S422. Assign different weights W to different products i according to demand. i ;

[0040] S423. Accumulate the price changes (R values) of multiple commodities (n) within the same time period (t), multiply by a base of 1000, and obtain the weekly index It; the formula for calculating the weekly index It is as follows:

[0041]

[0042]

[0043] S43. Combine each weekly index to form a continuous weekly index.

[0044] One of the above technical solutions has the following beneficial effects: By combining big data thinking and technology, and using an industrial internet platform to acquire data generated during transportation and transactions, and after multiple processing, it is possible to correctly obtain continuous weekly indices for various commodities. Then, by continuously updating and forming a visualized continuous weekly index view for various commodities, it is possible to review the price trends of commodities in different periods in the past, which is conducive to guiding market participants in transactions and is a necessary technical means for industrial digitalization and standardization. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a method for reproducing a ceramic raw material price index according to the present invention.

[0046] Figure 2 This is a flowchart illustrating the construction process of the ceramic raw material classification model of the present invention;

[0047] Figure 3 This is a continuous weekly index view according to one embodiment of the present invention;

[0048] Figure 4This is a schematic diagram of the dataset filtering logic for industry categories according to one embodiment of the present invention; Detailed Implementation

[0049] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0050] like Figure 1 As shown, a method for reproducing a ceramic raw material price index includes the following steps:

[0051] S1. Obtain structured data of the basic attributes of ceramic raw materials from the database of the industrial internet platform, import the structured data of the basic attributes of ceramic raw materials into the pre-trained ceramic raw material classification model for classification in accordance with the standards, and obtain various ceramic raw material datasets.

[0052] Since members conduct commodity transactions on the industrial internet platform, they fill in structured data related to basic commodity attributes such as unit price, quantity, commodity category, commodity subcategory, and ingredients. The Zhongtaolian industrial internet platform uses technical means and manual review of transaction contracts, delivery documents, logistics information, and account statements to ensure consistency between online order information and data.

[0053] To avoid errors in data entry due to misunderstandings of ceramic raw material classification by the data entry personnel or the machine, a ceramic raw material classification model was created. When using this pre-trained ceramic raw material classification model, the data entry personnel or the machine only need to input the ceramic raw material information as required to automatically classify and fill in the major and minor categories of ceramic raw materials.

[0054] For example, a new ceramic raw material A is introduced, with the following composition: Al2O3 (aluminum oxide): 15.7%, K2O (potassium oxide): 4.05%, Na2O (sodium oxide): 4.3%, whiteness: 24, and form: granules. It is commonly referred to in the market as "potassium-sodium stone powder." However, this can make it difficult for data entry personnel to distinguish between potassium stone and sodium stone. A pre-trained model, based on historical information, will classify it as: Major Category: Ceramic Raw Materials; Subcategory: Ceramic Raw Material Stones; Secondary Subcategory: Sodium Stone Powder; achieving automated and standardized classification.

[0055] In S1, the ceramic raw material classification model specifically includes the following pre-training steps:

[0056] S11. Data Source Acquisition: Obtain structured data of the basic attributes of historical ceramic raw materials from the database of the industrial internet platform. The structured data of the basic attributes of the ceramic raw materials includes commodity, commodity raw material composition and morphological information; for example, the structured data of the basic attributes of a certain historical ceramic raw material is as follows: Commodity: Potassium sand, Raw material composition: Al2O3 (aluminum oxide), CaO (calcium oxide), Fe2O3 (ferric oxide), K2O (potassium oxide), MgO (magnesium oxide), Na2O (sodium oxide), TiO2 (titanium oxide), Morphological information: whiteness, drying strength and other information;

[0057] S12. Perform logical associations according to the dataset filtering logic, and perform data topology on the data source based on the logical associations to obtain the dataset; wherein, the dataset filtering logic includes commodity major categories, commodity subcategories and commodity secondary subcategories, commodity, raw material composition and form information; wherein the dataset filtering logic for some industry major categories is as follows: Figure 3 As shown.

[0058] S13. Perform data cleaning on the dataset to obtain a cleaned basic dataset; wherein, the data cleaning includes format and content cleaning and outlier cleaning; specifically:

[0059] (1) When a column of data contains a small number of empty items, use the mode value of the column to replace them;

[0060] (2) Calculate the mean and standard value (sigma) of each column of data (datan). Remove any outlier (diff) that is greater than three times the standard value. The formula is: diff{|datan|sigma} n -mean|>3*sigma}.

[0061] S14. Perform data feature calculations on the basic dataset to obtain a dataset with completed feature engineering; wherein, the data feature calculation includes digital standardization of commodity, raw material composition and morphology information, and calculation of data features; specifically, for example, to distinguish between large and small ticket information, replace "potassium sand" with 0, "aluminum sand" with 1, "sodium sand" with 2, "potassium stone powder" with 3, "sodium stone powder" with 4, "kaolin" with 5, and "washed mud" with 6;

[0062] The output dataset after feature engineering is shown below, displayed vertically:

[0063]

[0064] S15. Divide the dataset with completed feature engineering into a training set and a test set;

[0065] Specifically, 70% of the dataset with completed feature engineering is divided into a training set, and the remaining 30% of the dataset with completed feature engineering is divided into a test set.

[0066] S16. Based on the XGBoost machine learning model, a ceramic raw material classification model is trained and established using the training set, and then tested using the test set. The ceramic raw material classification model is designed to take as input the raw material composition and morphology information and output the predicted commodity category, commodity sub-category, and commodity secondary sub-category. As shown in the table below, the ceramic raw material classification model filters out the kaolin secondary sub-category within the clay sub-category of the ceramic category.

[0067]

[0068]

[0069] S17. Evaluate the effectiveness of the ceramic raw material classification model based on the prediction error scores of the training and test sets. If the prediction error score reaches the predicted value, the ceramic raw material classification model is deemed usable. If the prediction error score does not reach the predicted value, the ceramic raw material classification model is deemed unusable. Adjust the model parameters of the ceramic raw material classification model and re-execute S15-S16.

[0070] The prediction error scores for the training and test sets include the model score R2_SCORE, which is calculated using the following formula:

[0071]

[0072] Where R² is the prediction error score; y is the actual value; Let y(i) be the predicted value of the i-th value; y(i) be the actual value of the i-th value. Let y be the average value.

[0073] When the R2 score is greater than 0.75 and less than 1, the ceramic raw material classification model is considered usable, and the next step of outputting the pre-trained ceramic raw material classification model can proceed. Otherwise, the ceramic raw material classification model needs to have its parameters readjusted, including the number of iterations n_estimators for the tree model, the tree depth max_depth, and the loss function threshold gamma when splitting nodes. Specifically, for the XGBoost machine learning model, grid training (GridSearchCV) is used for parameter tuning, with the following parameter ranges: n_estimators: 1, 5, 10, 15; max_depth: 5, 7, 8; gamma: 0.1, 0.2, 0.5; after training, the optimal parameters are: n_estimators: 5, max_depth: 7, gamma: 0.1.

[0074] By using a pre-trained ceramic raw material classification model, a unified standard for classifying ceramic raw materials between old and new data is achieved, avoiding classification imbalances caused by differences in composition in subsequent new data. Simultaneously, the new ceramic raw material data is also stored in the industrial internet platform's database for future use.

[0075] S2. Obtain historical transaction data, aggregated logistics information data, and basic information of end customers from the industrial internet platform's database. Perform initial processing on the above data and various ceramic raw material datasets to obtain various commodity datasets; specifically, this includes the following steps:

[0076] S21. Historical transaction data, aggregated logistics information data, and basic information of end customers are associated and merged with various ceramic raw material datasets through association conditions to obtain various commodity datasets; the association conditions include order number, commodity name, and end customer name.

[0077] The historical transaction data of the industrial internet platform includes the following fields: supplier, goods user, order number, transaction time and unit price;

[0078] The aggregated logistics information data includes the following fields: transportation time, unit, and quantity;

[0079] The basic information of the end customer includes the following fields: end customer name, goods user unit, and industry.

[0080] S22. Conduct exploratory analysis on various commodity datasets, and after analysis, unify the units, remove missing values ​​and outliers to obtain the correct commodity datasets.

[0081] Specifically, the following steps are included:

[0082] S121. Because the units used for the same products are inconsistent in multiple transactions, they need to be changed to a unified unit. For example, if the same product uses kilograms and tons respectively, the unit should be changed to "ton".

[0083] S122. Some transaction orders have serious data loss or missing important information (such as unit price, unit of use, etc.), so it is necessary to delete the data in that row. The method is as follows: count the number of empty values ​​in each row. When the proportion of empty values ​​is greater than 30%, delete that row.

[0084] S223. Obvious errors were found in some data entries. Obtain the numerical data (datan) for each column, along with the corresponding mean and standard value (sigma). Then, use the elimination formula:

[0085] diff{|datan-mean|>3×sigma}

[0086] Outliers with a difference greater than 3 times the standard value are removed.

[0087] The correct field information for various product datasets is displayed vertically as follows:

[0088]

[0089]

[0090] S3. Integrate or remove duplicate or discrete data from various product datasets to obtain unique product datasets; this includes the following steps:

[0091] S31. Count the number of orders for each type of product and remove products with fewer than 3 orders.

[0092] S32. Find data that has two or more identical rows, remove duplicate data, and obtain a dataset of each product that is unique.

[0093] S4. Aggregate and calculate the data of each unique commodity dataset using "week" as the time unit to obtain the continuous weekly index for each commodity; this includes the following steps:

[0094] S41. For non-repeating data sets of various products, change the delivery date within the same week to Monday of that week.

[0095] S42. Calculate the weekly index for various commodities:

[0096] S421. Count the quantity n of each type of commodity, obtain the price information of each commodity i, and divide the price at time t by the initial price of commodity i to obtain the price change R of commodity i at time t. ti ;

[0097] S422. Assign different weights W to different products i according to demand. i ;

[0098] S423. Accumulate the price changes (R values) of multiple commodities (n) within the same time period (t), multiply by a base of 1000, and obtain the weekly index It; the formula for calculating the weekly index It is as follows:

[0099]

[0100]

[0101] S43. Combine each weekly index to form a continuous weekly index;

[0102] S5. Transfer the continuous weekly indices of various commodities to the price index database for updating, and update and publish the continuous weekly index views of various commodities. Specifically, the price index database uses the FineBI database. It should be noted that the FineBI database consists of the existing FineBI business intelligence software, which is a business intelligence product launched by FineReport Software Co., Ltd. The usage and functionality of this product will not be elaborated here.

[0103] In summary, because the ceramic raw material trading industry is a traditional and closed sector, product pricing is determined through negotiation between buyers and sellers and has historically relied on word-of-mouth. This has resulted in the lack of electronic, information-based, and online methods for collecting and storing industry transaction and transportation information. Consequently, price indices and other technologies are absent from the ceramic raw material trading industry. Therefore, how to effectively utilize transaction data from the ceramic industry to create price indices has become a crucial and urgent issue that the industry needs to address.

[0104] This invention proposes a method for reproducing ceramic raw material price indices. Combining big data thinking and technology, it utilizes an industrial internet platform to acquire data generated during transportation and transactions. After multiple processing steps, it accurately obtains continuous weekly indices for various commodities. These continuous weekly indices are then continuously updated and visualized into a continuous weekly index view. Figure 2 As shown, this allows us to review the price trends of commodities in different periods, which helps guide market participants in their transactions and is a necessary technological means for the digitalization and standardization of industries.

[0105] The technical principles of the present invention have been described above with reference to specific embodiments. These descriptions are merely for explaining the principles of the invention and should not be construed as limiting the scope of protection of the invention in any way. Based on this explanation, those skilled in the art can readily conceive of other specific embodiments of the invention without inventive effort, and these equivalent variations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method of replicating a ceramic raw material price index, characterized by, Comprise the following steps: S1, obtain the structured data of the basic properties of ceramic raw materials in the database of the industrial internet platform, import the structured data of the basic properties of ceramic raw materials into the pre-trained ceramic raw material classification model for standard classification, and obtain the data set of each type of ceramic raw material; S2, obtain the historical transaction data, logistics information aggregation data and terminal customer basic information of the industrial internet platform in the database of the industrial internet platform, and perform initial processing on the above data and the data set of each type of ceramic raw material to obtain the data set of each type of commodity; S3, integrate or remove the repeated or discrete data in the data set of each type of commodity to obtain the non-repeated data set of each type of commodity; S4, aggregate and calculate the data of the non-repeated data set of each type of commodity in units of "week" to obtain the continuous week index of each type of commodity; S5, transmit the continuous week index of each type of commodity to the price index library for updating, and update and publish the continuous week index view of each type of commodity; In S1, the ceramic raw material classification model specifically comprises the following pre-training steps: S11, collect data sources: obtain the structured data of the basic properties of historical ceramic raw materials in the database of the industrial internet platform, and the structured data of the basic properties of the ceramic raw materials includes commodity, raw material composition and form information; S12, logically associate according to the data set screening logic, and perform data topology on the data source according to the logical association to obtain a data set; wherein the data set screening logic is commodity category, commodity subcategory and commodity secondary subcategory, commodity, raw material composition and form information; S13, clean the data set to obtain a cleaned basic data set; S14, perform data feature calculation on the basic data set to obtain a data set with completed feature engineering; S15, divide the data set with completed feature engineering into a training set and a test set; S16, based on the XGBoost machine learning model, train and establish a ceramic raw material classification model using the training set, and then test the ceramic raw material classification model using the test set; wherein the function of the ceramic raw material classification model is to input raw material composition and form information and output predicted commodity category, commodity subcategory and commodity secondary subcategory; S17, evaluate the effect of the ceramic raw material classification model according to the prediction error score of the training set and the test set; when the prediction error score reaches the prediction value, it is judged that the ceramic raw material classification model is available; when the prediction error score does not reach the prediction value, it is judged that the ceramic raw material classification model is not available, the model parameters of the ceramic raw material classification model are adjusted, and S15-S16 are re-executed; The historical transaction data of the industrial internet platform includes the following fields: supplier, cargo using unit, order number, transaction time and unit price; The logistics information aggregation data includes the following fields: transportation time, unit and quantity; The terminal customer basic information includes the following fields: terminal customer name, cargo using unit and industry to which it belongs; In S2, the historical transaction data, logistics information aggregation data, terminal customer basic information and various ceramic raw material data sets of the industrial internet platform are initially processed to obtain various commodity data sets, specifically including the following steps: S21, respectively, the historical transaction data, logistics information aggregation data and terminal customer basic information are associated with various ceramic raw material data sets through association conditions and processed to obtain various commodity data sets; wherein the association conditions include order number, commodity name and terminal customer name; S22, the various commodity data sets are analyzed, and after analysis, the units are unified, the missing values and abnormal values are removed, and the correct various commodity data sets are obtained; In S3, the repeated or discrete data in the various commodity data sets are integrated or removed to obtain non-repeated various commodity data sets; specifically including the following steps: S31, the number of orders corresponding to each type of commodity is counted, and products with less than 3 orders are removed; S32, find the data with two or more same rows, and remove the repeated data to obtain non-repeated various commodity data sets; In S4, the data of the non-repeated various commodity data sets are aggregated and calculated with "week" as the time unit to obtain the continuous week index of each type of commodity; specifically including the following steps: S41, the dates in the same week of the delivery time in the non-repeated various commodity data sets are uniformly changed to Monday of the week; S42, the week index of each type of commodity is calculated: S421、Statistics of various types of goods, the number of goods n, to obtain each commodity i price information, at time t price, divided by the initial price of commodity i, thus obtaining the time t price change value R of commodity i ti ; S422, different weights W are given to different commodities i according to demand i ; S423, the change R value of the price P of the multiple commodities n in the same time period t is accumulated, multiplied by the base 1000, and the week index It is obtained; the calculation formula of the week index It is as follows: S43, after each week index is combined, the continuous week index is formed.

2. The method of claim 1, wherein the ceramic raw material price index is reproduced by the following equation: ###0001### wherein, X is the ceramic raw material price index, Y is the price of the ceramic raw material, and n is the number of the ceramic raw material. In S13, the data cleaning includes format content cleaning and abnormal value cleaning.

3. The method of claim 2, wherein the ceramic raw material price index is reproduced by the following equation: ###0001### wherein, X is the ceramic raw material price index, Y is the price of the ceramic raw material, and n is the number of the ceramic raw material. In S14, the data feature calculation includes digital standardization of commodity, raw material composition and form information, and calculation of data features.

4. The method of claim 3, wherein the ceramic raw material price index is reproduced by the following equation: ###0001### wherein, X is the ceramic raw material price index, Y is the price of the ceramic raw material, and n is the number of the ceramic raw material. In S17, the prediction error score of the training set and the test set includes the model score R2_SCORE, and the calculation formula of R2_SCORE is as follows: where R2 is the prediction error score; y is the actual value; is the predicted value for the ith value; y(i) is the true value for the ith value; is the average value of y.

5. The method of claim 4, wherein the ceramic raw material price index is reproduced by the following equation: ###0001### wherein, X is the ceramic raw material price index, Y is the price of the ceramic raw material, and n is the number of the ceramic raw material. In S17, the model parameters of the ceramic raw material classification model include the tree model iteration number n_estimators, the tree depth of the tree model max_depth, and the loss function threshold value gamma when splitting nodes.

Images