Coal market analysis and purchase decision-making method and system based on big data

Abstract

The invention discloses a coal market analysis and purchase decision-making method and system based on big data, and belongs to the technical field of coal market data analysis, and the method comprises the steps: generating an integrated market analysis data set through multi-source data collection and data integration and cleaning; based on the integrated market analysis data set, data analysis and mining are carried out through index calculation, correlation analysis and trend prediction, and an analysis result containing market dynamics, price trend and supply conditions is obtained; displaying a preset report template based on the market analysis, the purchase suggestion and the data; and based on an analysis result, generating a coal purchase decision support report in combination with a report template and an automatic filling technology, and outputting the report in a specified format. According to the invention, factors in multiple aspects such as coal quality, purchase cost, transportation mode and supplier capability are comprehensively considered, and accurate and reliable coal market analysis and purchase decisions are provided for users through data acquisition and integration, data analysis and mining and automatic analysis report generation.

Description

Technical Field

[0001] The present invention belongs to the technical field of coal market data analysis, and particularly relates to a coal market analysis and procurement decision-making method and system based on big data. Background Art

[0002] In the coal market, power enterprises need to accurately understand market dynamics, price trends, and supply situations in order to formulate reasonable procurement strategies and inventory management plans. Previous analysis methods often relied on manual data collection and collation, which were inefficient and prone to errors, and could not meet the requirements of enterprises for the timeliness and accuracy of market analysis.

[0003] As an improvement, there have emerged many software processing systems for coal market analysis and procurement decision-making. However, at present, the existing coal data processing is cumbersome: in the face of a large amount of coal market data, such as coal price indices, production information of each coal mine, inventory situations, etc., the cost of manually processing and analyzing these data is high, time-consuming, and prone to errors. The analysis depth is limited: traditional analysis methods are difficult to deeply mine and correlate analyze data, and cannot accurately grasp market trends and cost influencing factors, resulting in the lack of scientificity and accuracy in procurement decisions. Report generation is lagging: manually generating analysis reports requires a lot of time and effort, cannot reflect market changes in a timely manner, and affects the decision-making efficiency and competitiveness of enterprises. Therefore, in summary, in the current coal data analysis and processing, there are problems such as relatively cumbersome data processing, difficulty in deeply mining and correlating analyzing data, and lagging report generation, which affect the decision-making efficiency and competitiveness of users. Summary of the Invention

[0004] The present invention provides a coal market analysis and procurement decision-making method and system based on big data, aiming to solve the problems in the current coal data analysis and processing, such as relatively cumbersome data processing, difficulty in deeply mining and correlating analyzing data, and lagging report generation, which in turn affect the decision-making efficiency and competitiveness of users.

[0005] To achieve the above object, the present invention adopts the following technical solutions: The present invention provides a coal market analysis and procurement decision-making method based on big data, including the following steps: S1. Through multi-source data collection, data integration and cleaning, generate an integrated market analysis data set and store it in a hierarchical database; The multi-source data includes coal price indices, coal mine production information, inventory data, transportation data, and power generation data in the power market; S2. Based on the integrated market analysis data set, conduct data analysis and mining through indicator calculation, association analysis, and trend prediction to obtain analysis results including market dynamics, price trends, and supply situations; Indicator calculation includes statistical calculations of coal price indices, inventory turnover rates, and transportation costs. Association analysis uses association rule mining algorithms, and trend prediction uses time series models or machine learning models; S3. Preset report templates based on market analysis, procurement suggestions, and data display; Based on the analysis results, generate a coal procurement decision support report by combining the report template and automatic filling technology, and output it in a specified format.

[0006] In some embodiments, in S1, multi-source data collection specifically includes: using web crawlers to capture web data, and configuring fault tolerance mechanisms for web crawlers to handle network exceptions; using Web API interfaces to call and obtain real-time data at preset frequencies; using database direct connection technology to extract structured data from coal mine production management systems, inventory management systems, and transportation information systems.

[0007] In some embodiments, in S1, the integration and cleaning of data include duplicate removal processing, error correction processing, and missing value filling; among them: Duplicate removal processing removes duplicate data records based on unique fields; error correction processing verifies the rationality of data through value range checks and corrects contradictory data through logical relationship checks; missing value filling fills numerical data with means or medians, and fills key indicators with model predictions.

[0008] In some embodiments, in S1, the hierarchical database includes original data tables, cleaned data tables, and analysis result tables, and the hierarchical database uses a relational database or a non-relational database; Among them, the relational database is MySQL or Oracle, and the non-relational database is MongoDB or Cassandra.

[0009] In some embodiments, in S2, indicator calculation includes: statistical calculations of coal price indices, inventory turnover rates, and transportation costs, and generates statistical reports based on preset periods.

[0010] In some embodiments, in S2, the association rule mining algorithm uses the Apriori algorithm or the FP-Growth algorithm, and the support and confidence thresholds are dynamically adjusted according to data characteristics.

[0011] In some embodiments, in S2, trend prediction uses time series models or machine learning models. The time series models are ARIMA, SARIMA, or Prophet; the machine learning models are multi-layer perceptrons, convolutional neural networks, or recurrent neural networks.

[0012] In some embodiments, S2 further includes cluster analysis, regression analysis, and model training optimization. The cluster analysis uses the K-Means algorithm or hierarchical clustering algorithm to segment coal mine production information and market data. The regression analysis uses linear regression, polynomial regression, or ridge regression models to establish the relationship between coal prices and inventory. For model training optimization, historical data is divided into training sets, validation sets, and test sets. Hyperparameters are adjusted and feature selection is performed, with the mean squared error or coefficient of determination as the evaluation metric.

[0013] In some embodiments, in S3, the report template is customized through a visual editor, including a cover page, table of contents, data charts, and analysis and interpretation modules. The template is classified and maintained through a template management system. The automatic filling technology includes extracting index calculation results, association rules, and predicted values from the analysis results and filling them into the corresponding positions in the template. A report is generated through the Free marker or Velocity template engine, supporting format conversion and multi-terminal display.

[0014] The present invention also provides a coal market analysis and procurement decision-making system based on big data. The system includes a data integration module, a data analysis module, and a decision-making generation module, where: Data integration module: used to generate an integrated market analysis data set through multi-source data collection, data integration, and cleaning, and store it in a hierarchical database. The multi-source data includes coal price indices, coal mine production information, inventory data, transportation data, and power generation data in the power market. Data analysis module: used to perform data analysis and mining based on the integrated market analysis data set through index calculation, association analysis, and trend prediction, and obtain analysis results including market dynamics, price trends, and supply situations. Index calculation includes statistical calculations of coal price indices, inventory turnover rates, and transportation costs. Association analysis uses association rule mining algorithms, and trend prediction uses time series models or machine learning models. Decision-making generation module: used to preset a report template based on market analysis, procurement suggestions, and data display; generate a coal procurement decision support report based on the analysis results, combined with the report template and automatic filling technology, and output it in a specified format.

[0015] Compared with the prior art, the coal market analysis and procurement decision-making method and system based on big data of the present invention have the following beneficial effects: The coal market analysis and procurement decision-making method based on big data of the present invention comprehensively considers various factors in coal procurement, providing comprehensive and accurate information support for procurement decisions. The present invention collects data through multiple channels, ensuring the comprehensiveness and timeliness of the data. It not only reduces the workload of manual data collection and collation but also avoids errors that may be brought about by manual operations, improving the accuracy and reliability of the data. The present invention integrates the massive data collected, uniformly manages and processes the scattered data in different channels and different formats to form a complete market analysis data set. The present invention uses a data cleaning algorithm to remove duplicate, incorrect, and invalid data, ensuring the quality of the data. This helps improve the accuracy and credibility of data analysis and provides reliable basic data support for decision-making.

[0016] In addition, the present invention adopts a variety of analysis methods to deeply mine the coal market data and provide a basis for decision-making. The present invention analyzes and predicts complex market data by using data mining algorithms, helping users better understand the market segmentation structure, predict market change trends and cost situations, and improving the accuracy and reliability of the analysis. Finally, the present invention presets a template for the analysis report, standardizes the report structure, and provides users with comprehensive market information. The data calculated and mined during the data analysis process is automatically filled into the corresponding positions in the report template to form the preliminary report content, greatly reducing the workload of manually writing the report and improving the efficiency of report generation. The key data and analysis results are deeply interpreted in the report to provide decision-making support for users. Through professional analysis and interpretation, it helps users better understand market dynamics, make reasonable decisions, and has better applicability. Brief Description of the Drawings

[0017] The accompanying drawings in the specification are used to provide a further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0018] Figure 1 It is a schematic flow chart of a coal market analysis and procurement decision-making method based on big data of the present invention; Figure 2 It is a schematic diagram of the data collection and processing flow in a coal market analysis and procurement decision-making method based on big data of the present invention; Figure 3 It is a schematic diagram of the data analysis and mining flow in a coal market analysis and procurement decision-making method based on big data of the present invention; Figure 4 It is a schematic diagram of the analysis report generation flow in a coal market analysis and procurement decision-making method based on big data of the present invention; Figure 5This is a schematic diagram of the architecture of the system in an embodiment of a coal market analysis and procurement decision-making system based on big data according to the present invention. Detailed implementation manners

[0019] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. Components of the embodiments of the present invention described and illustrated in the accompanying drawings here generally may be arranged and designed in a variety of different configurations.

[0020] Therefore, the detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0021] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0022] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper", "lower", "horizontal", "inner", etc. are used to indicate the orientation or positional relationship, it is based on the orientation or positional relationship shown in the accompanying drawings or the orientation or positional relationship in which the product of the present invention is usually placed during use. This is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be construed as a limitation to the present invention. In addition, terms such as "first", "second", etc. are only used for descriptive distinction and cannot be construed as indicating or implying relative importance.

[0023] In addition, if the term "horizontal" appears, it does not mean that the component is required to be absolutely horizontal, but it can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but it can be slightly inclined.

[0024] In the description of the embodiments of the present invention, it should also be noted that unless otherwise clearly specified and limited, if the terms "set", "installed", "connected", "connected" appear, they should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0025] As Figures 1-4 shown, a method for coal market analysis and procurement decision-making based on big data according to the present invention includes the following steps: S1. Through multi-source data collection, data integration and cleaning, generate an integrated market analysis data set and store it in a hierarchical database; The multi-source data includes the price index of coal, the production information of coal mines, inventory data, transportation data, and power generation data in the power market; S2. Based on the integrated market analysis data set, perform data analysis and mining through index calculation, association analysis and trend prediction to obtain analysis results including market dynamics, price trends and supply situations; The index calculation includes the statistical calculation of the coal price index, inventory turnover rate and transportation cost. The association analysis uses the association rule mining algorithm, and the trend prediction uses the time series model or machine learning model; S3. Preset a report template based on market analysis, procurement suggestions and data display; based on the analysis results, combine the report template and automatic filling technology to generate a coal procurement decision support report and output it in a specified format.

[0026] The present invention comprehensively considers various factors such as coal quality, procurement cost, transportation mode and supplier ability, and realizes providing accurate and reliable coal market analysis and procurement decision-making for users through data collection and integration, data analysis and mining, and automatic generation of analysis reports.

[0027] Data collection and integration of the present invention; Data sources: Collect coal market data from multiple channels, including the price index of the coal market, the production information of each coal mine, inventory data, transportation conditions, power generation data in the power market, etc. Data collection technology: Use technologies such as web crawlers, web api data interfaces, and direct database connections to realize automatic data collection and real-time update to ensure the timeliness and integrity of the data. Data integration and cleaning: Integrate and clean the collected data, remove duplicate, incorrect and invalid data, and ensure the accuracy and consistency of the data.

[0028] In data analysis and mining, the analysis methods include: Index Calculation and Statistics: Calculate various coal market indicators such as coal price index, inventory turnover rate, transportation cost, etc., and conduct statistical analysis to understand the overall market trends and changing patterns. Association Analysis: Explore the correlation relationships between different data, such as the relationship between coal prices and market supply and demand, the production situation of coal mines and inventory changes, etc., to provide a basis for market analysis and procurement decisions. Trend Prediction: Use techniques such as time series analysis and machine learning algorithms to predict trends in coal market prices, inventory, etc., to provide a reference for enterprises to formulate procurement plans and inventory management strategies.

[0029] Data mining algorithms also include clustering analysis and regression analysis. Clustering Analysis: Cluster similar coal mines, market situations, etc., in order to better understand the market segmentation structure and characteristics. Regression Analysis: Establish a regression model between coal price, inventory and other indicators and related factors to predict market change trends and cost situations. Neural Network Analysis: Utilize the non-linear fitting ability of neural networks to analyze and predict complex market data, improving the accuracy and reliability of the analysis.

[0030] Automatic generation of analysis reports specifically includes report template definition, data filling and analysis, and report generation and output. Report template definition is used to define the template of the analysis report, including the structure, content, format, etc. of the report, ensuring the standardization and consistency of the report.

[0031] Among them, data filling and analysis include data filling and analysis interpretation. Data filling: Fill the data calculated and mined during the analysis process into the corresponding positions in the report template to form the preliminary report content. Analysis interpretation: Interpret the key data and analysis results in the report, explain market trends, cost changes, etc., to provide decision-making support for enterprises.

[0032] Report generation and output include a report generation engine and report output formats. Develop a report generation engine to automatically generate an analysis report based on the report template and the filled data. The report supports multiple report output formats such as PDF, Excel, HTML, etc., facilitating users to read and use.

[0033] In some embodiments, the present invention provides comprehensive and accurate information support for procurement decisions. First, the present invention improves the efficiency and accuracy of data processing. Using technologies such as web crawlers, data interfaces, database connections, etc., collect data from multiple channels such as coal market price indices, production information of each coal mine, inventory data, transportation conditions, and power generation data in the power market, ensuring the comprehensiveness and timeliness of the data. It not only greatly reduces the workload of manual data collection and collation, but also avoids errors that may be brought by manual operations, improving the accuracy and reliability of the data.

[0034] Integrate the massive data collected, unify the management and processing of data scattered in different channels and in different formats, and form a complete market analysis data set. Effectively clean the data: Remove duplicate, incorrect, and invalid data through data cleaning algorithms to ensure the quality of the data. This helps improve the accuracy and credibility of data analysis and provides reliable basic data support for decision-making.

[0035] Furthermore, the present invention adopts various analysis methods such as index calculation and statistics, association analysis, and trend prediction to deeply mine the coal market data. For example, through association analysis, mine the associated factors such as the relationship between coal prices and market supply and demand, and the relationship between coal mine production and inventory changes, etc., to provide a basis for decision-making. The present invention uses advanced data mining algorithms such as clustering analysis, regression analysis, and neural network analysis to analyze and predict complex market data. Clustering analysis clusters similar coal mines, market conditions, etc., to help enterprises better understand the market segmentation structure; regression analysis establishes a regression model between indicators such as coal prices and inventory and related factors to predict market change trends and cost situations; neural network analysis utilizes its non-linear fitting ability to improve the accuracy and reliability of analysis.

[0036] The present invention can realize the automation of analysis reports. By defining the template of the analysis report and standardizing the report structure, the report template covers all aspects of the coal market, including market dynamics, price trends, supply situations, procurement suggestions, etc., and provides users with comprehensive market information. The present invention automatically fills the data calculated and mined during the data analysis process into the corresponding positions in the report template to form the preliminary report content. This greatly reduces the workload of manually writing reports and improves the efficiency of report generation. The present invention deeply interprets the key data and analysis results in the report, explains market trends, cost changes, etc., to provide decision-making support for enterprises. Through professional analysis and interpretation, it helps enterprise managers better understand market dynamics and make scientific and reasonable decisions. And the present invention adopts diversified report output formats: supports multiple report output formats such as PDF, Excel, HTML, etc., to meet the various needs of users and facilitate users to select the appropriate report form according to their own needs and usage habits.

[0037] The present invention also provides a coal market analysis and procurement decision-making system based on big data. The system includes a data integration module, a data analysis module, and a decision generation module, where: The data integration module: is used to generate an integrated market analysis data set through multi-source data collection, data integration and cleaning, and store it in a hierarchical database; The multi-source data includes coal price indices, coal mine production information, inventory data, transportation data, and power generation data in the power market; Data analysis module: Used for data analysis and mining based on the integrated market analysis data set through index calculation, association analysis, and trend prediction, and obtaining analysis results including market dynamics, price trends, and supply situations; Index calculation includes statistical calculations of coal price indices, inventory turnover rates, and transportation costs. Association analysis uses association rule mining algorithms, and trend prediction uses time series models or machine learning models; Decision generation module: Used for presetting report templates based on market analysis, procurement suggestions, and data display; generating a coal procurement decision support report based on the analysis results, combined with the report template and automatic filling technology, and outputting it in a specified format.

[0038] As Figure 5 shown, as an embodiment, the system architecture design of the coal market analysis and procurement decision support system based on big data realizes the full-process automation of data acquisition, processing, analysis, and report generation for each component and the relationships between them.

[0039] The system architecture includes a data layer, a data processing layer, and an application layer. The data layer includes data sources and data storage and management; The data sources include: establishing connections with multiple data sources, including price index platforms in the coal market, production management systems of each coal mine, inventory management systems, transportation information systems of transportation companies, power generation data acquisition systems in the power market, etc.

[0040] In data storage and management, database selection: According to the data volume and data processing requirements, select a suitable database management system, such as a relational database (such as MySQL, Oracle) or a non-relational database (MongoDB, Cassandra). Data storage method: Adopt a hierarchical storage method to store the original data, processed data, and analysis results in different tables or collections respectively to ensure data independence and scalability.

[0041] In the data acquisition module of the data processing layer, web crawler development: Use web crawler technology to write crawler programs to regularly crawl coal market data from various data source websites. The crawler program needs to have efficient network request processing capabilities, data parsing capabilities, and error handling mechanisms to ensure the timeliness and accuracy of data acquisition. Data interface integration: For some data sources that provide data interfaces, develop data interface clients to achieve seamless integration with the data sources and directly obtain data from the interfaces. Database connection and data import: Import the collected data into the corresponding tables or collections in the data storage layer through database connection technology.

[0042] Data cleaning and integration module; Implementation of data cleaning algorithms: Design data cleaning algorithms to perform operations such as duplicate removal, error correction, and filling missing values on the collected data. For example, for duplicate data records, perform duplicate removal by comparing unique fields of the data; for incorrect data values, correct them according to the value range and logical relationships of the data; for missing data values, use methods such as mean, median, or other appropriate methods to fill them. Data integration and association: Integrate the cleaned data according to certain rules, associate data in different tables or sets, and form a complete market analysis data set. For example, associate the production information of coal mines with inventory information and price information to analyze the impact of coal mine production on inventory and prices.

[0043] In the data analysis and mining module, Index calculation and statistics module: Develop an index calculation and statistics module to calculate and statistically analyze market data according to preset index calculation rules. For example, calculate indicators such as coal price index, inventory turnover rate, and transportation cost, and generate corresponding statistical reports. Association analysis module: Implement the association analysis function to mine the association relationships between different data through data mining algorithms. For example, use the Apriori algorithm or FP-Growth algorithm to mine association rules such as the relationship between coal prices and market supply and demand, and the relationship between coal mine production and inventory changes. Trend prediction module: Use time series analysis and machine learning algorithms to implement the trend prediction function. For example, use models such as ARIMA, SARIMA, and Prophet to predict trends in data such as coal market prices and inventory.

[0044] In the implementation of data mining algorithms, Clustering analysis module: Develop a clustering analysis module to implement the clustering analysis function. Through clustering algorithms, cluster similar coal mines, market situations, etc. to form different market segmentation groups. For example, use algorithms such as K-Means algorithm and hierarchical clustering algorithm to perform clustering analysis on coal mines.

[0045] Regression analysis module: Implement the regression analysis function to establish regression models between indicators such as coal prices and inventory and related factors. For example, use algorithms such as linear regression, polynomial regression, and ridge regression to establish regression models to predict market change trends and cost situations. Neural network analysis module: Develop a neural network analysis module to utilize the non-linear fitting ability of neural networks to analyze and predict complex market data. For example, use neural network models such as multi-layer perceptron (MLP), convolutional neural network (CNN), and recurrent neural network (RNN) to analyze and predict market data.

[0046] In the system architecture of the embodiments of the present invention, a user interface is designed at the application layer. By designing a simple and friendly user interface, functions such as data query, analysis report generation, and system settings are provided. The user interface should display data and analysis results in a visual way to facilitate user understanding and operation.

[0047] Among them, in the analysis report generation module, report template definition and management: Define the templates of analysis reports, including the structure, content, format, etc. of the reports. The templates can be customized through a visual editor to facilitate users to modify and expand according to their own needs. At the same time, establish a report template management system to classify, manage, and maintain different types of report templates. Report data filling and generation: According to the user's needs, extract the corresponding data from the data analysis results and fill it into the corresponding positions in the report template to generate an analysis report. The report generation process can be automated, and reports can be quickly generated according to the preset report generation rules and processes.

[0048] Report output and display: Support multiple report output formats, such as PDF, Excel, HTML, etc. Output the generated report in the format selected by the user and display it on the user interface. Users can use the report by printing, saving, viewing online, etc.

[0049] The following further details a method and system for coal market analysis and procurement decision-making based on big data of the present invention through specific embodiments.

[0050] As Figure 2 shown, the data collection and processing process details the whole process from communicating with each department of the enterprise to determine data requirements and set the collection frequency, to collecting data, storing and backing up using web crawlers and data interfaces, and then cleaning and integrating and associating the data. Among them: In the formulation of the data collection plan, data requirement determination: Communicate with relevant departments such as the fuel management department, procurement department, and production department of the enterprise to understand their data requirements for the coal market, and determine the data indicators and data scope to be collected. Collection frequency setting: Set the data collection frequency according to the importance and change frequency of the data. For important data indicators, such as coal price index, inventory turnover rate, etc., a higher collection frequency should be set to ensure the timeliness of the data; for some relatively stable data indicators, such as the basic information of coal mines, the collection frequency can be appropriately reduced.

[0051] During data collection execution, the web crawler runs: According to the data collection plan, start the web crawler program to regularly scrape coal market data from various data source websites. The web crawler should have good stability and fault tolerance, and be able to handle network anomalies and data scraping failures. Data interface call: For data sources that provide data interfaces, call the data interface client, send data requests according to the requirements of the interface document, and obtain data. The data interface call should ensure the security and legality of the data, and avoid data leakage and illegal access. Data storage and backup: Store the collected data in the corresponding tables or collections in the data storage layer, and perform data backup. The data backup should be carried out regularly to ensure the security and recoverability of the data.

[0052] Data cleaning and integration. In the formulation of data cleaning rules, duplicate data processing: Formulate duplicate data processing rules, and remove duplicate data records by comparing the unique fields of the data. In error data correction, value range check: Check the value range of the data to ensure that the data values are within a reasonable range. For example, for coal price data, check whether the price is positive and within a reasonable price range. Logical relationship check: Check whether the logical relationships between the data are correct. For example, for inventory data and production data, check whether the inventory changes match the production situation.

[0053] In missing data filling, mean filling: For numerical data, if the missing data is less, the mean filling method can be used to replace the missing data values with the mean of the data in that column. Model prediction filling: For some important data indicators, if there is a lot of missing data, the model prediction method can be used to establish a model using the existing data and predict and fill the missing data.

[0054] In data integration and association, data table association: According to the relationships between the data, associate the data in different tables. For example, associate the production information table of coal mines with the inventory information table and the price information table, and connect them through fields such as the coal mine number or name to form a complete market analysis data set. Data fusion: Perform fusion processing on the associated data, remove redundant information, and extract useful information. For example, for the same data indicators provided by multiple data sources, perform data fusion to ensure the consistency and accuracy of the data.

[0055] As Figure 3 shown, the data analysis and mining process is an analysis process from determining market indicators and calculation rules and performing indicator calculation and statistics, to selecting association algorithms to mine data association relationships, and then to selecting prediction models to perform trend prediction and model training optimization evaluation on the data. Among them: Indicator calculation and statistics; Definition of indicators and determination of calculation rules; Definition of market indicators: Determine the market indicators to be calculated, such as coal price index, inventory turnover rate, transportation cost, etc. Clearly define the calculation methods and formulas for each indicator to ensure the comparability and accuracy of the calculation results. Setting of statistical period: Set the statistical period of the indicators, such as daily, weekly, monthly, quarterly, annually, etc. Select a suitable statistical period according to the characteristics of the data and the analysis requirements to understand the market dynamics and trends in a timely manner.

[0056] Calculation of indicators and implementation of statistics; Data reading and processing: Read the data required for calculating indicators from the data storage layer, and perform data cleaning and preprocessing to ensure the quality and integrity of the data. Indicator calculation: Calculate the data according to the determined calculation methods and formulas to obtain the values of each indicator. Generation of statistical reports: Conduct statistical analysis on the calculated indicator values to generate statistical reports. Statistical reports can be presented in the form of charts, such as bar charts, line charts, pie charts, etc., to make the data more intuitive and clear.

[0057] Association analysis; Selection of association rule mining algorithms; Common association algorithms: Select suitable association analysis algorithms, such as Apriori algorithm, FP-Growth algorithm, etc. These algorithms can mine the association relationships between data from a large amount of data, providing a basis for market analysis and decision-making.

[0058] Setting of algorithm parameters; Setting of support and confidence: Set the support and confidence thresholds of association rules according to the characteristics of the data and the analysis requirements. Support represents the probability that two item sets appear simultaneously in the data set, and confidence represents the probability that another item set appears simultaneously when one item set appears. By adjusting the support and confidence thresholds, different levels of association rules can be mined. Setting of the number of iterations: For some complex data sets, it is necessary to set an appropriate number of iterations to ensure that the algorithm can converge to the optimal solution.

[0059] Mining and analysis of association rules; Data preprocessing: Preprocess the data and convert it into a format suitable for association analysis. For example, perform discretization processing on the data to convert continuous data into discrete data for algorithm calculation. Mining of association rules: Use the selected association analysis algorithm to mine association rules from the data. The algorithm will mine the association rules that meet the support and confidence thresholds in the data set and display these rules in the form of a list. Analysis of association rules: Analyze the mined association rules to explain the meaning and significance of the rules. For example, by analyzing the association rules between coal prices and market supply and demand relationships, understand the reasons and influencing factors for market price fluctuations.

[0060] Trend prediction; Prediction model selection; Time series analysis model: Select a suitable time series analysis model, such as ARIMA model, SARIMA model, Prophet model, etc. These models can model and predict time series data, and analyze the trend, seasonality, and periodic change laws of the data. Machine learning prediction model: Consider using machine learning algorithms, such as neural networks, support vector machines, decision trees, etc., to establish a prediction model. These models have strong non-linear fitting ability and generalization ability, and can predict complex market data.

[0061] Prediction model training and optimization; Data preparation: Organize and preprocess historical data, and divide the data into training set, validation set, and test set. The training set is used to train the model, the validation set is used to adjust the model parameters, and the test set is used to evaluate the performance of the model. Model training: Use the training set data to train the selected prediction model, and adjust the model parameters to make the model better fit the historical data.

[0062] Model evaluation and optimization; Model evaluation metrics: Select appropriate model evaluation metrics, such as mean squared error (MSE), mean absolute error (MAE), coefficient of determination (R²), etc., to evaluate the trained model. The evaluation metrics can reflect the prediction accuracy and performance of the model. Model optimization: According to the model evaluation results, optimize and adjust the model. For example, adjust the model parameters, increase the complexity of the model, select better features, etc., to improve the prediction performance of the model.

[0063] As Figure 4 shown, the analysis report generation process; The analysis report generation process starts with designing the report template (covering structure, content elements, and format specifications), goes through extracting data from the analysis results and filling it into the template, processing it using the report generation engine, checking the format specifications, and finally outputting and presenting the report.

[0064] Report template definition; Report structure design: Design the structure of the analysis report, including the cover, table of contents, preface, market analysis, data interpretation, conclusions and recommendations, appendix, etc. The content and format of each part should be clearly defined to ensure that the structure of the report is clear and hierarchical.

[0065] Report content element definition; Data display elements: Determine the data elements to be displayed in the report, such as coal price index, inventory turnover rate, production information of each coal mine, inventory status, etc. Clearly define the display methods and formats for each data element, such as charts, tables, text descriptions, etc. Analysis and interpretation elements: Define the content of data analysis and interpretation, including market trend analysis, cost change analysis, correlation analysis, etc. Clearly define the writing requirements and key points for each analysis and interpretation element to ensure that the report can deeply analyze the market situation and provide support for decision-making.

[0066] Report format specifications; Font and font size: Specify the font and font size of the report, generally using Song typeface and small four font size to ensure the readability of the report. Line spacing and page margins: Set appropriate line spacing and page margins to make the report page layout reasonable, beautiful and generous. Chart format: For the charts used in the report, specify the format requirements for chart types, sizes, colors, titles, axis labels, etc. to ensure that the charts can clearly display data and analysis results.

[0067] Data filling and report generation; Data extraction and filling; Analysis result extraction: Extract the data that needs to be filled into the report from the data analysis results, including index calculation results, correlation analysis results, trend prediction results, etc. Ensure that the extracted data is accurate and error-free and closely related to the report content. Data filling: Fill the extracted data into the corresponding positions according to the requirements of the report template. When filling data, attention should be paid to the consistency and accuracy of the data to avoid data errors or conflicts.

[0068] Report generation engine implementation; Template engine selection: Select a suitable template engine, such as Freemarker, Velocity, etc., to achieve automatic report generation. The template engine can integrate the report template with the filled data to generate the final analysis.

[0069] Finally, it should be noted that the above description is only the preferred embodiment of the present invention and does not impose any form of limitation on the present invention; any ordinary technical personnel in the industry can smoothly implement the present invention according to the description in the specification and the above description. Any equivalent changes made by slightly modifying and evolving the technical content disclosed above are equivalent embodiments of the present invention; at the same time, any equivalent changes made to the above embodiments based on the essential technology of the present invention, including modifications and evolutions, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for coal market analysis and procurement decision-making based on big data, characterized in that The steps include: S1. Generate an integrated market analysis data set through multi-source data collection, data integration and cleaning, and store it in a hierarchical database; Multi-source data includes coal price index, coal mine production information, inventory data, transportation data, and power generation data from the electricity market; S2. Based on the integrated market analysis data set, data analysis and mining are performed through indicator calculation, correlation analysis, and trend forecasting to obtain analytical results including market dynamics, price trends, and supply conditions; Indicator calculations include statistical calculations of coal price index, inventory turnover rate, and transportation costs. Association analysis uses association rule mining algorithms, and trend forecasting uses time series models or machine learning models. S3, preset report templates based on market analysis, procurement recommendations and data presentation; Based on the analysis results, a coal procurement decision support report is generated by combining report templates and automatic filling technology and output in a specified format.

2. The coal market analysis and procurement decision-making method based on big data according to claim 1, characterized in that In S1, multi-source data collection specifically includes: using a web crawler to capture web page data, and configuring a fault-tolerant mechanism for the web crawler to handle network anomalies; using a Web API interface call to obtain real-time data at a preset frequency; and using database direct connection technology to extract structured data from the coal mine production management system, inventory management system, and transportation information system.

3. A method for coal market analysis and procurement decision-making based on big data according to claim 1, characterized in that, In S1, data integration and cleaning include deduplication, error correction, and missing value filling; wherein: Deduplication processing removes duplicate data records based on unique fields; error correction processing verifies the rationality of data through value range checks and corrects contradictory data through logical relationship checks; missing value filling uses the mean or median to fill in numerical data, and model prediction to fill in key indicators.

4. A method for coal market analysis and procurement decision-making based on big data according to claim 1, characterized in that, In S1, the hierarchical database includes an original data table, a cleaned data table, and an analysis result table, and the hierarchical database adopts a relational database or a non-relational database; Among them, the relational database is MySQL or Oracle, and the non-relational database is MongoDB or Cassandra.

5. A method for coal market analysis and procurement decision-making based on big data according to claim 1, characterized in that In said S2, the index calculation includes: statistical calculation of coal price index, inventory turnover rate and transportation cost, and generating statistical reports based on a preset period.

6. A method for coal market analysis and procurement decision-making based on big data according to claim 1, characterized in that, In S2, the association rule mining algorithm adopts the Apriori algorithm or the FP-Growth algorithm, and the support and confidence thresholds are dynamically adjusted according to data characteristics.

7. A method for coal market analysis and procurement decision-making based on big data according to claim 1, characterized in that In S2, trend prediction adopts a time series model or a machine learning model, the time series model is ARIMA, SARIMA or Prophet; the machine learning model is a multi-layer perceptron, a convolutional neural network or a recurrent neural network.

8. A method for coal market analysis and procurement decision-making based on big data according to claim 1, characterized in that Said S2 also includes cluster analysis, regression analysis and model training. The cluster analysis uses K-Means algorithm or hierarchical clustering algorithm to segment coal mine production information and market data. The regression analysis uses linear regression, polynomial regression or ridge regression model to establish coal price and inventory; Model training optimization is achieved by dividing historical data into training set, validation set and test set, adjusting hyperparameters and feature selection, and using mean square error or determination coefficient as evaluation indicators.

9. A method for coal market analysis and procurement decision-making based on big data according to claim 1, characterized in that, In step S3, the report template is customized through a visual editor, including a cover page, a table of contents, data charts, and an analysis and interpretation module, and the template is classified and maintained through a template management system; The automatic filling technology includes extracting index calculation results, association rules, and predicted values from the analysis results, filling them into the corresponding positions of the template, generating a report through the Free marker or Velocity template engine, and supporting format conversion and multi-terminal display.

10. The system on which the method for coal market analysis and procurement decision-making based on big data according to any one of claims 1-9 is based, characterized in that, The system includes a data integration module, a data analysis module, and a decision generation module, where: Data integration module: used to generate an integrated market analysis data set through multi-source data collection, data integration and cleaning, and store it in a hierarchical database; The multi-source data includes the price index of coal, the production information of coal mines, inventory data, transportation data, and power generation data in the power market; Data analysis module: used to perform data analysis and mining through index calculation, association analysis, and trend prediction based on the integrated market analysis data set, and obtain analysis results including market dynamics, price trends, and supply situations; Index calculation includes statistical calculations of coal price index, inventory turnover rate, and transportation cost. Association analysis uses the association rule mining algorithm, and trend prediction uses a time series model or a machine learning model; Decision generation module: used to preset a report template based on market analysis, procurement suggestions, and data display; generate a coal procurement decision support report based on the analysis results, combined with the report template and automatic filling technology, and output it in a specified format.

Images