A system and method for monitoring and visualizing agricultural product prices big data

Abstract

The invention relates to a system and method for monitoring and visualizing big data of agricultural product prices, and belongs to the technical field of agricultural big data. A data acquisition module collects historical wholesale prices of each agricultural product in a first preset area and a second preset area and stores them in a data storage module. A data quality control module preprocesses the historical wholesale prices in a price information database and extracts time characteristics of a date to be predicted. A data mining and analysis module determines the average wholesale price of the agricultural product to be predicted on a predicted date according to a plurality of consecutive preprocessed historical average wholesale prices and the time characteristics of the date to be predicted. A data visualization module selects a visualization tool and finally sends the information to an information push terminal for visualization display, thereby realizing monitoring and visualization display of agricultural product prices.

Description

Technical Field

[0001] The present invention relates to the field of agricultural big data technology, and in particular to a system and method for monitoring and visualizing agricultural product price big data. Background Art

[0002] Agriculture is a production sector that provides the necessities of human survival. It is also the premise for other sectors in the national economy to become independent production sectors and the basis for their development. The price of agricultural products can affect the input of agricultural producers in agricultural production, and thus have an important impact on agricultural development. Data mining and analysis of agricultural product prices can extract valuable information (laws) from them, which can provide effective decision-making basis for the formulation and adjustment of agricultural production and sales strategies, thereby promoting and driving the development of the agricultural economy.

[0003] At present, big data is rapidly developing into a new generation of information technology and service formats that discover new knowledge, create new value, and enhance new capabilities. With the rapid development of the Internet of Things, cloud computing, mobile Internet, LBS, remote sensing and geographic information technology, agricultural data is showing a trend of massive explosion, and agriculture and rural areas are gradually becoming an important area for the generation and application of big data.

[0004] Agricultural product price information is the "weather vane" and "thermometer" of changes in the agricultural product market. Real-time monitoring of agricultural product price information can not only help farmers to understand comprehensive agricultural price information in a timely manner, so as to make reasonable decisions in the production and sales process and effectively increase farmers' income, but also provide effective decision-making support for agricultural management agencies and provide a basis and suggestions for solving sudden problems. The existing agricultural product price information monitoring system has problems such as single data collection type, small coverage, difficulty in obtaining information, and insufficient sharing. At the same time, most price information has not been deeply processed and mined and analyzed. Therefore, it is necessary to build a set of agricultural product price monitoring and analysis and prediction system with a new architecture. Summary of the invention

[0005] The purpose of the present invention is to provide a system and method for monitoring and visualizing agricultural product prices big data, so as to realize the monitoring and visual display of agricultural product prices.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A big data monitoring and visualization system for agricultural product prices, the system comprising: a data acquisition module, a data storage module, a data quality control module, a data mining and analysis module, a data visualization module and an information push terminal;

[0008] The data collection module is connected to the data storage module, and the data collection module is used to collect the historical wholesale price of each agricultural product in each second preset area in the first preset area; the historical wholesale price includes the historical wholesale daily price, the historical wholesale weekly price and the historical wholesale monthly price;

[0009] The data storage module is connected to the data quality control module, and the data storage module is used to construct an agricultural product category library and a price information library, store the agricultural product names in the agricultural product category library, and store the historical wholesale prices in the price information library; the price information library includes a daily price information library, a weekly price information library, and a monthly price information library; the data quality control module is used to preprocess the historical wholesale prices in the price information library, and store the preprocessed historical wholesale prices in the price information library;

[0010] The data mining and analysis module is connected to the data quality control module, and the data quality control module is further used to obtain a plurality of consecutive pre-processed historical average wholesale prices of the agricultural product to be predicted before the date to be predicted from the data storage module according to the agricultural product to be predicted and the date to be predicted, and extract the time characteristics of the date to be predicted; the data mining and analysis module is used to determine the average wholesale price of the agricultural product to be predicted on the date to be predicted according to the plurality of consecutive pre-processed historical average wholesale prices and the time characteristics of the date to be predicted;

[0011] The data visualization module is connected to the data quality control module and the information push terminal respectively. The data visualization module is used to select a visualization tool according to the average wholesale price of the agricultural product to be predicted on the predicted date, multiple consecutive historical average wholesale prices and dates, and map the average wholesale price of the agricultural product to be predicted on the predicted date, multiple consecutive historical average wholesale daily prices and dates to the selected visualization tool and then send them to the information push terminal for visual display.

[0012] Optionally, the data acquisition module includes:

[0013] A collection unit, used to determine a collection data source of the agricultural product to be predicted, and use a crawler system to collect multiple historical wholesale daily prices of the agricultural product to be predicted in each second preset area in each wholesale market from the collection data source every day, and add the prices to a data collection pool;

[0014] The building unit is used to build a streaming collection and analysis system based on streaming computing;

[0015] A streaming collection unit, used for collecting data from a data collection pool using a streaming collection and analysis system;

[0016] A type acquisition unit, used to acquire first interface data and types of the first interface data between various modules in the streaming acquisition and analysis system;

[0017] A statistical unit, used for counting a first range of each type of feature value within a first predetermined time period;

[0018] A second interface data acquisition unit, used to acquire second interface data between various modules in the streaming acquisition system;

[0019] A judging unit, used to judge whether the characteristic value of the second interface data is within the first range, and if so, use a time sliding window to perform convergence analysis, otherwise, obtain another analysis and calculation engine from the streaming system for separate processing;

[0020] An analysis unit is used to determine whether the collected data source has been calculated and analyzed and obtain a determination result;

[0021] An ending unit, used for ending and exiting if the judgment result indicates yes, and obtaining the required historical wholesale daily prices of the agricultural products to be predicted in each second preset area for multiple consecutive days;

[0022] A loop unit, used for returning to the step of "obtaining first interface data and its type between various modules in the streaming acquisition and analysis system" if the judgment result indicates no;

[0023] The calculation unit is used to calculate the historical daily average wholesale price of the agricultural product to be predicted in the first preset area for a number of consecutive days according to the historical daily wholesale prices of the agricultural product to be predicted in each second preset area for a number of consecutive days.

[0024] Optionally, the data mining and analysis module specifically includes:

[0025] A sample division unit, used to divide the historical wholesale daily average price samples of previous years into a plurality of daily average price sample sequences using a sliding window of a preset sliding window length;

[0026] A time feature acquisition unit is used to obtain the time feature of the last historical wholesale daily average price in each daily average price sample sequence;

[0027] The sample data set forming unit is used to take the last historical wholesale daily average price in each daily average price sample sequence as a label, and all historical wholesale daily average prices other than the last historical wholesale daily average price in each daily average price sample sequence and the time feature of the last historical wholesale daily average price as input quantities to form a daily average price sample data set;

[0028] A data set division unit, used to divide the daily average price sample data set into a training set and a validation set;

[0029] A training unit, used for training the extreme random tree model using the training set to obtain a trained extreme random tree model;

[0030] An optimization unit, used for optimizing the trained extreme random tree model using a validation set to obtain an optimized extreme random tree model;

[0031] The application unit is used to input the historical average daily wholesale price of multiple consecutive days and the time characteristics of the date to be predicted as input into the trained extreme random tree model, and output the average daily wholesale price of the agricultural product to be predicted on the date to be predicted.

[0032] Optionally, the data mining and analysis module further includes:

[0033] A training error obtaining unit, used to obtain a training error based on the training set and using the trained extreme random tree model;

[0034] The confidence interval output unit is used to input the training error into the generalized autoregressive conditional heteroskedasticity model based on a preset confidence level, and output the confidence interval under the preset confidence level.

[0035] A method for monitoring and visualizing agricultural product prices big data applied to the aforementioned agricultural product prices big data monitoring and visualization system, the method comprising:

[0036] The agricultural big data automatic collection method based on streaming computing is used to collect the historical wholesale daily average price of the agricultural product to be predicted in the first preset area for multiple consecutive days;

[0037] Extracting the time characteristics of the forecast date of the agricultural product to be forecasted; the time characteristics include the day of the week, the weeks of the month, the months of the year and the quarter of the year for the forecast date;

[0038] The historical average daily wholesale price of multiple consecutive days and the time characteristics of the day to be predicted are used as inputs to the trained extreme random tree model, and the average daily wholesale price of the agricultural product to be predicted on the day to be predicted is output;

[0039] The average daily wholesale price of the agricultural product to be predicted on the predicted day, the historical average daily wholesale price for several consecutive days and the date are used as the data to be visualized, and a visualization tool is selected for visualization display according to the characteristics of the data to be visualized.

[0040] Optionally, the method for automatically collecting agricultural big data based on streaming computing is used to collect the historical average daily wholesale prices of the agricultural products to be predicted in the first preset area for multiple consecutive days, specifically including:

[0041] Determine a data source for collecting the agricultural products to be predicted, and use a crawler system to collect multiple historical wholesale daily prices of the agricultural products to be predicted in each second preset area in each wholesale market from the data source every day, and add them to a data collection pool;

[0042] Build a streaming collection and analysis system based on streaming computing;

[0043] Collect data from the data collection pool using a streaming collection and analysis system;

[0044] Obtaining first interface data and its type between various modules in the streaming acquisition and analysis system;

[0045] Counting a first range of each type of feature value within a first predetermined time period;

[0046] Acquire second interface data between modules in the streaming acquisition system;

[0047] Determine whether the characteristic value of the second interface data is within the first range. If so, use the time sliding window to perform convergence analysis. Otherwise, obtain another analysis and calculation engine from the streaming system for separate processing.

[0048] Determine whether the collected data source has been calculated and analyzed, and obtain the judgment result;

[0049] If the judgment result indicates yes, then exit and obtain the required historical wholesale daily prices of the agricultural products to be predicted in each second preset area for multiple consecutive days;

[0050] If the judgment result indicates no, then return to the step of "obtaining the first interface data and its type between each module in the streaming acquisition and analysis system";

[0051] According to the historical wholesale daily prices of the agricultural products to be predicted in each second preset area for a number of consecutive days, the historical average daily wholesale prices of the agricultural products to be predicted in the first preset area for a number of consecutive days are calculated.

[0052] Optionally, the average daily wholesale price of the agricultural product to be predicted on the predicted day, the historical average daily wholesale price of multiple consecutive days and the date are used as the data to be visualized, and a visualization tool is selected for visualization display according to the characteristics of the data to be visualized, specifically including:

[0053] Determine the visual mapping structure of the average daily wholesale price of the agricultural product to be predicted on the predicted day, the historical average daily wholesale price for consecutive days, and the date; the visual mapping structure includes a list structure, a coordinate structure, a time flow structure, a hierarchical structure, and a network structure;

[0054] A visualization space layout diagram matching the visualization mapping structure is selected, and the average daily wholesale price of the agricultural product to be predicted on the predicted day, the average daily wholesale price for consecutive days and the date are displayed on the matching visualization space layout diagram as the data mapping to be visualized; the visualization space layout diagram includes a statistical diagram, a time series diagram, a hierarchical diagram, a network diagram, a geographic location diagram and a tag cloud.

[0055] Optionally, the method of automatically collecting agricultural big data based on streaming computing is used to collect the historical average daily wholesale prices of the agricultural products to be predicted in the first preset area for multiple consecutive days, and then further includes:

[0056] Perform noise cleaning on the historical wholesale daily average price;

[0057] Use the historical average method or time average method to repair the missing data of the historical wholesale daily average price after noise cleaning;

[0058] The calculation formula of the historical average method is: Among them, M is the missing historical wholesale daily average price, y j is the historical wholesale daily average price on the same weekday as the missing historical wholesale daily average price in the j weeks before the missing historical wholesale daily average price, and K is the week number;

[0059] The calculation formula of the time averaging method is: Among them, x 1 、x 2 They are the historical wholesale daily average prices of the day before and the day after the missing historical wholesale daily average price.

[0060] Optionally, the method for determining the trained extreme random tree model is:

[0061] The automatic collection method of agricultural big data based on streaming computing is used to collect multiple historical wholesale daily prices of the agricultural products to be predicted in the first preset area in the same wholesale market over the years.

[0062] Calculate the historical average daily wholesale price samples of the agricultural products to be predicted in the first preset area over the years according to the collected multiple historical wholesale daily prices;

[0063] The historical wholesale daily average price samples of previous years are divided into multiple daily average price sample sequences using a sliding window with a preset sliding window length;

[0064] Extract the time characteristics of the last historical wholesale daily average price in each daily average price sample sequence;

[0065] The last historical wholesale daily average price in each daily average price sample sequence is used as a label, and all historical wholesale daily average prices other than the last historical wholesale daily average price in each daily average price sample sequence and the time characteristics of the last historical wholesale daily average price are used as input to form a daily average price sample data set;

[0066] Divide the daily average price sample data set into a training set and a validation set;

[0067] Using the training set to train the extreme random tree model, obtaining a trained extreme random tree model;

[0068] The trained extreme randomized tree model is optimized using the validation set to obtain an optimized extreme randomized tree model.

[0069] Optionally, the method further includes taking the average daily wholesale price of the agricultural product to be predicted on the predicted day, the average daily wholesale price of consecutive days and the date as the data to be visualized, and selecting a visualization tool for visualization according to the characteristics of the data to be visualized, and then further including:

[0070] According to the training set, using the trained extreme random tree model, obtaining a training error;

[0071] Based on the preset confidence level, the training error is input into the generalized autoregressive conditional heteroskedasticity model, and the confidence interval under the preset confidence level is output;

[0072] The generalized autoregressive conditional heteroskedasticity model is Among them, h it is the conditional heteroskedasticity of the ith random variable at time t, and are the first, second and third model parameters, m and n are the first and second lag orders, M and N are the total number of the first and second lag orders, ε it-m is the observed value of the i-th random variable at time tm, h it-n is the conditional heteroskedasticity of the ith random variable at time tn.

[0073] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0074] The present invention discloses a system and method for monitoring and visualizing big data of agricultural product prices. A data acquisition module collects historical wholesale prices of each agricultural product in a first preset area and a second preset area and stores them in a data storage module. A data quality control module preprocesses the historical wholesale prices in a price information database and extracts time characteristics of a date to be predicted. A data mining and analysis module determines the average wholesale price of the agricultural product to be predicted on a predicted date according to a plurality of consecutive preprocessed historical average wholesale prices and the time characteristics of the date to be predicted. A data visualization module selects a visualization tool and finally sends the information to an information push terminal for visualization display, thereby realizing monitoring and visualization display of agricultural product prices. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0076] Figure 1 A structural diagram of a big data monitoring and visualization system for agricultural product prices provided by the present invention;

[0077] Figure 2 The soybean price trend chart provided by the present invention; Figure 2 Part (a) shows the daily average price trend of soybeans. Figure 2 Part (b) shows the weekly average price trend of soybeans. Figure 2 Part (c) shows the monthly average price trend of soybeans. Figure 2 Part (d) shows the seasonal average price trend of soybeans. Figure 2 Part (e) shows the annual average price trend of soybeans;

[0078] Figure 3 The average price comparison chart provided by the present invention; Figure 3 Part (a) is a comparison chart of the average daily soybean prices in various regions. Figure 3 Part (b) is a comparison chart of the average quarterly prices over the years;

[0079] Figure 4 A flowchart of the method for monitoring and visualizing agricultural product price big data provided by the present invention;

[0080] Figure 5 The agricultural big data visualization flow chart provided by the present invention;

[0081] Figure 6 A soybean model training result diagram for the first preset area provided by the present invention;

[0082] Figure 7 A soybean price forecast evaluation result diagram for the first preset area provided by the present invention;

[0083] Figure 8 The present invention provides a short-term mean forecast and interval forecast chart of daily soybean prices in the first preset area. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] The purpose of the present invention is to provide a system and method for monitoring and visualizing agricultural product prices big data, so as to realize the monitoring and visual display of agricultural product prices.

[0086] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0087] A big data monitoring and visualization system for agricultural product prices, such as Figure 1As shown, the system includes: a data acquisition module, a data storage module, a data quality control module, a data mining and analysis module, a data visualization module and an information push terminal.

[0088] The data collection module is connected to the data storage module, and the data collection module is used to collect the historical wholesale prices of each agricultural product in the first preset area and the second preset area; the historical wholesale prices include the historical wholesale daily prices, the historical wholesale weekly prices and the historical wholesale monthly prices;

[0089] The data storage module is connected to the data quality control module. The data storage module is used to construct an agricultural product category library and a price information library, store the names of agricultural products in the agricultural product category library, and store historical wholesale prices in the price information library; the price information library includes a daily price information library, a weekly price information library, and a monthly price information library; the data quality control module is used to preprocess the historical wholesale prices in the price information library, and store the preprocessed historical wholesale prices in the price information library;

[0090] The data mining and analysis module is connected to the data quality control module. The data quality control module is also used to obtain the continuous pre-processed historical average wholesale prices of the agricultural products to be predicted before the date to be predicted from the data storage module according to the agricultural products to be predicted and the date to be predicted, and extract the time characteristics of the date to be predicted; the data mining and analysis module is used to determine the average wholesale price of the agricultural products to be predicted on the date to be predicted according to the continuous pre-processed historical average wholesale prices and the time characteristics of the date to be predicted;

[0091] The data visualization module is connected to the data quality control module and the information push terminal respectively. The data visualization module is used to select visualization tools according to the average wholesale price of the agricultural product to be predicted on the predicted date, multiple consecutive historical average wholesale prices and dates, and map the average wholesale price of the agricultural product to be predicted on the predicted date, multiple consecutive historical average wholesale daily prices and dates to the selected visualization tools and then send them to the information push terminal for visual display.

[0092] Taking China as an example, the first preset area refers to the whole country, and the second preset area refers to the province. The present invention collects daily prices of different types of agricultural products (covering seven categories of nearly 500 types, including vegetables, fruits, livestock products, aquatic products, grains and oils) in 34 provinces across the country by designing a computer automatic collection program and an agricultural product price collection system APP, and performs data cleaning, including deduplication and filtering of abnormal data points. It covers more than 1,000 key collection sources across the country, involving bulk wholesale market prices, market retail prices and market transaction volume information. From Figure 1 It can also be seen that data collection is achieved through manual review and entry into the system.

[0093] The price trend is composed of the average wholesale price of the agricultural product on the predicted date and the average price of multiple consecutive historical wholesale days. The price trend function reflects the trend of agricultural product prices at different time granularities in the form of line graphs and bar graphs, including two sub-modules: average price trend and average price comparison.

[0094] The average price trend submodule is divided into five parts: daily average price trend, weekly average price trend, monthly average price trend, quarterly average price trend and annual average price trend. Figure 2 shown.

[0095] The average price comparison submodule is divided into seven parts: daily average price comparison, weekly average price comparison, monthly average price comparison, quarterly average price comparison, annual average price comparison, monthly average price comparison and quarterly average price comparison. Figure 3 shown.

[0096] The data acquisition module includes:

[0097] A collection unit, used to determine a collection data source of the agricultural product to be predicted, and use a crawler system to collect multiple historical wholesale daily prices of the agricultural product to be predicted in each second preset area in each wholesale market from the collection data source every day, and add the prices to a data collection pool;

[0098] The building unit is used to build a streaming collection and analysis system based on streaming computing;

[0099] A streaming collection unit, used for collecting data from a data collection pool using a streaming collection and analysis system;

[0100] A type acquisition unit, used to acquire first interface data and its type between various modules in the streaming acquisition and analysis system;

[0101] A statistical unit is used to count the first range of each type of characteristic value within the first predetermined time period. A second interface data acquisition unit is used to acquire the second interface data between each module in the streaming acquisition system. A judgment unit is used to judge whether the characteristic value of the second interface data is within the first range. If it is within the first range, a time sliding window is used for convergence analysis. Otherwise, another analysis and calculation engine is obtained from the streaming system for separate processing. An analysis unit is used to judge whether the collection data source has been calculated and analyzed, and obtain a judgment result. An end unit is used to exit if the judgment result indicates yes, and obtain the required historical wholesale daily prices of the agricultural products to be predicted in each second preset area for multiple consecutive days. A loop unit is used to return to the step of "obtaining the first interface data and its type between each module in the streaming acquisition and analysis system" if the judgment result indicates no. A calculation unit is used to calculate the historical wholesale daily average price of the agricultural products to be predicted in the first preset area for multiple consecutive days based on the historical wholesale daily prices of the agricultural products to be predicted in each second preset area for multiple consecutive days.

[0102] In addition to building a database of agricultural product categories and a price information database, data storage and management also builds a wholesale market information database and a decision-making information database.

[0103] Since the daily price information published by the data source may be different, it is necessary to remove abnormal data. The data quality control module includes: a noise cleaning unit and a missing data repair unit. The noise cleaning unit is used to perform noise cleaning on the historical wholesale daily average price. The missing data repair unit is used to repair the missing data of the historical wholesale daily average price after noise cleaning using the historical average method or the time average method.

[0104] The calculation method of the historical average method is: determine the day of the week the missing data is, assuming it is Tuesday, then take the average of the prices on Tuesdays of the previous three weeks and fill in the missing data.

[0105] The calculation formula of the historical average method is: Among them, M is the missing historical wholesale daily average price, y j is the historical wholesale daily average price on the same weekday as the missing historical wholesale daily average price in the j weeks before the missing historical wholesale daily average price, and K is the week number.

[0106] The calculation method of the time average method is: take the average price of the two days before and after the date of the missing data, and fill in the missing data.

[0107] The calculation formula of the time averaging method is: Among them, x 1 、x 2 They are the historical wholesale daily average prices of the day before and the day after the missing historical wholesale daily average price.

[0108] The method for repairing missing data is not limited to the historical average method and the time average method. Other methods that can fill in missing data are also applicable to the present invention.

[0109] The data quality control module also has functions such as data deduplication, data missing check, outlier detection, data regularization, data fusion, feature transformation and feature selection.

[0110] The data mining and analysis module includes: a sample division unit, a time feature acquisition unit, a sample data set composition unit, a data set division unit, a training unit, an optimization unit and an application unit.

[0111] The sample division unit is used to divide the historical wholesale daily average price samples of previous years into multiple daily average price sample sequences using a sliding window of a preset sliding window length. The time feature acquisition unit is used to obtain the time feature of the last historical wholesale daily average price in each daily average price sample sequence. The sample data set construction unit is used to use the last historical wholesale daily average price in each daily average price sample sequence as a label, and all historical wholesale daily average prices other than the last historical wholesale daily average price in each daily average price sample sequence and the time feature of the last historical wholesale daily average price as input to form a daily average price sample data set. The data set division unit is used to divide the daily average price sample data set into a training set and a validation set. The training unit is used to train an extremely randomized tree (Extremely randomized trees, Extra-Trees) model using the training set to obtain a trained extremely randomized tree model. The optimization unit is used to optimize the trained extremely randomized tree model using the validation set to obtain an optimized extremely randomized tree model. The application unit is used to input the historical average daily wholesale price of multiple consecutive days and the time characteristics of the date to be predicted as input into the trained extreme random tree model, and output the average daily wholesale price of the agricultural product to be predicted on the date to be predicted.

[0112] When the historical wholesale average price sample is a historical wholesale daily average price sample, the time characteristics include the day of the week, the week of the month, the month of the year, and the quarter of the year on which the last historical wholesale daily average price falls. The format of the sample data is (y 1 ,y 2 ,y 3 ,y 4 ,y 5 ,y 6 ,dayofweek,weekofmonth,dayofmonth,season,y).

[0113] When the historical wholesale average price sample is a historical wholesale weekly average price sample, the time characteristics include the week number of the month, the month number of the year, and the quarter number of the week in which the last historical wholesale weekly average price belongs. The format of the sample data is (y 1 ,y 2 ,y 3 ,y 4 ,y 5 ,y 6 ,weekofmonth,dayofmonth,season,y).

[0114] When the historical wholesale average price sample is a historical wholesale monthly average price sample, the time characteristics include the month and quarter of the year in which the last historical wholesale monthly average price belongs. The format of the sample data is (y1 ,y 2 ,y 3 ,y 4 ,y 5 ,y 6 ,dayofmonth,season,y).

[0115] The data mining analysis module also includes: a training error acquisition unit and a confidence interval output unit. The training error acquisition unit is used to obtain the training error based on the training set using the trained extreme random tree model. The confidence interval output unit is used to input the training error into the generalized autoregressive conditional heteroskedasticity model (GARCH) based on the preset confidence level, and output the confidence interval under the preset confidence level. The generalized autoregressive conditional heteroskedasticity model is Among them, h it is the conditional heteroskedasticity of the ith random variable at time t, and are the first, second and third model parameters, m and n are the first and second lag orders, M and N are the total number of the first and second lag orders, ε it-m is the observed value of the i-th random variable at time tm, h it-n is the conditional heteroskedasticity of the ith random variable at time tn.

[0116] The big data analysis technologies used in the data mining and analysis module include data mining, deep learning, and natural language processing technologies. Data mining includes classification, regression, and time series prediction. Deep learning includes deep convolutional neural networks and deep recurrent neural networks. Natural language processing technologies include lexical analysis, syntactic analysis, and semantic analysis. Big data analysis tools include TensorFlow / Keras, Lucene / Solr, MLlib / Mahout, and Spark / Hadoop.

[0117] After big data analysis by the data mining and analysis module, daily price quotes, rise and fall warnings, trend forecasts, key monitoring, price trends, price cloud maps, price seasons, price forecasts and market briefings can be formed.

[0118] The data visualization module includes: a visualization mapping structure determination unit and a visualization space layout diagram selection unit. The visualization mapping structure determination unit is used to determine the visualization mapping structure of the average wholesale price of the agricultural product to be predicted on the predicted day, the historical average wholesale price for consecutive days, and the date; the visualization mapping structure includes a list structure, a coordinate structure, a time flow structure, a hierarchical structure, and a network structure. The visualization space layout diagram selection unit is used to select a visualization space layout diagram that matches the visualization mapping structure, and display the average wholesale price of the agricultural product to be predicted on the predicted day, the historical average wholesale price for consecutive days, and the date as the data mapping to be visualized on the matching visualization space layout diagram; the visualization space layout diagram includes a statistical diagram, a time series diagram, a hierarchical diagram, a network diagram, a geographic location diagram, and a tag cloud.

[0119] The information push terminal is an intelligent terminal for agricultural big data service push for multi-level users, which can be a PC, tablet computer, mobile phone, touch screen, or electronic large screen. The present invention intuitively displays the historical price and predicted price of agricultural products on the terminal used by the user, which is convenient for the user to check and analyze, and provides practical and targeted agricultural information services to the majority of users.

[0120] The present invention is applied to the aforementioned agricultural product price big data monitoring and visualization system, and also provides an agricultural product price big data monitoring and visualization method, such as Figure 4 As shown, the method includes:

[0121] Step 1: Use an automatic agricultural big data collection method based on streaming computing to collect the historical average daily wholesale prices of the agricultural products to be predicted in the first preset area for multiple consecutive days.

[0122] The traditional data acquisition system architecture collects typical long-cycle data and is usually suitable for scenarios where data is analyzed and processed offline. Because data processing capabilities are prioritized, data is often processed in batches under this architecture. However, this type of architecture is more effective in scenarios with low real-time requirements and cannot effectively process continuous short-term, high-real-time data.

[0123] Specifically include:

[0124] Determine a data source for collecting the agricultural products to be predicted, and use a crawler system to collect multiple historical wholesale daily prices of the agricultural products to be predicted in each second preset area in each wholesale market from the data source every day, and add them to a data collection pool;

[0125] Build a streaming collection and analysis system based on streaming computing;

[0126] Collect data from the data collection pool using a streaming collection and analysis system;

[0127] Obtaining first interface data and its type between various modules in the streaming acquisition and analysis system;

[0128] Counting a first range of each type of feature value within a first predetermined time period;

[0129] Acquire second interface data between modules in the streaming acquisition system;

[0130] Determine whether the characteristic value of the second interface data is within the first range. If so, use the time sliding window to perform convergence analysis. Otherwise, obtain another analysis and calculation engine from the streaming system for separate processing.

[0131] Determine whether the collected data source has been calculated and analyzed, and obtain the judgment result;

[0132] If the judgment result indicates yes, then exit and obtain the required historical wholesale daily prices of the agricultural products to be predicted in each second preset area for multiple consecutive days;

[0133] If the judgment result indicates no, then return to the step of "obtaining the first interface data and its type between each module in the streaming acquisition and analysis system";

[0134] According to the historical wholesale daily prices of the agricultural products to be predicted in each second preset area for a number of consecutive days, the historical average daily wholesale prices of the agricultural products to be predicted in the first preset area for a number of consecutive days are calculated.

[0135] After data collection, the data also needs to be preprocessed: noise cleaning of the historical wholesale daily average price; use the historical average method or time average method to repair missing data in the historical wholesale daily average price after noise cleaning.

[0136] The calculation formula of the historical average method is: Among them, M is the missing historical wholesale daily average price, y j is the historical wholesale daily average price on the same weekday as the missing historical wholesale daily average price in the j weeks before the missing historical wholesale daily average price, and K is the week number;

[0137] The calculation formula of the time averaging method is: Among them, x 1 、x 2 They are the historical wholesale daily average prices of the day before and the day after the missing historical wholesale daily average price.

[0138] Step 2, extracting the time characteristics of the predicted day of the agricultural product to be predicted; the time characteristics include the day of the week, the number of weeks in a month, the number of months in a year, and the number of quarters of the year for the predicted day.

[0139] Step 3: Take the historical average daily wholesale price of multiple consecutive days and the time characteristics of the day to be predicted as input, input them into the trained extreme random tree model, and output the average daily wholesale price of the agricultural product to be predicted on the day to be predicted.

[0140] The trained extreme random tree model is determined as follows:

[0141] The automatic collection method of agricultural big data based on streaming computing is used to collect multiple historical wholesale daily prices of the agricultural products to be predicted in the first preset area in the same wholesale market over the years.

[0142] Calculate the historical average daily wholesale price samples of the agricultural products to be predicted in the first preset area over the years according to the collected multiple historical wholesale daily prices;

[0143] The historical wholesale daily average price samples of previous years are divided into multiple daily average price sample sequences using a sliding window with a preset sliding window length;

[0144] Extract the time characteristics of the last historical wholesale daily average price in each daily average price sample sequence;

[0145] The last historical wholesale daily average price in each daily average price sample sequence is used as a label, and all historical wholesale daily average prices other than the last historical wholesale daily average price in each daily average price sample sequence and the time characteristics of the last historical wholesale daily average price are used as input to form a daily average price sample data set;

[0146] Divide the daily average price sample data set into a training set and a validation set;

[0147] Using the training set to train the extreme random tree model, obtaining a trained extreme random tree model;

[0148] The trained extreme randomized tree model is optimized using the validation set to obtain an optimized extreme randomized tree model.

[0149] Step 4: Use the average daily wholesale price of the agricultural product to be predicted on the predicted day, the historical average daily wholesale price for several consecutive days, and the date as the data to be visualized, and select a visualization tool for visualization according to the characteristics of the data to be visualized.

[0150] Big data visualization refers to converting structured or unstructured data into appropriate visualization charts, and then presenting the information hidden behind the data to users in an intuitive and vivid way. Existing big data visualization mainly focuses on data visualization based on traditional static charts or their combination. Agricultural big data is rich in variety and huge in volume, and has the significant characteristics of dynamic heterogeneity.

[0151] Reference Figure 5 ,The process of agricultural big data visualization is as follows:

[0152] Determine the visual mapping structure of the average daily wholesale price of the agricultural product to be predicted, the historical average daily wholesale price of consecutive days and the date; the visual mapping structure includes a list structure, a coordinate structure, a time flow structure, a hierarchical structure and a network structure;

[0153] A visualization space layout diagram that matches the visualization mapping structure is selected, and the average daily wholesale price of the agricultural product to be predicted on the predicted day, the historical average daily wholesale price for multiple consecutive days and the date are displayed on the matching visualization space layout diagram as the data mapping to be visualized; the visualization space layout diagram includes statistical diagrams, time series diagrams, hierarchical diagrams, network diagrams, geographic location diagrams and tag clouds.

[0154] In addition to the daily average wholesale price of the agricultural product to be predicted, the visualized data can also display the confidence interval of the daily average wholesale price on the predicted day. The output method of the confidence interval is: according to the training set, the trained extreme random tree model is used to obtain the training error; based on the preset confidence level, the training error is input into the generalized autoregressive conditional heteroskedasticity model, and the confidence interval under the preset confidence level is output. The generalized autoregressive conditional heteroskedasticity model is Among them, h it is the conditional heteroskedasticity of the ith random variable at time t, and are the first, second and third model parameters, m and n are the first and second lag orders, M and N are the total number of the first and second lag orders, ε it-m is the observed value of the i-th random variable at time tm, h it-n is the conditional heteroskedasticity of the ith random variable at time tn.

[0155] The price prediction function builds an agricultural product price prediction model based on traditional shallow machine learning algorithms (such as support vector machine regression, random forest, etc.) and modern deep learning algorithms (such as deep convolutional neural networks, deep recurrent neural networks, etc.) to effectively predict agricultural product prices in the short term in the future. It specifically includes three sub-modules: model training, prediction evaluation, and short-term prediction.

[0156] The model training submodule learns historical price data based on the above-mentioned machine learning algorithm to train the model and calibrate the model parameters. Figure 6 An example of model training is given. The degree of fit between the two broken lines in the figure can show the quality of model training.

[0157] The prediction evaluation submodule reflects the degree of fit between the predicted price curve and the actual price curve, and reflects the accuracy of the short-term price prediction model. Figure 7 shown.

[0158] The short-term prediction submodule includes point prediction and interval prediction. Point prediction is to build a machine learning prediction model based on the historical daily prices of a certain type of agricultural products, and predict the daily average price in the short term in the future in the form of discrete prices. Interval prediction is to build a machine learning prediction model based on the historical daily prices of a certain type of agricultural products and the prediction deviation, and make a probabilistic prediction of the interval that the daily price may fall into in the short term in the future. Interval prediction, also known as reliability prediction, is a machine learning method for learning and predicting data patterns in an uncertain environment. Figure 8 It is the short-term mean forecast and interval forecast chart of soybean daily price in the first preset area. The confidence interval of interval forecast includes the upper and lower limits of 90% forecast interval and the upper and lower limits of 80% forecast interval. Figure 8 The gray area at the end of the midline is the confidence interval.

[0159] Aiming at the entire chain of pre-production, intrapartum and postpartum, the project builds an agricultural big data catalog resource system, decision-making function system, and platform architecture system through comprehensive data collection and sorting. It uses cutting-edge AI technologies such as natural language processing, computer vision, machine learning, and knowledge graphs to develop intelligent mining methods, models, and technologies for agricultural price big data. By developing dynamic interactive visualization methods and service push intelligent terminal systems, an agricultural big data platform with a closed-loop CPS architecture of "state perception → real-time analysis → scientific decision-making → precise execution" is established. Through AI empowerment, intelligent decision-making support for agricultural production, management, and circulation is achieved. This is a beneficial exploration of AI technology in the field of agricultural informatization, and it has important practical significance for accelerating the process of agricultural informatization.

[0160] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0161] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A big data monitoring and visualization system for agricultural product prices, characterized in that: The system includes: a data acquisition module, a data storage module, a data quality control module, a data mining and analysis module, a data visualization module and an information push terminal; The data collection module is connected to the data storage module, and the data collection module is used to collect the historical wholesale price of each agricultural product in each second preset area in the first preset area; the historical wholesale price includes the historical wholesale daily price, the historical wholesale weekly price and the historical wholesale monthly price; The data storage module is connected to the data quality control module, and the data storage module is used to construct an agricultural product category library and a price information library, store the agricultural product names in the agricultural product category library, and store the historical wholesale prices in the price information library; the price information library includes a daily price information library, a weekly price information library, and a monthly price information library; the data quality control module is used to preprocess the historical wholesale prices in the price information library, and store the preprocessed historical wholesale prices in the price information library; The data mining and analysis module is connected to the data quality control module, and the data quality control module is further used to obtain a plurality of consecutive pre-processed historical average wholesale prices of the agricultural product to be predicted before the date to be predicted from the data storage module according to the agricultural product to be predicted and the date to be predicted, and extract the time characteristics of the date to be predicted; the data mining and analysis module is used to determine the average wholesale price of the agricultural product to be predicted on the date to be predicted according to the plurality of consecutive pre-processed historical average wholesale prices and the time characteristics of the date to be predicted; The data visualization module is connected to the data quality control module and the information push terminal respectively, and the data visualization module is used to select a visualization tool according to the average wholesale price of the agricultural product to be predicted on the predicted date, multiple consecutive historical average wholesale prices and dates, and map the average wholesale price of the agricultural product to be predicted on the predicted date, multiple consecutive historical average wholesale prices and dates to the selected visualization tool and then send it to the information push terminal for visualization display; The data mining and analysis module specifically includes: A sample division unit, used to divide the historical wholesale daily average price samples of previous years into a plurality of daily average price sample sequences using a sliding window of a preset sliding window length; A time feature acquisition unit is used to obtain the time feature of the last historical wholesale daily average price in each daily average price sample sequence; The sample data set forming unit is used to take the last historical wholesale daily average price in each daily average price sample sequence as a label, and all historical wholesale daily average prices other than the last historical wholesale daily average price in each daily average price sample sequence and the time feature of the last historical wholesale daily average price as input quantities to form a daily average price sample data set; A data set division unit, used to divide the daily average price sample data set into a training set and a validation set; A training unit, used for training the extreme random tree model using the training set to obtain a trained extreme random tree model; An optimization unit, used for optimizing the trained extreme random tree model using a validation set to obtain an optimized extreme random tree model; The application unit is used to input the historical average daily wholesale price of multiple consecutive days and the time characteristics of the date to be predicted as input into the trained extreme random tree model, and output the average daily wholesale price of the agricultural product to be predicted on the date to be predicted.

2. The agricultural product price big data monitoring and visualization system according to claim 1 is characterized in that: The data acquisition module comprises: A collection unit, used to determine a collection data source of the agricultural product to be predicted, and use a crawler system to collect multiple historical wholesale daily prices of the agricultural product to be predicted in each second preset area in each wholesale market from the collection data source every day, and add the prices to a data collection pool; The building unit is used to build a streaming collection and analysis system based on streaming computing; A streaming collection unit, used for collecting data from a data collection pool using a streaming collection and analysis system; A type acquisition unit, used to acquire first interface data and types of the first interface data between various modules in the streaming acquisition and analysis system; A statistical unit, used for counting a first range of each type of feature value within a first predetermined time period; A second interface data acquisition unit, used to acquire second interface data between various modules in the streaming acquisition system; A judging unit is used to judge whether the characteristic value of the second interface data is within a first range, and if it is within the first range, The time sliding window is used for convergence analysis. Otherwise, another analysis and calculation engine is obtained from the streaming system for separate processing. An analysis unit is used to determine whether the collected data source has been calculated and analyzed and obtain a determination result; An ending unit, used for ending and exiting if the judgment result indicates yes, and obtaining the required historical wholesale daily prices of the agricultural products to be predicted in each second preset area for multiple consecutive days; A loop unit, used for returning to the step of "obtaining the first interface data and its type between each module in the streaming acquisition and analysis system" if the judgment result indicates no; The calculation unit is used to calculate the historical daily average wholesale price of the agricultural product to be predicted in the first preset area for a number of consecutive days according to the historical daily wholesale prices of the agricultural product to be predicted in each second preset area for a number of consecutive days.

3. The agricultural product price big data monitoring and visualization system according to claim 1 is characterized in that: The data mining and analysis module also includes: A training error obtaining unit, used to obtain a training error based on the training set and using the trained extreme random tree model; The confidence interval output unit is used to input the training error into the generalized autoregressive conditional heteroskedasticity model based on a preset confidence level, and output the confidence interval under the preset confidence level.

4. A method for monitoring and visualizing agricultural product prices big data applied to the agricultural product prices big data monitoring and visualization system according to any one of claims 1 to 3, characterized in that: The method comprises: The agricultural big data automatic collection method based on streaming computing is used to collect the historical wholesale daily average price of the agricultural product to be predicted in the first preset area for multiple consecutive days; Extracting the time characteristics of the forecast date of the agricultural product to be forecasted; the time characteristics include the day of the week, the weeks of the month, the months of the year and the quarter of the year for the forecast date; The historical average daily wholesale price of multiple consecutive days and the time characteristics of the day to be predicted are used as inputs to the trained extreme random tree model, and the average daily wholesale price of the agricultural product to be predicted on the day to be predicted is output; The average daily wholesale price of the agricultural product to be predicted on the predicted day, the historical average daily wholesale price for several consecutive days and the date are used as the data to be visualized, and a visualization tool is selected for visualization display according to the characteristics of the data to be visualized.

5. The method for monitoring and visualizing agricultural product prices big data according to claim 4, characterized in that: The method for automatically collecting agricultural big data based on streaming computing is used to collect the historical average daily wholesale prices of agricultural products to be predicted in the first preset area for multiple consecutive days, specifically including: Determine a data source for collecting the agricultural products to be predicted, and use a crawler system to collect multiple historical wholesale daily prices of the agricultural products to be predicted in each second preset area in each wholesale market from the data source every day, and add them to a data collection pool; Build a streaming collection and analysis system based on streaming computing; Collect data from the data collection pool using a streaming collection and analysis system; Obtaining first interface data and its type between various modules in the streaming acquisition and analysis system; Counting a first range of each type of feature value within a first predetermined time period; Acquire second interface data between modules in the streaming acquisition system; Determine whether the characteristic value of the second interface data is within the first range. If so, use the time sliding window to perform convergence analysis. Otherwise, obtain another analysis and calculation engine from the streaming system for separate processing. Determine whether the collected data source has been calculated and analyzed, and obtain the judgment result; If the judgment result indicates yes, then exit and obtain the required historical wholesale daily prices of the agricultural products to be predicted in each second preset area for multiple consecutive days; If the judgment result indicates no, then return to the step of "obtaining the first interface data and its type between each module in the streaming acquisition and analysis system"; According to the historical wholesale daily prices of the agricultural products to be predicted in each second preset area for a number of consecutive days, the historical average daily wholesale prices of the agricultural products to be predicted in the first preset area for a number of consecutive days are calculated.

6. The method for monitoring and visualizing agricultural product prices based on big data according to claim 4, characterized in that: The method uses the average daily wholesale price of the agricultural product to be predicted on the predicted day, the average daily wholesale price of the agricultural product to be predicted for several consecutive days, and the date as the data to be visualized, and selects a visualization tool for visualization display according to the characteristics of the data to be visualized, specifically including: Determine the visual mapping structure of the average daily wholesale price of the agricultural product to be predicted on the predicted day, the historical average daily wholesale price for consecutive days, and the date; the visual mapping structure includes a list structure, a coordinate structure, a time flow structure, a hierarchical structure, and a network structure; A visualization space layout diagram matching the visualization mapping structure is selected, and the average daily wholesale price of the agricultural product to be predicted on the predicted day, the average daily wholesale price for consecutive days and the date are displayed on the matching visualization space layout diagram as the data mapping to be visualized; the visualization space layout diagram includes a statistical diagram, a time series diagram, a hierarchical diagram, a network diagram, a geographic location diagram and a tag cloud.

7. The method for monitoring and visualizing agricultural product prices based on big data according to claim 4, characterized in that: The method for automatically collecting agricultural big data based on streaming computing is used to collect the historical average daily wholesale prices of agricultural products to be predicted in the first preset area for multiple consecutive days, and then further includes: Perform noise cleaning on the historical wholesale daily average price; Use the historical average method or time average method to repair the missing data of the historical wholesale daily average price after noise cleaning; The calculation formula of the historical average method is: Among them, M is the missing historical wholesale daily average price, y j is the historical wholesale daily average price on the same weekday as the missing historical wholesale daily average price in the j weeks before the missing historical wholesale daily average price, and K is the week number; The calculation formula of the time averaging method is: Among them, x1 and x2 are the historical wholesale daily average prices of the day before and the day after the missing historical wholesale daily average price.

8. The method for monitoring and visualizing agricultural product prices based on big data according to claim 4, characterized in that: The method for determining the trained extreme random tree model is: The automatic collection method of agricultural big data based on streaming computing is used to collect multiple historical wholesale daily prices of the agricultural products to be predicted in the first preset area in the same wholesale market over the years. Calculate the historical average daily wholesale price samples of the agricultural products to be predicted in the first preset area over the years according to the collected multiple historical wholesale daily prices; The historical wholesale daily average price samples of previous years are divided into multiple daily average price sample sequences using a sliding window with a preset sliding window length; Extract the time characteristics of the last historical wholesale daily average price in each daily average price sample sequence; The last historical wholesale daily average price in each daily average price sample sequence is used as a label, and all historical wholesale daily average prices other than the last historical wholesale daily average price in each daily average price sample sequence and the time characteristics of the last historical wholesale daily average price are used as input to form a daily average price sample data set; Divide the daily average price sample data set into a training set and a validation set; Using the training set to train the extreme random tree model, obtaining a trained extreme random tree model; The trained extreme randomized tree model is optimized using the validation set to obtain an optimized extreme randomized tree model.

9. The method for monitoring and visualizing agricultural product prices based on big data according to claim 8, characterized in that: The method further includes taking the average daily wholesale price of the agricultural product to be predicted on the predicted day, the average daily wholesale price of the agricultural product to be predicted for consecutive days and the date as the data to be visualized, and selecting a visualization tool for visualization display according to the characteristics of the data to be visualized, and then further including: According to the training set, using the trained extreme random tree model, obtaining a training error; Based on the preset confidence level, the training error is input into the generalized autoregressive conditional heteroskedasticity model, and the confidence interval under the preset confidence level is output; The generalized autoregressive conditional heteroskedasticity model is Among them, h it is the conditional heteroskedasticity of the ith random variable at time t, and are the first, second and third model parameters, m and n are the first and second lag orders, M and N are the total number of the first and second lag orders, ε it-m is the observed value of the i-th random variable at time tm, h it-n is the conditional heteroskedasticity of the ith random variable at time tn.

Images