Map-based economic data automatic analysis system and method
By designing a map-based automatic economic data analysis system, combining principal component analysis and machine learning models for data analysis, and using GIS tools to display the results, the problem of difficult to identify and update the changing trends of economic data in the existing technology is solved, and efficient and accurate economic data analysis and prediction are achieved.
Patent Information
- Application Number
- CN202411694299.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-14
AI Technical Summary
The existing technology is difficult to quickly identify the trends and rules of economic data changes, and cannot be updated in real time.
Design a map-based automatic economic data analysis system, including data acquisition module, data processing module, data analysis module and visualization module. Data is collected through API interfaces and network crawler technology, data is processed using Python Pandas library, data analysis is performed in combination with principal component analysis and machine learning models, and analysis results are displayed in map form using GIS tools.
It realizes rapid collection and real-time update of economic data, can capture complex economic phenomena more accurately, improve the robustness and accuracy of predictions, and make the data analysis results more intuitive.
Smart Images
Figure CN119624514A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data analysis, and in particular, relates to a system and method for automatic analysis of economic data based on a map. Background Art
[0002] With the rapid development of big data technology, the collection and analysis of economic data has become increasingly important. Economic data not only provides important decision-making basis for policymakers, but also provides researchers and enterprises with tools to understand market dynamics and predict trends. In today's era of information explosion, the amount and types of data are increasing dramatically. How to effectively extract, analyze and use this data has become a key issue. Traditional economic data analysis methods usually rely on static tables and graphs. Although these methods can help people understand economic characteristics to a certain extent, their intuitiveness and practicality have significant limitations. First, static tables can only reflect data at a single point in time and lack a dynamic display of data changes over time. This static method cannot effectively show the spatial relationship between economic data, such as the comparison of economic levels in different regions and the differences in resource distribution. Therefore, when decision makers try to analyze the impact of a policy on the economy of different regions, the analysis that relies solely on tables and graphs appears very one-sided. Secondly, many economic phenomena have significant geographical characteristics. For example, the employment rate of a city is closely related to its geographical location, transportation convenience, and industrial structure. In this case, traditional data display methods are difficult to reflect the spatial distribution and correlation of data. This requires analysts to rely on in-depth background knowledge and experience judgment, resulting in increased subjectivity in the analysis conclusions. In this context, map-based economic data analysis systems have gradually become more important. Such systems use geographic information system (GIS) technology to combine economic data with geographic information, thereby intuitively displaying the geographical distribution of economic activities and their dynamic changes. For example, the GDP level of a city can be displayed through a heat map, helping decision makers quickly identify areas with faster or slower economic development. This visual display method greatly improves the readability of information and decision-making efficiency. Summary of the invention
[0003] The purpose of the present invention is to provide a map-based automatic economic data analysis system and method to solve the problem in the prior art that trends and laws of economic data changes are difficult to quickly identify and cannot be updated in real time.
[0004] To achieve the above-mentioned purpose, the present invention provides a map-based automatic economic data analysis system, comprising a data acquisition module, a data processing module, a data analysis module and a visualization module;
[0005] The data collection module collects economic data and geographic information data autonomously through API interface and web crawler technology;
[0006] The data processing module uses the Python Pandas library to process the original data;
[0007] The data analysis module uses statistical analysis tools to perform data analysis and predict future economic indicators;
[0008] The visualization module uses GIS tools to display the analysis results in the form of a map, highlighting areas with high / low economic indicators.
[0009] Preferably, the economic data includes GDP, unemployment rate and consumer price index.
[0010] Preferably, the geographic information data includes city boundaries and population density.
[0011] Preferably, the GIS tool is ArcGIS or Leaflet library.
[0012] The present invention also provides a method for automatically analyzing economic data based on a map, comprising the following steps:
[0013] Step 1: Collect economic data and geographic information data autonomously through API interface and web crawler technology, and set tasks for regular automatic updates;
[0014] Step 2: Use the Python Pandas library to process the raw data and store the processed data in a relational database such as MySQL or a NoSQL database.
[0015] Step 3: Use statistical analysis tools to analyze data and predict future economic indicators;
[0016] Step 4. Use GIS tools to display the analysis results in the form of a map, and users can view them according to the different economic indicators they choose. Specifically, based on the results predicted in the analysis phase, use Pandas to import data, use Folium to create a base map, and set the starting position of the map. Dynamically add markers based on different economic indicators (such as GDP, unemployment rate), and finally display them on the map. Use front-end frameworks such as React, Vue.js, or Dash, Flask, etc. to create a web application that allows users to view them according to the selected economic indicators. Finally, based on the selected indicators, the map is dynamically updated to display the corresponding city markers.
[0017] Preferably, the specific process of setting the task of periodic automatic update in step 1 is as follows:
[0018] S11. Determine the update requirements, including data types and update frequency; specifically, identify the data that needs to be updated regularly, and then clarify the update cycle for each type of data;
[0019] S12. Use Windows Task Scheduler to set up tasks; the specific process is as follows:
[0020] S121. Open the Task Scheduler.
[0021] S122. Select “Create Basic Task”.
[0022] S123. Fill in the task name and description according to the wizard.
[0023] S124. Select a trigger (eg, "daily").
[0024] S125. Set the start time and complete other basic settings.
[0025] S126. Select "Start Program" in the "Action" option and enter the Python path and script path.
[0026] S127, complete the setting;
[0027] S13. Implement monitoring and logging; establish a monitoring mechanism to record the execution of each task.
[0028] Preferably, in step 3, the process of using statistical analysis tools to perform data analysis and predict future economic indicators is as follows:
[0029] S31. Obtain economic data in various fields of the city;
[0030] S32, using PCA to reduce the dimension of the data and extract the first p principal components as training samples;
[0031] S33, input the training samples extracted in S32 into the BP neural network and the random forest respectively, and obtain economic forecasting models Z1 and Z2 respectively;
[0032] S34, calculating and obtaining predicted economic values G1, G2 according to economic forecasting models Z1, Z2;
[0033] S35, introducing the three sets of predicted economic values into the regression equation w1*G1+w2*G2=g, and calculating the voting weights w1 and w2, where g is the actual economic value;
[0034] S36. Predict the city's future economy, calculate the prediction results g1 and g2 of the economic prediction models Z1 and Z2 respectively, and calculate the final economic prediction result F=w1*g1+w2*g2 based on the voting weights w1 and w2.
[0035] Preferably, the process of using PCA to reduce the dimension of data in S32 is as follows:
[0036] S321. Data centering: mean centering for each feature. The formula is:
[0037]
[0038] Where X is the original data matrix, is the mean vector of each column feature, X centered represents the data center matrix;
[0039] S322. Calculate the covariance matrix Cov(X) of the data matrix. The formula is:
[0040]
[0041] Where m is the number of samples;
[0042] S323, eigenvalue decomposition, perform eigenvalue decomposition on the covariance matrix, the formula is:
[0043] Cov(X)=QΛQ T
[0044] Where Q is the eigenvector matrix, Λ is the diagonal eigenvalue matrix;
[0045] S324, select the principal component, and select the eigenvectors with the largest eigenvalues as the principal component;
[0046] S3245, construct projection matrix Q selected , the formula is
[0047] Q selected =[v 1 ,v 2 ,...,v k ]
[0048] Among them, v k represents the kth eigenvector.
[0049] Preferably, in S33, the expression of inputting the training sample into the BP neural network to obtain the economic forecasting model Z1 is as follows:
[0050]
[0051] Where L is the input of the output layer node, v j is the connection weight from the hidden layer to the output layer; γ is the threshold of the output layer; p is the hidden layer; b j is the output of the hidden layer nodes.
[0052] Preferably, the process of inputting the training samples into the random forest to obtain the economic forecasting model Z2 in S33 is as follows:
[0053] S331. Use the Gini algorithm to select the feature with the smallest Gini index from m features as the classification point for splitting to form a single classification tree. The calculation expression is as follows:
[0054]
[0055] In the formula, T represents the number of decision trees, p j represents the frequency of T in category j;
[0056] S332, calculate feature X j The importance at classification node m is calculated as follows:
[0057] V ij (Gini) =GI m -GI f -GI g
[0058] Where V ij (Gini) Represents each feature X j Gini index score expression; GI m Represents the Gini index of node m, GI f and GI g Represents the Gini index of the two new nodes obtained after node m is classified;
[0059] S333, feature X j The nodes appearing in the classification tree m form a set M. The Gini coefficient of each indicator on the classification tree is calculated to determine the indicator importance, and the economic forecasting model Z2 is obtained. The calculation expression is as follows:
[0060]
[0061] Where D' n,m represents the training set with n classification trees, (n, ν) is the coefficient parameter, m is the number of classification trees, and ν is the number of random indicators.
[0062] Therefore, the present invention adopts the above-mentioned map-based economic data automatic analysis system and method, which has the following beneficial effects:
[0063] (1) Autonomous collection of economic and geographic information data through API interfaces and web crawler technology greatly reduces manual intervention, thereby improving the efficiency of data acquisition. At the same time, regular automatic data updates ensure the timeliness of information and help decision makers grasp economic dynamics in real time;
[0064] (2) Combining principal component analysis (PCA) and machine learning models (BP neural network and random forest) for data analysis and economic forecasting can more accurately capture complex economic phenomena, improve the robustness and accuracy of forecasts, integrate the forecast results of different models, and reduce the potential errors of a single model;
[0065] (3) Use GIS tools (such as ArcGIS or Leaflet library) to display economic data in the form of maps, making the data analysis results more intuitive and easy to understand.
[0066] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 It is a structural schematic diagram of a map-based automatic economic data analysis system of the present invention;
[0068] Figure 2 The present invention is a flowchart of an overall process of a map-based automatic economic data analysis method. DETAILED DESCRIPTION
[0069] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0070] See also Figure 1 , an automatic economic data analysis system based on a map, including a data acquisition module, a data processing module, a data analysis module and a visualization module;
[0071] The data collection module collects economic data and geographic information data autonomously through API interfaces and web crawler technology; economic data includes GDP, unemployment rate and consumption index; geographic information data includes city boundaries and population density;
[0072] The data processing module uses the Python Pandas library to process the raw data;
[0073] The data analysis module uses statistical analysis tools to analyze data and predict future economic indicators;
[0074] The visualization module uses GIS tools to display the analysis results in the form of maps, highlighting areas with high / low economic indicators; the GIS tools are ArcGIS or Leaflet library.
[0075] See also Figure 2, a map-based automatic economic data analysis method, comprising the following steps:
[0076] Step 1: Use API interface and web crawler technology to autonomously collect economic data and geographic information data, and set up regular automatic update tasks; the specific process of setting up regular automatic update tasks is as follows:
[0077] S11. Determine the update requirements, including data types and update frequency; specifically, identify the data that needs to be updated regularly, and then clarify the update cycle for each type of data;
[0078] S12. Use Windows Task Scheduler to set up tasks; the specific process is as follows:
[0079] S121. Open the Task Scheduler.
[0080] S122. Select “Create Basic Task”.
[0081] S123. Fill in the task name and description according to the wizard.
[0082] S124. Select a trigger (eg, "daily").
[0083] S125. Set the start time and complete other basic settings.
[0084] S126. Select "Start Program" in the "Action" option and enter the Python path and script path.
[0085] S127. Complete the settings.
[0086] S13. Implement monitoring and logging; establish a monitoring mechanism to record the execution of each task.
[0087] Step 2: Use the Python Pandas library to process the raw data and store the processed data in a relational database such as MySQL or a NoSQL database.
[0088] Step 3: Use statistical analysis tools to analyze data and predict future economic indicators; the specific process is as follows:
[0089] S31. Obtain economic data in various fields of the city;
[0090] S32, using PCA to reduce the data dimension, extracting the first p principal components as training samples; wherein, the process of using PCA to reduce the data dimension is as follows:
[0091] S321. Data centering: mean centering for each feature. The formula is:
[0092]
[0093] Where X is the original data matrix, is the mean vector of each column feature, X centered represents the data center matrix;
[0094] S322. Calculate the covariance matrix Cov(X) of the data matrix. The formula is:
[0095]
[0096] Where m is the number of samples;
[0097] S323, eigenvalue decomposition, perform eigenvalue decomposition on the covariance matrix, the formula is:
[0098] Cov(X)=QΛQ T
[0099] Where Q is the eigenvector matrix, Λ is the diagonal eigenvalue matrix;
[0100] S324, select the principal component, and select the eigenvectors with the largest eigenvalues as the principal component;
[0101] S3245, construct projection matrix Q selected , the formula is
[0102] Q selected =[v 1 ,v 2 ,...,v k ]
[0103] Among them, v k represents the kth eigenvector.
[0104] S33, input the training samples extracted in S32 into the BP neural network and the random forest respectively, and obtain economic forecasting models Z1 and Z2 respectively;
[0105] Among them, the expression of the economic forecasting model Z1 obtained by inputting the training samples into the BP neural network is as follows:
[0106]
[0107] Where L is the input of the output layer node, v j is the connection weight from the hidden layer to the output layer; γ is the threshold of the output layer; p is the hidden layer; b j is the output of the hidden layer nodes.
[0108] The process of inputting the training samples into the random forest to obtain the economic forecasting model Z2 is as follows:
[0109] S331. Use the Gini algorithm to select the feature with the smallest Gini index from m features as the classification point for splitting to form a single classification tree. The calculation expression is as follows:
[0110]
[0111] In the formula, T represents the number of decision trees, p j represents the frequency of T in category j;
[0112] S332, calculate feature X j The importance at classification node m is calculated as follows:
[0113] V ij (Gini) =GI m -GI f -GI g
[0114] Where V ij (Gini) Represents each feature X j Gini index score expression; GI m Represents the Gini index of node m, GI f and GI g Represents the Gini index of the two new nodes obtained after node m is classified;
[0115] S333, feature X j The nodes appearing in the classification tree m form a set M. The Gini coefficient of each indicator on the classification tree is calculated to determine the indicator importance, and the economic forecasting model Z2 is obtained. The calculation expression is as follows:
[0116]
[0117] Where D' n,m Represents a training set with n classification trees, (n, ν) is the coefficient parameter, m is the number of classification trees, and ν is the number of random indicators
[0118] S34, calculating and obtaining predicted economic values G1, G2 according to economic forecasting models Z1, Z2;
[0119] S35, introducing the three sets of predicted economic values into the regression equation w1*G1+w2*G2=g, and calculating the voting weights w1 and w2, where g is the actual economic value;
[0120] S36. Predict the city's future economy, calculate the prediction results g1 and g2 of the economic prediction models Z1 and Z2 respectively, and calculate the final economic prediction result F=w1*g1+w2*g2 based on the voting weights w1 and w2.
[0121] Step 4. Use GIS tools to display the analysis results in the form of a map, and users can view them according to the different economic indicators they choose. Specifically, based on the results predicted in the analysis phase, use Pandas to import data, use Folium to create a base map, and set the starting position of the map, dynamically add markers according to different economic indicators (such as GDP, unemployment rate), and finally display them on the map. Use front-end frameworks such as React, Vue.js, or Dash, Flask, etc. to create a web application that allows users to view them according to the selected economic indicators. Finally, based on the selected indicators, the map is dynamically updated to display the corresponding city markers.
[0122] Therefore, the present invention adopts the above-mentioned map-based automatic economic data analysis system and method, combines principal component analysis, BP neural network and random forest to perform data analysis and economic forecasting, which can more accurately capture complex economic phenomena, improve the robustness and accuracy of forecasts, integrate the forecast results of different models, and reduce the potential errors of a single model. Finally, GIS tools are used to display economic data in the form of maps, making the data analysis results more intuitive and easy to understand.
[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A map-based automatic economic data analysis system, characterized by: It includes data acquisition module, data processing module, data analysis module and visualization module; The data collection module collects economic data and geographic information data autonomously through API interface and web crawler technology; The data processing module uses the Python Pandas library to process the original data; The data analysis module uses statistical analysis tools to perform data analysis and predict future economic indicators; The visualization module uses GIS tools to display the analysis results in the form of a map, highlighting areas with high / low economic indicators.
2. The map-based automatic economic data analysis system according to claim 1, characterized in that: Economic data includes GDP, unemployment rate and consumer price index.
3. The map-based automatic economic data analysis system according to claim 1, characterized in that: Geographic information data includes city boundaries and population density.
4. The map-based automatic economic data analysis system according to claim 1, characterized in that: GIS tools include ArcGIS or Leaflet library.
5. A map-based automatic economic data analysis method, characterized in that: The following steps are involved: Step 1: Collect economic data and geographic information data autonomously through API interface and web crawler technology, and set tasks for regular automatic updates; Step 2: Use the Python Pandas library to process the raw data and store the processed data in a relational database such as MySQL or a NoSQL database. Step 3: Use statistical analysis tools to analyze data and predict future economic indicators; Step 4: Use GIS tools to display the analysis results in the form of a map, and users can view them based on the different economic indicators they choose.
6. The method for automatic analysis of economic data based on a map according to claim 5, characterized in that: The specific process of setting up the task of regular automatic update in step 1 is as follows: S11. Determine the update requirements, including data types and update frequency; specifically, identify the data that needs to be updated regularly, and then clarify the update cycle for each type of data; S12. Use Windows Task Scheduler to set up tasks; S13. Implement monitoring and logging; establish a monitoring mechanism to record the execution of each task.
7. The method for automatic analysis of economic data based on a map according to claim 6, characterized in that: In step 3, statistical analysis tools are used to analyze data and predict future economic indicators as follows: S31. Obtain economic data in various fields of the city; S32, using PCA to reduce the dimension of the data and extract the first p principal components as training samples; S33, input the training samples extracted in S32 into the BP neural network and the random forest respectively, and obtain economic forecasting models Z1 and Z2 respectively; S34, calculating and obtaining predicted economic values G1, G2 according to economic forecasting models Z1, Z2; S35, introducing the three sets of predicted economic values into the regression equation w1*G1+w2*G2=g, and calculating the voting weights w1 and w2, where g is the actual economic value; S36. Predict the future economy of the city, calculate the prediction results g1 and g2 of the economic prediction models Z1 and Z2 respectively, and calculate the final economic prediction result F=w1*g1+w2*g2 according to the voting weights w1 and w2.
8. The method for automatic analysis of economic data based on a map according to claim 7, characterized in that: The process of using PCA to reduce the dimension of data in S32 is as follows: S321. Data centering: mean centering for each feature. The formula is: Where X is the original data matrix, is the mean vector of each column feature, X centered represents the data center matrix; S322. Calculate the covariance matrix Cov(X) of the data matrix. The formula is: Where m is the number of samples; S323, eigenvalue decomposition, perform eigenvalue decomposition on the covariance matrix, the formula is: Cov(X)=QΛQ T Where Q is the eigenvector matrix, Λ is the diagonal eigenvalue matrix; S324, select the principal component, and select the eigenvectors with the largest eigenvalues as the principal component; S3245, construct projection matrix Q selected , the formula is Q selected =[v1,v2,...,v k ] Among them, v k represents the kth eigenvector.
9. The method for automatic analysis of economic data based on a map according to claim 8, characterized in that: In S33, the training samples are input into the BP neural network to obtain the expression of the economic forecasting model Z1 as follows: Where L is the input of the output layer node, ν j is the connection weight from the hidden layer to the output layer; γ is the threshold of the output layer; p is the hidden layer; b j is the output of the hidden layer nodes.
10. The method for automatic analysis of economic data based on a map according to claim 9, characterized in that: The process of inputting the training samples into the random forest to obtain the economic forecasting model Z2 in S33 is as follows: S331. Use the Gini algorithm to select the feature with the smallest Gini index from m features as the classification point for splitting to form a single classification tree. The calculation expression is as follows: In the formula, T represents the number of decision trees, p j represents the frequency of T in category j; S332, calculate feature X j The importance at classification node m is calculated as follows: V ij (Gini) =G Im -GI f -GI g Where V ij (Gini) Represents each feature X j Gini index score expression; GI m Represents the Gini index of node m, GI f and GI g Represents the Gini index of the two new nodes obtained after node m is classified; S333, feature X j The nodes appearing in the classification tree m form a set M. The Gini coefficient of each indicator on the classification tree is calculated to determine the indicator importance, and the economic forecasting model Z2 is obtained. The calculation expression is as follows: Where D' n,m represents the training set with n classification trees, (n, ν) is the coefficient parameter, m is the number of classification trees, and ν is the number of random indicators.