Resident consumption economic data analysis method and system
Through network crawlers collecting and preprocessing economic data, combining time series and econometric models to analyze and predict, the problems of low data quality and inaccurate analysis of models in the existing technology are solved, and more efficient and accurate economic data prediction is achieved.
Patent Information
- Application Number
- CN202411694289.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-14
AI Technical Summary
The existing economic data analysis methods in the fields of residents' income and consumption have problems such as low data quality, lack of and abnormal data not being processed, and the inability of traditional linear models to accurately analyze different regions and consumption fields.
A large amount of economic data is collected by keyword search through web crawlers, pre-processing and identifying missing values, outliers and duplicate data, and preliminary analysis and prediction are performed using time series models, and further predictions are made in combination with econometric models.
It improves the accuracy and richness of economic data and enhances the accuracy and robustness of data predictions.
Smart Images
Figure CN119624498A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data analysis, and in particular to a method and system for analyzing resident consumption economic data. Background Art
[0002] Economic data is information that quantifies and records all aspects of economic activities, including overall economic indicators, such as gross domestic product (GDP), gross national product (GNP), etc.; production-related data, such as industrial added value, agricultural production data, service industry-related data, etc.; price-related data, such as consumer price index (CPI), producer price index (PPI), etc.; employment and labor market data, such as unemployment rate, labor force participation rate, new employment, income and consumption data, etc.; investment-related data, such as fixed asset investment, real estate investment, etc.; trade and balance of payments data, such as total imports and exports, foreign exchange reserves, etc.; financial-related data, such as money supply, interest rates, etc. With the rapid development of big data technology, the collection and analysis of economic data has become increasingly important. Economic data provides companies and researchers with tools to understand market dynamics and predict trends.
[0003] The existing economic data analysis methods in the field of residents' income and consumption have the following problems: data quality: due to the manual collection of economic data from official websites, the amount of economic data collected is small, which affects the accuracy of the forecast; integrity problem: the missing data collected are not processed accordingly, which affects the accuracy of the forecast; forecast model establishment problem: many traditional analysis methods are based on linear assumptions, but in fact, the impact of income on consumption may be different in different regions and different consumption fields. It is impossible to accurately analyze and predict economic data through a simple linear model. Summary of the invention
[0004] The purpose of the present invention is to provide a method and system for analyzing economic data on resident consumption, which collects a large amount of economic data by searching by keywords through a web crawler to increase the richness of the data, pre-processes the collected economic data, identifies missing values, outliers and duplicate data in the data and processes them to improve the accuracy of the data, and conducts preliminary analysis and prediction of the economic data through a time series model to complete the preliminary prediction, uses the prediction results as input variables, and conducts further predictions in combination with the variables of the econometric model to improve the accuracy of data predictions.
[0005] To achieve the above object, the present invention provides a method for analyzing resident consumption economic data, comprising the following steps:
[0006] Step S1, data collection: collect economic data of each province by searching by keywords through a web crawler;
[0007] Step S2, data processing and cleaning: pre-process the collected economic data, identify missing values, outliers and duplicate data in the data and process them;
[0008] Step S3, data analysis and forecasting: Analyze the economic data through the time series model, complete the preliminary forecast, use the forecast results as input variables, and combine them with the variables of the econometric model to make further forecasts;
[0009] Step S4, data storage: the analyzed and predicted data are stored in the cloud via a cloud server.
[0010] Preferably, step S1 includes the following steps:
[0011] S11, set keywords and determine target websites, including provincial and municipal statistical bureau websites, professional data platforms and financial websites, and academic research databases;
[0012] S12, select Python's Scrapy framework and Java's WebMagic to crawl, parse, extract and store Internet web page data to complete the collection of economic data.
[0013] Preferably, step S2 includes the following steps:
[0014] S21, data preprocessing: the collected economic data are classified according to consumption areas and income groups; consumption areas include food, clothing, housing, daily necessities and services, transportation and communication, education, entertainment and culture, and medical care; income groups include urban and rural residents’ income and income of groups of different ages;
[0015] S22, Identification and processing of missing values: draw scatter plots, observe the distribution of economic data, identify missing values, and construct difference functions through cubic spline interpolation to process missing values;
[0016] S23, Identification and processing of outliers: draw a box plot to show the distribution of economic data. If the outliers are less than 5% of the total data, delete the outliers. If the outliers are greater than or equal to 5% of the total data, determine the median of the non-outliers in the economic data and replace the outliers with the median.
[0017] S24, identification and processing of duplicate data: Identify duplicate data through Excel's conditional formatting or pivot table and delete them.
[0018] Preferably, step S3 includes the following steps:
[0019] S31, Analyze and forecast economic data through time series models;
[0020] S32, analyze and predict economic data through econometric models.
[0021] Preferably, S31 includes the following steps:
[0022] S311, divide 80% of the economic data into training sets and the remaining data into test sets;
[0023] S312, based on the research questions, identify key variables related to income and consumption areas of residents of different ages, and transform the variables into a form that meets the requirements of the time series model through logarithmic transformation;
[0024] S313, select the autoregressive integrated moving average model ARIMA, convert the data in the training set into a stationary sequence by the difference method, and then perform modeling;
[0025] S314, using the training set data to estimate the selected autoregressive integrated moving average model ARIMA, determine the autoregressive order p, the moving average order q and the number of differences d in the ARIMA model, and evaluate the performance of the model by analyzing the autocorrelation, normality and stability of the residuals;
[0026] The formula for the autoregressive integrated moving average model (ARIMA(p,d,q)) is as follows:
[0027] Φ(L)(1-L) d Y t =θ(L)λ t +c;
[0028] Among them, Y t is the value of the time series at time; t is the white noise error term; c is the constant term; L is the lag operator, LY t =Y t-1 ;Φ(L)=1-Lφ1-L 2 φ2-…L n φ n , φ1, φ2...φ n is the autoregressive coefficient; (1-L) d Indicates a secondary differential operation. When d = 1, (1-L)Y t =Y t -Y t-1 ;θ(L)=1+Lθ1+L 2 θ2+…+L m θ m ,θ1,θ2...θ m is the moving average coefficient;
[0029] S315, use the estimated model to predict the relationship between the income of residents of different ages and different consumption areas in the future, and predict the income and consumption of residents of different ages every month in the next year.
[0030] Preferably, S32 includes the following steps:
[0031] S321, taking the prediction result output in S31 as an input variable, and determining another new input variable as the urban and rural residents' income data;
[0032] S322, select the structural equation model, take the two input variables as observed variables, and the latent variable is the consumption capacity of urban and rural residents of different ages;
[0033] S323, based on economic theory and existing research, proposes hypotheses about the relationship between latent variables and observed variables, which form the basis of the structural equation model;
[0034] S324, Use graphical representation to draw the path diagram of the structural equation model to show the causal relationship and influence direction between latent variables and observed variables.
[0035] S325, use the maximum likelihood estimation method to estimate the parameters of the structural equation model, and verify the estimated structural equation model;
[0036] S326, substitute the known observed variable values into the model, calculate the estimated value of the latent variable through the model, and predict the consumption data of urban and rural residents of different ages every month in the next year based on the relationship between the latent variable and the economic data.
[0037] Preferably, the structural equation model includes measurement equations and structural equations in the following form:
[0038] Measurement equation:
[0039] For observed variables:
[0040]
[0041]
[0042] Among them, x1 is a vector of exogenous observed variables; x2 is a vector of endogenous observed variables; is the relationship matrix between exogenous observed variables and exogenous latent variables; is the relationship matrix between endogenous observed variables and endogenous latent variables; ξ is the vector of exogenous latent variables; η is the vector of endogenous latent variables; α is the vector of measurement errors of exogenous observed variables; β is the vector of measurement errors of endogenous observed variables.
[0043] Structural equation:
[0044] η=Qη+Pξ+ζ
[0045] Among them, Q is the relationship matrix between endogenous latent variables, which represents the influence between endogenous latent variables; P is the influence matrix of exogenous latent variables on endogenous latent variables; ζ is the error term of the structural equation.
[0046] Preferably, in step S4, the prediction data obtained in step S3 is uploaded to the cloud server for storage, and the data is encrypted, and identity authentication and authorization are required for use.
[0047] The present invention also provides a system for analyzing the residents' consumption economic data, comprising:
[0048] Automatic data collection module: collect economic data of each province by searching by keywords through web crawlers;
[0049] Data processing and cleaning module: pre-process and classify the collected economic data, identify missing values, outliers and duplicate data in the data and process them;
[0050] Data analysis and forecasting module: forecast economic data by combining time series models with variables of econometric models;
[0051] Data storage and viewing module: The analyzed and predicted data are stored in the cloud through the cloud server, and the data can be viewed and retrieved by authorized users logging in;
[0052] System security and management module: performs user authentication and authorization, encrypts and backs up data, monitors the system, records the system's operation logs and security logs, and promptly detects and prevents network attacks to maintain system security.
[0053] Therefore, the present invention adopts the above-mentioned method and system for analyzing resident consumption economic data, which has the following beneficial effects:
[0054] (1) The present invention collects a large amount of economic data by automatically searching by keywords through a web crawler, which not only reduces manual intervention but also increases the richness of data, thereby improving the efficiency of economic data collection, laying a foundation for subsequent data analysis and prediction, and improving the accuracy of prediction; by identifying missing values, abnormal values, and duplicate data in economic data and processing them, erroneous data is reduced, the accuracy of data is improved, and the accuracy of subsequent data prediction is improved;
[0055] (2) Make preliminary forecasts of economic data through time series models, use the forecast results as input variables, and make further forecasts in combination with the variables of the econometric model, so as to organically combine the forecasting methods of the two models and improve the robustness and accuracy of the forecast.
[0056] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical scheme and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention are further described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention and are not intended to limit the embodiments of the present invention. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions.
[0059] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or devices.
[0060] Like reference numerals and letters denote similar items in the following drawings, and thus, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0061] In the description of the present invention, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.
[0062] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "setting", "installation" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0063] Example
[0064] like Figure 1 As shown, a method for analyzing resident consumption economic data of the present invention comprises the following steps:
[0065] Step S1, data collection: collect economic data of each province by searching by keywords through web crawlers. The specific steps are as follows:
[0066] S11, set keywords and determine target websites, including provincial and municipal statistical bureau websites, professional data platforms and financial websites, and academic research databases;
[0067] S12, select Python's Scrapy framework and Java's WebMagic to crawl, parse, extract and store Internet web page data to complete the collection of economic data.
[0068] Step S2, data processing and cleaning: pre-process the collected economic data, identify missing values, outliers and duplicate data in the data and process them. The specific steps are as follows:
[0069] S21, data preprocessing: the collected economic data are classified according to consumption areas and income groups; consumption areas include food, clothing, housing, daily necessities and services, transportation and communication, education, entertainment and culture, and medical care; income groups include urban and rural residents’ income and income of groups of different ages;
[0070] S22, Identification and processing of missing values: draw scatter plots, observe the distribution of economic data, identify missing values, and construct difference functions through cubic spline interpolation to process missing values;
[0071] S23, Identification and processing of outliers: draw a box plot to show the distribution of economic data. If the outliers are less than 5% of the total data, delete the outliers. If the outliers are greater than or equal to 5% of the total data, determine the median of the non-outliers in the economic data and replace the outliers with the median.
[0072] S24, identification and processing of duplicate data: Identify duplicate data through Excel's conditional formatting or pivot table and delete them.
[0073] Step S3, data analysis and forecasting: Analyze economic data through time series model, complete preliminary forecast, use forecast results as input variables, and combine with variables of econometric model for further forecasting. The specific steps are as follows:
[0074] S31, Analyze and predict economic data through time series models:
[0075] S311, divide 80% of the economic data into training sets and the remaining data into test sets;
[0076] S312, based on the research questions, identify key variables related to income and consumption areas of residents of different ages, and transform the variables into a form that meets the requirements of the time series model through logarithmic transformation;
[0077] S313, select the autoregressive integrated moving average model ARIMA, convert the data in the training set into a stationary sequence by the difference method, and then perform modeling;
[0078] S314, using the training set data to estimate the selected autoregressive integrated moving average model ARIMA, determine the autoregressive order p, the moving average order q and the number of differences d in the ARIMA model, and evaluate the performance of the model by analyzing the autocorrelation, normality and stability of the residuals;
[0079] The formula for the autoregressive integrated moving average model (ARIMA(p,d,q)) is as follows:
[0080] Φ(L)(1-L) d Y t =θ(L)λ t +c;
[0081] Among them, Y t is the value of the time series at time; t is the white noise error term; c is the constant term; L is the lag operator, LY t =Y t-1 ;Φ(L)=1-Lφ1-L 2 φ2-…L n φ n , φ1, φ2...φ n is the autoregressive coefficient; (1-L) d Indicates a secondary differential operation. When d = 1, (1-L)Y t =Y t -Y t-1 ;θ(L)=1+Lθ1+L 2 θ2+…+L m θ m ,θ1,θ2...θ m is the moving average coefficient;
[0082] S315, use the estimated model to predict the relationship between the income of residents of different ages and different consumption areas in the future, and predict the income and consumption of residents of different ages every month in the next year.
[0083] S32, Analyze and forecast economic data through econometric models:
[0084] S321, taking the prediction result output in S31 as an input variable, and determining another new input variable as the urban and rural residents' income data;
[0085] S322, select the structural equation model, take the two input variables as observed variables, and the latent variable is the consumption capacity of urban and rural residents of different ages;
[0086] S323, based on economic theory and existing research, proposes hypotheses about the relationship between latent variables and observed variables, which form the basis of the structural equation model;
[0087] S324, Use graphical representation to draw the path diagram of the structural equation model to show the causal relationship and influence direction between latent variables and observed variables.
[0088] S325, use the maximum likelihood estimation method to estimate the parameters of the structural equation model, and verify the estimated structural equation model;
[0089] S326, substitute the known observed variable values into the model, calculate the estimated value of the latent variable through the model, and predict the consumption data of urban and rural residents of different ages every month in the next year based on the relationship between the latent variable and the economic data.
[0090] The structural equation model includes measurement equations and structural equations in the following form:
[0091] Measurement equation:
[0092] For observed variables:
[0093]
[0094]
[0095] Among them, x1 is a vector of exogenous observed variables; x2 is a vector of endogenous observed variables; is the relationship matrix between exogenous observed variables and exogenous latent variables; is the relationship matrix between endogenous observed variables and endogenous latent variables; ξ is the vector of exogenous latent variables; η is the vector of endogenous latent variables; α is the vector of measurement errors of exogenous observed variables; β is the vector of measurement errors of endogenous observed variables.
[0096] Structural equation:
[0097] η=Qη+Pξ+ζ
[0098] Among them, Q is the relationship matrix between endogenous latent variables, which represents the influence between endogenous latent variables; P is the influence matrix of exogenous latent variables on endogenous latent variables; ζ is the error term of the structural equation.
[0099] Step S4, data storage: upload the prediction data obtained in step S3 to the cloud server for storage, and encrypt the data, requiring identity authentication and authorization when using it.
[0100] The system of the method for analyzing residents' consumption economic data of the present invention comprises:
[0101] Automatic data collection module: collect economic data of each province by searching by keywords through web crawlers;
[0102] Data processing and cleaning module: pre-process and classify the collected economic data, identify missing values, outliers and duplicate data in the data and process them;
[0103] Data analysis and forecasting module: forecast economic data by combining time series models with variables of econometric models;
[0104] Data storage and viewing module: The analyzed and predicted data are stored in the cloud through the cloud server, and the data can be viewed and retrieved by authorized users logging in;
[0105] System security and management module: performs user authentication and authorization, encrypts and backs up data, monitors the system, records the system's operation logs and security logs, and promptly detects and prevents network attacks to maintain system security.
[0106] Therefore, the present invention adopts the above-mentioned method and system for analyzing economic data on resident consumption, collects a large amount of economic data by searching by keywords through a web crawler, increases the richness of the data, pre-processes the collected economic data, identifies missing values, outliers, and duplicate data in the data and processes them, improves the accuracy of the data, and conducts preliminary analysis and prediction of the economic data through a time series model to complete the preliminary prediction, uses the prediction results as input variables, and conducts further predictions in combination with the variables of the econometric model to improve the accuracy of data predictions.
[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A method for analyzing resident consumption economic data, characterized by: The following steps are involved: Step S1, data collection: collect economic data of each province by searching by keywords through a web crawler; Step S2, data processing and cleaning: pre-process the collected economic data, identify missing values, outliers and duplicate data in the data and process them; Step S3, data analysis and forecasting: Analyze the economic data through the time series model, complete the preliminary forecast, use the forecast results as input variables, and combine them with the variables of the econometric model to make further forecasts; Step S4, data storage: the analyzed and predicted data are stored in the cloud via a cloud server.
2. The method for analyzing resident consumption economic data according to claim 1, characterized in that: Step S1 includes the following steps: S11, set keywords and determine target websites, including provincial and municipal statistical bureau websites, professional data platforms and financial websites, and academic research databases; S12, select Python's Scrapy framework and Java's WebMagic to crawl, parse, extract and store Internet web page data to complete the collection of economic data.
3. The method for analyzing resident consumption economic data according to claim 2, characterized in that: Step S2 includes the following steps: S21, data preprocessing: the collected economic data are classified according to consumption areas and income groups; consumption areas include food, clothing, housing, daily necessities and services, transportation and communication, education, entertainment and culture, and medical care; income groups include urban and rural residents’ income and income of groups of different ages; S22, Identification and processing of missing values: draw scatter plots, observe the distribution of economic data, identify missing values, and construct difference functions through cubic spline interpolation to process missing values; S23, Identification and processing of outliers: draw a box plot to show the distribution of economic data. If the outliers are less than 5% of the total data, delete the outliers. If the outliers are greater than or equal to 5% of the total data, determine the median of the non-outliers in the economic data and replace the outliers with the median. S24, identification and processing of duplicate data: Identify duplicate data through Excel's conditional formatting or pivot table and delete them.
4. The method for analyzing resident consumption economic data according to claim 3 is characterized by: Step S3 includes the following steps: S31, Analyze and forecast economic data through time series models; S32, analyze and predict economic data through econometric models.
5. The method for analyzing resident consumption economic data according to claim 4 is characterized by: S31 includes the following steps: S311, divide 80% of the economic data into training sets and the remaining data into test sets; S312, according to the research questions, determine the key variables related to the income and consumption areas of residents of different ages, and transform the variables into a form that meets the requirements of the time series model through logarithmic transformation; S313, select the autoregressive integrated moving average model ARIMA, convert the data in the training set into a stationary sequence by the difference method, and then perform modeling; S314, using the training set data to estimate the selected autoregressive integrated moving average model ARIMA, determine the autoregressive order p, the moving average order q and the number of differences d in the ARIMA model, and evaluate the performance of the model by analyzing the autocorrelation, normality and stability of the residuals; The formula for the autoregressive integrated moving average model (ARIMA(p,d,q)) is as follows: Φ(L)(1-L) d Y t =θ(L)λ t +c; Among them, Y t is the value of the time series at time; t is the white noise error term; c is the constant term; L is the lag operator, LY t =Y t-1 ;Φ(L)=1-Lφ1-L 2 φ2-…L n φ n , φ1, φ2...φ n is the autoregressive coefficient; (1-L) d Indicates a secondary differential operation. When d = 1, (1-L)Y t =Y t -Y t-1 ;θ(L)=1+Lθ1+L 2 θ2+…+L m θ m ,θ1,θ2...θ m is the moving average coefficient; S315, use the estimated model to predict the relationship between the income of residents of different ages and different consumption areas in the future, and predict the income and consumption of residents of different ages every month in the next year.
6. The method for analyzing resident consumption economic data according to claim 5 is characterized by: S32 includes the following steps: S321, taking the prediction result output in S31 as an input variable, and determining another new input variable as the urban and rural residents' income data; S322, select the structural equation model, take the two input variables as observed variables, and the latent variable is the consumption capacity of urban and rural residents of different ages; S323, based on economic theory and existing research, proposes hypotheses about the relationship between latent variables and observed variables, which form the basis of the structural equation model; S324, Use graphical representation to draw the path diagram of the structural equation model to show the causal relationship and influence direction between latent variables and observed variables. S325, use the maximum likelihood estimation method to estimate the parameters of the structural equation model, and verify the estimated structural equation model; S326, substitute the known observed variable values into the model, calculate the estimated value of the latent variable through the model, and predict the consumption data of urban and rural residents of different ages every month in the next year based on the relationship between the latent variable and the economic data.
7. The method for analyzing resident consumption economic data according to claim 6 is characterized by: The structural equation model includes measurement equations and structural equations in the following form:
1. Measurement equation: For observed variables: x1=L x1 ξ+a; x2=L x2 n+b; Among them, x1 is the vector of exogenous observation variables; x2 is the vector of endogenous observation variables; Λ x1 is the relationship matrix between exogenous observed variables and exogenous latent variables; x2 is the relationship matrix between endogenous observed variables and endogenous latent variables; ξ is the vector of exogenous latent variables; η is the vector of endogenous latent variables; α is the vector of measurement errors of exogenous observed variables; β is the vector of measurement errors of endogenous observed variables. Structural equation: η=Qη+Pξ+ζ; Among them, Q is the relationship matrix between endogenous latent variables, which represents the influence between endogenous latent variables; P is the influence matrix of exogenous latent variables on endogenous latent variables; ζ is the error term of the structural equation.
8. The method for analyzing resident consumption economic data according to claim 7, characterized in that: In step S4, the prediction data obtained in step S3 is uploaded to the cloud server for storage, and the data is encrypted, and identity authentication and authorization are required for use.
9. The system of the method for analyzing residents' consumption economic data according to claim 8 is characterized by: include Automatic data collection module: collect economic data of each province by searching by keywords through web crawlers; Data processing and cleaning module: pre-process and classify the collected economic data, identify missing values, outliers and duplicate data in the data and process them; Data analysis and forecasting module: forecast economic data by combining time series models with variables of econometric models; Data storage and viewing module: The analyzed and predicted data are stored in the cloud through the cloud server, and the data can be viewed and retrieved by authorized users logging in; System security and management module: performs user authentication and authorization, encrypts and backs up data, monitors the system, records the system's operation logs and security logs, and promptly detects and prevents network attacks to maintain system security.