Digital economy index calculation method and system based on multi-source heterogeneous data

Abstract

The application belongs to the technical field of data processing and statistical analysis, in order to solve the technical problems of rigid index system, low data processing efficiency and lack of dynamic early warning in the prior art, a digital economy index calculation method and system based on multi-source heterogeneous data are provided, through standardized index mapping logic and adaptive weight adjustment algorithm, combined with automatic processing and model calculation system based on digital economy index, the whole process automatic processing of digital economy index from multi-source heterogeneous data collection, standardized splitting, adaptive model calculation to classification verification display is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing and statistical analysis technology, specifically to a method and system for the automated construction, calculation, analysis, and early warning of digital economic indicators. Background Technology

[0002] As the digital economy continues to rise in importance within the national economic system, it involves multiple dimensions such as industrial structure, information infrastructure, digital industrialization, and industrial digitalization. Its data sources are complex and its definitions are diverse, making it difficult for traditional manual statistical and analytical methods to meet management needs.

[0003] In existing technologies, the calculation schemes for the digital economy typically include: manually selecting several indicators based on experience to form a fixed indicator system, and then calculating the results by collecting data; assigning fixed weights to each indicator and calculating a comprehensive index through linear weighting or other methods; and analysis schemes that mainly rely on statistical yearbooks or data from a single business system.

[0004] However, existing technologies have at least the following drawbacks: The indicator system lacks flexibility; once the indicators are set, the adjustment costs are high, making it difficult to adapt to the characteristics of the digital economy in different regions and at different stages of development.

[0005] Static weight settings: Indicator weights are usually fixed values, which cannot reflect the changes in the importance of indicators over different time periods.

[0006] Low data utilization efficiency: It is difficult to process and utilize multi-source heterogeneous data in a unified manner, which affects the accuracy of the measurement results.

[0007] Lack of dynamic analysis and early warning capabilities: Most solutions only provide results display and cannot achieve trend analysis and risk warning.

[0008] Therefore, there is an urgent need for a technical solution that can automatically, dynamically, and accurately quantify, assess, monitor, and provide early warnings of the development level of the digital economy. Summary of the Invention

[0009] To address the technical problems of rigid indicator systems, static weights, low data processing efficiency, and lack of dynamic early warning in existing technologies, this invention provides a method and system for calculating digital economic indicators based on multi-source heterogeneous data. To achieve the above objectives, this invention adopts the following technical solution: On the one hand, the method for measuring digital economy indicators based on multi-source heterogeneous data includes the following steps: S1. Multi-source data acquisition: Collect multi-source heterogeneous raw data related to the digital economy from different platforms through application programming interfaces to obtain raw data set A. After data preprocessing, a standardized dataset B is formed. S2. Indicator Mapping and Construction: Based on the preset indicator mapping rule library, the data items in the standardized dataset B are mapped and calculated as basic indicators of the digital economy, generating a basic indicator set D, and the validity of the indicators in the basic indicator set D is verified.

[0010] S3. Adaptive Calculation of Indicator Weights: Based on the historical change characteristics and stability parameters of the indicators in the basic indicator set D, the weights of each indicator are dynamically adjusted and normalized to generate a weight set E.

[0011] S4. Comprehensive Index Model Calculation: Calculate the comprehensive development index F of the digital economy based on the index values ​​in the basic index set D and the corresponding weights in the weight set E.

[0012] S5. Situation Analysis and Early Warning Level Generation: Perform time series trend analysis on the digital economy comprehensive development index F and / or individual indicators, and compare it with preset thresholds to generate an early warning level or risk warning result G.

[0013] S6. Result Output and Display: Store the comprehensive development index of digital economy F, the early warning level or risk warning result G, and display them in a visual manner.

[0014] On the other hand, a digital economy indicator measurement system based on multi-source heterogeneous data includes at least: The data acquisition module is used to realize the automatic access and periodic acquisition of multi-source heterogeneous data. The acquired raw data set is uniformly written into the raw data storage area and transmitted to the data preprocessing module. The data preprocessing module is used to standardize the raw data to ensure the accuracy and comparability of subsequent indicator calculations. After generating a standardized data set, it is input into the indicator construction module. The indicator construction module can automatically generate digital economy indicators based on a preset indicator rule library, and the valid indicator set after verification is transferred to the weight calculation module. The weight calculation module can dynamically adjust the weights of the indicators, and the final weight set is passed to the model calculation module. The model calculation module can calculate the comprehensive development index of the digital economy based on the input weight set, generate the comprehensive index result after the calculation is completed, and output it to the early warning analysis module. The early warning analysis module is used to generate situation analysis, early warning level or risk warning results, and transmit them to the results display module for visualization.

[0015] Technical effects: This invention addresses two key challenges in existing technologies. First, it addresses the bottlenecks of excessive manual intervention, delayed calculation results, and inconsistent data dimensions by employing standardized indicator mapping logic and adaptive weight adjustment algorithms. Second, it automates the entire process of digital economic indicators—from multi-source heterogeneous data collection, standardized decomposition, adaptive model calculation, to classification, verification, and display—by constructing an automated processing and model calculation system based on digital economic indicators. Supporting multi-source data fusion, this invention accurately maps digital economic indicators and introduces adaptive weight calculation logic and economic attribute classification models. This effectively solves the technical shortcomings of existing technologies, such as redundant raw data, cross-interference of evaluation results, and low thresholds for triggering survey modes, ensuring the accuracy of digital economic measurement results and the intuitiveness of the display. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the overall process of the method in this embodiment.

[0017] Figure 2 This is a system architecture block diagram for this embodiment. Detailed Implementation

[0018] Terminology Explanation: Multi-source heterogeneous data: refers to data from different systems, departments, data structures, and definitions, including statistical data, business data, and data from third-party platforms.

[0019] Digital economy indicators: A set of quantitative indicators used to reflect the development level, structural characteristics and operational status of the digital economy.

[0020] Indicator mapping rules: Rules or logical relationships that transform raw data into computable indicators.

[0021] Comprehensive Development Index: A comprehensive quantitative result formed by weighting multiple indicators.

[0022] Dynamic early warning: The process of automatically generating risk warnings or risk level classifications based on the changing trends of indicators or indices.

[0023] This invention provides an automated processing method and system for measuring and analyzing the operational status of digital economic indicators based on a computer system. By deploying the system program in a distributed computing environment, a layered structure is constructed, consisting of a data acquisition layer, a data processing layer, a model calculation layer, and a result application layer, achieving fully automated operation from acquiring raw digital economic data to outputting comprehensive indices and generating monitoring and early warnings.

[0024] On the one hand, the method for measuring digital economy indicators based on multi-source heterogeneous data includes the following steps: S1. Multi-source data acquisition: Collect multi-source heterogeneous raw data related to the digital economy from different platforms through API data interfaces, and perform deduplication verification to obtain raw data set A; After performing data preprocessing on the raw data set A, including format unification, missing value filling, outlier identification and correction, and standardization processing, a standardized dataset B is formed.

[0025] Furthermore, the sources of raw data include, but are not limited to: structured data tables in statistical databases, transaction data in business systems, and data streams provided by third-party platforms through interfaces. Data formats include numeric fields, text fields, and time-series data.

[0026] The data acquisition process consists of four steps: first, establishing a connection with the external system; second, verifying interface access permissions; third, calling the data interface according to the preset acquisition cycle; and fourth, writing the acquired data to the local database and cache. When an interface call fails, an error log is automatically recorded, and a retry mechanism is initiated. When the number of retries exceeds the request threshold, a data acquisition exception notification is triggered.

[0027] The preset data strategy is as follows: data collection will select all raw digital economic data for a given region within a preset period (e.g., 6 months). Each raw data point will be tagged with a timestamp and numbered in ascending order. Duplicates will be determined using hash verification or semantic consistency comparison. If duplicates are found, deduplication will be performed, retaining only the single data item generated at that time to construct a standard raw dataset.

[0028] Through the above data acquisition sources, acquisition process steps and acquisition strategies, the original data set A will eventually be obtained, and it will be stored in the database according to a unified data structure.

[0029] Furthermore, in the data preprocessing: Data format unification processing: Field mapping is performed on the original dataset A, converting data fields from different sources into uniform field names. When inconsistent field types are detected, automatic type conversion is performed; if conversion fails, an exception is marked.

[0030] Missing value imputation: Check if there are empty values ​​in each field. When the missing value ratio is lower than the preset ratio, imputation rules are used to fill in the missing values. When the missing value ratio exceeds the preset ratio, the field is marked as removed.

[0031] Outlier identification and correction: Range detection is performed on numerical data. If the value exceeds the reasonable range, it is marked as an anomaly and corrected or replaced.

[0032] Data standardization processing: Normalize or proportionally transform data of different dimensions according to preset standardization rules.

[0033] S2. Indicator Mapping and Construction: Based on the standardized data set B, an indicator generation operation is performed. Relevant fields are extracted according to the mapping rules, and indicator items and data combinations are calculated to generate a candidate indicator set C. Verification is performed on the candidate indicator set C, such as continuity checks, data integrity checks, and time series rationality checks. When a certain indicator has missing consecutive periods, it is marked as temporarily disabled. After the above indicator mapping and construction processes, a basic digital economy indicator set D is generated, and the indicators in the basic indicator set D are validated.

[0034] S3. Adaptive Calculation of Indicator Weights: Input the indicator set D and its historical data sequence. For each indicator, calculate its change amplitude and trend parameters within a preset time window, calculate the fluctuation amplitude of the indicator over time, and generate stability parameters. Construct a weight adjustment factor based on the change characteristic parameters and stability parameters. When indicator fluctuations are abnormal, automatically reduce the weight to avoid excessive influence on the composite index. The adjusted weights are proportionalized to ensure that the sum of the weights is a preset value, generating a weight set E.

[0035] S4. Comprehensive Index Model Calculation: Calculate the comprehensive development index F of the digital economy based on the index values ​​in the basic index set D and the corresponding weights in the weight set E.

[0036] In the comprehensive index model's scale calculation, the production approach and the income approach are mainly used. The production approach is used for areas with relatively good data foundations, while the income approach is used for areas with significant data gaps, requiring further data collection. According to the definition of the digital economy, it comprises two main parts: the digital industrialization component and the industrial digitalization component.

[0037] The comprehensive development index of the digital economy, F, is calculated as follows: (X) = (Scale of digital industrialization + (Y) = Scale of industrial digitalization. The scale of digital industrialization X = V Electricity+ V Base Electricity + V Internet+ V Soft clothing + V new, in: V Electrical: Value added of the electronic information manufacturing industry V Basic telecommunications: Value added of the basic telecommunications industry V Internet: The added value of the Internet industry V Software services: the added value of the software service industry. V Emerging: Value added of other digital-related emerging industries; Industrial digitalization scale , in: i Representing all traditional industries except for those related to digital industrialization. For the industry i Total output growth , (Total output in the current period) = Industry i Current final demand + industry i Current intermediate demand, of which final demand is industry i The value of products / services used for consumption, investment, and export; intermediate demand is the industry. i The value of intermediate products / services used in the production of other industries.

[0038] For the industry i The average contribution share of ICT capital For the industry i exist t ICT's share of capital contribution in 2018.

[0039] Following the above calculation rules, the set of indicator values ​​D and the set of weights E are weighted and calculated. The reasonableness of the results is verified to check if the comprehensive index is within a reasonable range. If it exceeds the reasonable range, the verification process is triggered and the anomaly is recorded. Finally, the comprehensive digital economy index F is output.

[0040] S5. Situation Analysis and Early Warning Generation: Time series trend analysis is performed on the comprehensive digital economy development index F and / or individual indicators. Through trend analysis, the trend change characteristics of the comprehensive index F are calculated. Subsequently, a threshold comparison is performed, comparing the current index with the tiered thresholds. When the index falls below a certain threshold: a corresponding level early warning indicator is automatically generated and the generated early warning result G is output.

[0041] S6. Result Output and Display: Output the calculation results, analysis conclusions, and early warning information to the application system. Write the digital economy comprehensive development index F and the early warning result G into the database. Generate trend charts, structure diagrams, and early warning indicator graphics for visualization, and log the data of this calculation process.

[0042] In this embodiment, the above steps are executed by a computer program, and intermediate data results are stored in a database. Data is transferred between processing steps via a standard API data interface or an internal data bus.

[0043] On the other hand, this embodiment provides a digital economic indicator calculation system based on multi-source heterogeneous data. By deploying data processing programs in a distributed computing environment, it constructs a layered structure of data acquisition layer, data processing layer, model calculation layer, and result application layer, realizing fully automated operation from raw data acquisition to comprehensive index output and early warning generation. The technical solution uses a computer program to control the processor to execute each step of the operation and stores intermediate data results in memory. Data is transferred between processing steps through standard data interfaces or internal data buses.

[0044] like Figure 2 As shown, the data acquisition module includes an interface management unit, a data capture unit, and an exception handling unit. The interface management unit is used to establish communication connections with external data sources. Data sources include, but are not limited to, business system database interfaces, third-party open platform API interfaces, statistical reporting systems, and file import channels. The interface management unit supports configuring data source addresses, authentication methods, access permissions, and data acquisition cycles, and can perform real-time detection of interface connectivity. The data capture unit is used to execute data reading tasks according to a preset acquisition cycle. The exception handling unit is used to execute a retry mechanism and generate exception logs when interface or data exceptions occur. The number of retries and the time interval can be parameterized; if the attempt fails after exceeding the preset number of retries, the abnormal data is marked and pushed to the exception queue, and an operation log is recorded for subsequent traceability.

[0045] The data capture unit includes two modes: full data collection and incremental data collection. In this implementation, the system uses timestamps or primary key identifiers to achieve incremental identification, thereby reducing the system load caused by repeated data collection.

[0046] The data preprocessing module includes a field mapping unit, a missing value handling unit, an anomaly detection unit, and a standardization unit. The field mapping unit establishes mapping relationships between data fields from different sources and system standard fields. By configuring a mapping rule base, it achieves unified conversion of different data structures, thereby forming a dataset with a standard field structure. The missing value handling unit identifies null values ​​or missing fields in the data and processes them according to preset rules. Processing methods include mean imputation, historical year-on-year imputation, interpolation, or marking missing values. Preferably, this embodiment supports multiple configurable strategies. The anomaly detection unit identifies abnormal values ​​using methods such as standard deviation, box plots, or preset thresholds. Abnormal data is marked rather than directly deleted to preserve the integrity of the original data. The standardization unit performs unified processing on data of different dimensions. Standardization methods include range standardization or Z-score standardization. For inverse indicators, the system automatically performs direction conversion to ensure consistent indicator direction. After data preprocessing, a standardized dataset is formed and output to the indicator construction module.

[0047] The indicator construction module includes a rule invocation unit, an indicator generation unit, and an indicator verification unit. The rule invocation unit reads the indicator mapping rule library, which includes the indicator name, calculation formula, data source fields, calculation period, and indicator attributes (forward or inverse). The indicator generation unit performs automated calculations according to the rules, supporting basic arithmetic operations, year-on-year calculations, month-on-month calculations, and multi-indicator combination calculations. Preferably, the indicator generation process supports batch execution and periodic scheduling. The indicator verification unit performs continuity and integrity checks on the generated indicators, including time series continuity checks, regional coverage integrity checks, and abnormal fluctuation alerts. The valid set of indicators that passes verification is then passed to the weight calculation module.

[0048] The weight calculation module is used to dynamically adjust the indicator weights, including a change characteristic analysis unit, a stability analysis unit, a weight correction unit, and a weight normalization unit. The change characteristic analysis unit calculates the change parameters of the indicator over time, including the growth rate, trend slope, and fluctuation amplitude. The stability analysis unit calculates the standard deviation and coefficient of variation of the indicator to measure its stability. The weight correction unit generates a weight adjustment factor based on the change characteristic parameters and stability parameters. Preferably, this embodiment supports using expert weighting results as the base weights and combining them with the dynamic correction factor for adjustment. The weight normalization unit proportionalizes the corrected weights so that the sum of all indicator weights is 1.

[0049] In the model calculation module, the index calculation unit performs linear weighted calculations based on standardized indicator data and corresponding weights to generate a comprehensive index and multi-dimensional index results. Preferably, a multi-level summary calculation structure is supported. The result verification unit is used to check the reasonableness of the calculation results, including detecting abnormal year-on-year fluctuations and conducting historical trend comparison analysis.

[0050] The early warning analysis module first calculates the index change trend parameters and identifies accelerating, slowing, or declining trends. It then compares the current index result with preset grading thresholds and generates an early warning level based on the comparison results. Preferably, the early warning levels are divided into four levels: normal, attention, warning, and severe.

[0051] The results display module is used to connect to a third-party visualization system. First, it writes the comprehensive index, weight results and warning results output by the early warning analysis module into the database, supports historical version archiving, and generates visualization results such as trend charts, structure charts and regional distribution maps. It provides data service interfaces to the front-end display system or third-party system through the interface end and supports access control.

[0052] The modules mentioned above interact with each other through different interfaces or communication protocols. Data transmission between them can be performed sequentially or in parallel in different computing threads.

[0053] All content not described in detail in this specification belongs to the prior art known to those skilled in the art. Furthermore, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for calculating digital economic indicators based on multi-source heterogeneous data, characterized in that, Includes the following steps: S1. Multi-source data acquisition: Collect multi-source heterogeneous raw data related to the digital economy from different platforms through application programming interfaces to obtain raw data set A. After data preprocessing, a standardized dataset B is formed. S2. Indicator Mapping and Construction: Based on the preset indicator mapping rule library, the data items in the standardized dataset B are mapped and calculated as basic indicators of the digital economy, generating a basic indicator set D, and the validity of the indicators in the basic indicator set D is verified. S3. Adaptive Calculation of Indicator Weights: Based on the historical change characteristics and stability parameters of the indicators in the basic indicator set D, the weights of each indicator are dynamically adjusted and normalized to generate a weight set E. S4. Comprehensive Index Model Calculation: Calculate the comprehensive development index F of the digital economy based on the index values ​​in the basic index set D and the corresponding weights in the weight set E. S5. Situation Analysis and Early Warning Level Generation: Perform time series trend analysis on the digital economy comprehensive development index F and / or individual indicators, and compare it with preset thresholds to generate an early warning level or risk warning result G. S6. Result Output and Display: Store the comprehensive development index of digital economy F, the early warning level or risk warning result G, and display them in a visual manner.

2. The method for calculating digital economic indicators based on multi-source heterogeneous data according to claim 1, characterized in that, The data preprocessing in S1 specifically includes format standardization, missing value completion, outlier identification and correction, and standardization.

3. The method for calculating digital economic indicators based on multi-source heterogeneous data according to claim 1, characterized in that, S3 specifically involves calculating the change amplitude and trend parameters of each indicator in the basic indicator set D within a preset time window, calculating the fluctuation amplitude of the indicator in the time series, and generating a stability parameter; constructing a weight adjustment factor based on the change characteristic parameter and the stability parameter, and automatically reducing the weight when the indicator fluctuates abnormally. The adjusted weights are proportionalized to ensure that the sum of the weights is the preset value, and a weight set E is generated.

4. The method for calculating digital economic indicators based on multi-source heterogeneous data according to claim 1, characterized in that, In S4, the comprehensive development index of the digital economy F = the scale of the digital industrialization component X + the scale of the industrial digitalization component Y. The scale of digital industrialization X = V Electricity+ V Base Electricity + V Internet+ V Soft clothing + V new, in: V Electrical: Value added of the electronic information manufacturing industry V Basic telecommunications: Value added of the basic telecommunications industry V Internet: The added value of the Internet industry V Software services: the added value of the software service industry. V Emerging: Value added of other digital-related emerging industries; Industrial digitalization scale , in: i Representing all traditional industries except for those related to digital industrialization. For the industry i Total output growth , (Total output in the current period) = Industry i Current final demand + industry i Current intermediate demand, of which final demand is industry i The value of products / services used for consumption, investment, and export; intermediate demand is the industry. i The value of intermediate products / services used in production in other industries; For the industry i The average contribution share of ICT capital For the industry i exist t ICT's share of capital contribution in 2018.

5. A digital economic indicator calculation system based on multi-source heterogeneous data, characterized in that, It includes at least a data acquisition module, which is used to realize the automatic access and periodic acquisition of multi-source heterogeneous data. The acquired raw data set is uniformly written into the raw data storage area and transmitted to the data preprocessing module. The data preprocessing module is used to standardize the raw data to ensure the accuracy and comparability of subsequent indicator calculations. After generating a standardized data set, it is input into the indicator construction module. The indicator construction module can automatically generate digital economy indicators based on a preset indicator rule library, and the valid indicator set after verification is transferred to the weight calculation module. The weight calculation module can dynamically adjust the weights of indicators, and the final weight set is passed to the model calculation module. The model calculation module can calculate the comprehensive development index of the digital economy based on the input weight set, generate the comprehensive index result after the calculation is completed, and output it to the early warning analysis module. The early warning analysis module is used to generate situation analysis, early warning level or risk warning results, and transmit them to the results display module for visualization.

6. The digital economic indicator calculation system based on multi-source heterogeneous data according to claim 5, characterized in that, The indicator construction module includes a rule calling unit, an indicator generation unit, and an indicator verification unit: the rule calling unit is used to read the indicator mapping rule library, the indicator generation unit performs automated calculations according to the rules, and the calculation methods support basic arithmetic operations, year-on-year calculations, month-on-month calculations, and multi-indicator combination calculations. The indicator verification unit is used to perform continuity and integrity checks on the generated indicators. The checks include time series continuity checks, regional coverage integrity checks, and abnormal fluctuation alerts.

7. The digital economic indicator calculation system based on multi-source heterogeneous data according to claim 5, characterized in that, The weight calculation module includes a change feature analysis unit, a stability analysis unit, a weight correction unit, and a weight normalization unit: the change feature analysis unit is used to calculate the change parameters of the indicator in the time dimension, including the growth rate, trend slope, and fluctuation amplitude; the stability analysis unit is used to calculate the standard deviation and coefficient of variation of the indicator to measure the stability of the indicator. The weight correction unit generates a weight adjustment factor based on the changing characteristic parameters and stability parameters; The weight normalization unit is used to proportionalize the corrected weights so that the sum of all index weights is 1.

Images