Digital economic information management system based on big data

Abstract

The invention discloses a digital economic information management system based on big data, and relates to the technical field of digital economic information management. A data collection module, a data storage monitoring module, a data synchronization quality evaluation module, a high-quality data processing module and a synchronization optimization module are constructed; the problems of data synchronization delay and inconsistency caused by system delay or faults in digital economic information management based on big data are systematically solved, and the real-time consistency and reliability of data among nodes are ensured; for high-quality synchronous data, a real-time report and a visual chart are generated, and valuable economic trend and dynamic analysis are provided; and for low-quality synchronized data, the quality and efficiency of data synchronization are improved through an automatic repair and dynamic optimization strategy, so that a decision maker is supported to make a scientific and timely decision, the accuracy of service operation and the user experience are improved, and economic loss and trust crisis caused by data inconsistency are prevented.

Description

Technical Field

[0001] This invention relates to the field of digital economy information management technology, and specifically to a digital economy information management system based on big data. Background Technology

[0002] Digital economy information management refers to the process of collecting, storing, analyzing, and utilizing information related to the digital economy through modern information technology and big data technology. This management process includes collecting massive amounts of data from various data sources (such as the Internet, the Internet of Things, and e-commerce platforms), preserving the data using advanced data storage technologies, and analyzing it through data processing algorithms to obtain valuable business intelligence and decision support. The goal of digital economy information management is to improve information utilization efficiency, optimize resource allocation, and enhance market competitiveness. In this process, information management systems can monitor and analyze market dynamics, consumer behavior, and industry trends in real time, thereby helping enterprises and governments to better formulate strategic plans and business decisions. Through this systematic management, the various components of the digital economy can operate more coordinatedly and efficiently, achieving the goals of economic growth and social development.

[0003] The digital economy involves massive amounts of data, typically measured in terabytes (TB), petabytes (PB), or even higher. E-commerce platforms generate millions of transaction records and hundreds of millions of user behavior data points daily; social media generates massive amounts of user posts, comments, likes, and other interactive data every day. Such vast amounts of data require efficient storage and management systems for real-time processing and analysis.

[0004] The existing technology has the following shortcomings:

[0005] In big data-driven digital economy information management, delays in data synchronization due to system latency or malfunctions can lead to inconsistencies in data status between different nodes, resulting in data processing errors. Furthermore, data inconsistency can cause deviations in analytical results and decision-making processes, impacting business operations and user experience. For example, a financial trading platform might experience transaction errors due to outdated account balances, leading to financial losses and a crisis of customer trust. Summary of the Invention

[0006] The purpose of this invention is to provide a digital economy information management system based on big data to address the shortcomings of the prior art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a digital economy information management system based on big data, comprising a data collection module, a data storage monitoring module, a data synchronization quality assessment module, a high-quality data processing module, and a synchronization optimization module;

[0008] Data collection module: The data collector is determined based on the data source and collection frequency. It is used to collect digital economy data in real time and transmit the collected economic data from the data source to several distributed storage nodes.

[0009] Data storage monitoring module: Writes data from the buffer to the target storage system through each distributed storage node, assigns a unique version number to each data record, evaluates the consistency of data versions by comparing the data hash values ​​of each node, monitors the update status of data during data updates, judges the latency fluctuation of data transmission between nodes, and evaluates the real-time performance of data updates.

[0010] Data synchronization quality assessment module: Based on the consistency of data versions and the real-time nature of data updates, the data synchronization quality is assessed, and based on the assessment results, the data synchronization quality is divided into high-quality synchronization and low-quality synchronization.

[0011] High-quality data processing module: Further processes high-quality synchronized data to generate real-time reports and visualization charts, and visualizes data trends and economic dynamics;

[0012] Synchronization optimization module: For low-quality synchronized data, an automatic repair mechanism is triggered to resynchronize the data. The accuracy of the resynchronized data is further analyzed, and the data synchronization strategy is optimized based on the analysis results. The synchronization frequency and synchronization method are dynamically adjusted to improve the quality of data synchronization.

[0013] Preferably, in the data storage monitoring module, consistency analysis is performed on the data hash values ​​of each node to generate a hash value anomaly index. The consistency of data versions is then evaluated based on the hash value anomaly index. The method for obtaining the hash value anomaly index is as follows:

[0014] Obtain the hash value of the same data record on each node, treat each hash value as a string, calculate the frequency of each character, and calculate the entropy of each hash value using the information entropy formula. The specific calculation expression is as follows: Where H is the entropy value, pi is the frequency of the i-th character, and n is the total number of characters; by comparing the entropy of the hash values ​​of each node, the hash value anomaly index is calculated, with entropy values ​​H1, H2, ..., Hm respectively; the formula for calculating the hash value anomaly index ES is: Where ES is the hash anomaly index, and H is the hash value anomaly index. j is the entropy value of the j-th node, and TH is the average entropy value of all nodes.

[0015] Preferably, the obtained hash value anomaly index is compared with a pre-set hash value anomaly index reference threshold. If the hash value anomaly index is greater than or equal to the pre-set hash value anomaly index reference threshold, the data version consistency is low, and a data hash value anomaly signal is generated. If the hash value anomaly index is less than the pre-set hash value anomaly index reference threshold, the data version consistency is high, and a data hash value normal signal is generated.

[0016] Preferably, in the data storage monitoring module, a transmission delay fluctuation index is generated based on the fluctuation of data transmission delay between nodes to assess the real-time performance of data updates. The method for obtaining the transmission delay fluctuation index is as follows:

[0017] Obtain the data transmission delay between nodes within the time period s, establish the corresponding time series, determine the sliding time window k, and calculate the moving average delay time within each time window. The formula for calculating the moving average is: In the formula, MK s Q is the moving average over time s, k is the size of the time window, and Q is the moving average over time s. x This is the delay time of the x-th data point; the transmission delay fluctuation index is calculated using a moving average, and the specific calculation formula is as follows:

[0018] Where MA is the transmission delay fluctuation index, n is the total number of data points, and MK x is the moving average at the x-th time point, and EMK is the average of all moving averages.

[0019] Preferably, the obtained transmission delay fluctuation index is compared with a preset transmission delay fluctuation index reference threshold. If the transmission delay fluctuation index is greater than or equal to the preset transmission delay fluctuation index reference threshold, the real-time performance of data updates is low, and a data transmission time abnormal signal is generated. If the transmission delay fluctuation index is less than the preset transmission delay fluctuation index reference threshold, the real-time performance of data updates is high, and a data transmission time normal signal is generated.

[0020] Preferably, in the data synchronization quality assessment module, the hash value anomaly index and the transmission delay fluctuation index are normalized, and the data synchronization quality assessment coefficient is calculated using the normalized hash value anomaly index and the transmission delay fluctuation index.

[0021] Preferably, the obtained data synchronization quality evaluation coefficient is compared with a pre-set data synchronization quality evaluation coefficient reference threshold. If the data synchronization quality evaluation coefficient is greater than or equal to the pre-set data synchronization quality evaluation coefficient reference threshold, it is classified as high-quality synchronization and a high-quality synchronization signal is generated. If the data synchronization quality evaluation coefficient is less than the pre-set data synchronization quality evaluation coefficient reference threshold, it is classified as low-quality synchronization and a low-quality synchronization signal is generated.

[0022] Preferably, in the synchronization optimization module, the consistency of data on each node is ensured by performing a hash value consistency check on the resynchronized data, and the hash value of the data on each node is calculated. The specific calculation expression is: Hz = hash(Dz); where Hz is the hash value of the z-th node and Dz is the data of the z-th node.

[0023] Further analysis of the accuracy of the resynchronized data is conducted, and the accuracy assessment formula is as follows: Where A is the data accuracy evaluation index, and N is the total number of nodes.

[0024] The result of comparing the hash value of the z-th node with that of the reference node is 1 if they are equal, and 0 otherwise. Calculate the accuracy index of the data for all nodes. If the data is consistent across all nodes, the accuracy index is 1.

[0025] Preferably, based on the results of data accuracy analysis, the frequency and method of data synchronization are dynamically adjusted. The new synchronization frequency Fnew is calculated using the formula Fnew=Fbase×(1+α(Atarget-Acurrent)). Wherein, the base synchronization frequency Fbase is a preset value, the adjustment coefficient α determines the sensitivity of the frequency adjustment, Atarget is the target accuracy index, and Acurrent is the current accuracy index. When the current accuracy is less than the target value, the synchronization frequency will increase, and vice versa.

[0026] Synchronization optimization is achieved through formulas The system defines the synchronization method as follows: Mnew is the new synchronization method, Mfull is the full synchronization method, Mincremental is the incremental synchronization method, and Athreshold is the accuracy threshold. When the current accuracy is less than the accuracy threshold, the system uses the full synchronization method to ensure that all data is updated. When the accuracy is greater than or equal to the accuracy threshold, the system uses the incremental synchronization method to improve synchronization efficiency.

[0027] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0028] 1. This invention significantly improves the reliability and real-time performance of a big data-based digital economy information management system through a systematic modular design. The data collection module ensures the real-time acquisition and transmission of high-quality data; the data storage monitoring module guarantees data consistency across nodes by assigning unique version numbers and performing hash value consistency checks, and assesses and monitors the real-time performance of data updates; the data synchronization quality assessment module further classifies the quality of data synchronization and, through automated repair mechanisms and dynamic optimization strategies, promptly repairs and optimizes low-quality synchronized data, ensuring efficiency and accuracy during the data synchronization process.

[0029] 2. This invention generates real-time reports and visual charts through a high-quality data processing module, effectively showcasing data trends and economic dynamics. This supports decision-makers in making scientific and timely decisions, improving the accuracy of business operations and user experience. The synchronization optimization module calculates the data synchronization quality assessment coefficient using the normalized hash value anomaly index and transmission delay fluctuation index, and dynamically adjusts the synchronization frequency and method, further improving the efficiency and quality of data synchronization. This prevents economic losses and trust crises caused by data inconsistency, achieving a comprehensive improvement in system performance. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0031] Figure 1 This is a system module diagram of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1

[0034] Please see Figure 1 As shown in this embodiment, a digital economy information management system based on big data includes a data collection module, a data storage monitoring module, a data synchronization quality assessment module, a high-quality data processing module, and a synchronization optimization module.

[0035] Data collection module: The data collector is determined based on the data source and collection frequency. It is used to collect digital economy data in real time and transmit the collected economic data from the data source to several distributed storage nodes.

[0036] Data storage monitoring module: Writes data from the buffer to the target storage system through each distributed storage node, assigns a unique version number to each data record, evaluates the consistency of data versions by comparing the data hash values ​​of each node, monitors the update status of data during data updates, judges the latency fluctuation of data transmission between nodes, and evaluates the real-time performance of data updates.

[0037] Data synchronization quality assessment module: Based on the consistency of data versions and the real-time nature of data updates, the data synchronization quality is assessed, and based on the assessment results, the data synchronization quality is divided into high-quality synchronization and low-quality synchronization.

[0038] High-quality data processing module: Further processes high-quality synchronized data to generate real-time reports and visualization charts, and visualizes data trends and economic dynamics;

[0039] Synchronization optimization module: For low-quality synchronized data, an automatic repair mechanism is triggered to resynchronize the data. The accuracy of the resynchronized data is further analyzed, and the data synchronization strategy is optimized based on the analysis results. The synchronization frequency and synchronization method are dynamically adjusted to improve the quality of data synchronization.

[0040] In the data collection module, a data collector is determined based on the data source and collection frequency. This collector is used to collect digital economic data in real time and transmit the collected economic data from the data source to several distributed storage nodes.

[0041] The data collector is determined based on the data source and collection frequency to collect digital economy data in real time. The collected economic data is then transmitted from the data source to several distributed storage nodes. This means that in the digital economy information management system, by configuring and using appropriate data collectors, economic-related data can be acquired in real time according to different data sources and set collection frequencies, and this data can be transmitted to various nodes of the distributed storage system efficiently and securely.

[0042] Data sources refer to the sources of digital economy data, including but not limited to e-commerce platforms, social media, financial transaction systems, IoT devices, and government databases.

[0043] Based on the system's requirements, identify all possible data sources. These sources can be internal systems (such as an enterprise's internal transaction database) or external systems (such as data interfaces provided by third parties). Examples include: e-commerce platforms: sales data, user behavior data, inventory data, etc.; social media: user comments, likes, shares, and other interaction data; IoT devices: sensor data, device status data, etc.; and government databases: publicly available economic statistics, policy documents, etc.

[0044] Collection frequency refers to the time interval at which a data collector acquires data, which can be in real time, per minute, per hour, per day, etc.

[0045] Set the collection frequency based on the real-time requirements of the data: different data sources may require different collection frequencies. For example, financial transaction data may need to be collected in real time, while market research data may be collected daily. Examples of real-time collection include: financial transaction data and IoT device data. Scheduled collection includes: daily sales data and user behavior data.

[0046] A data collector is a software or hardware tool used to extract data from a data source and perform preliminary processing, such as cleaning and format conversion. Based on the determined data source and collection frequency, the data collector's parameters are configured, including the data source interface, collection frequency, and data format. For example, an API data collector is configured for an e-commerce platform, set to collect sales data hourly. An IoT gateway is configured to collect sensor data in real time and transmit it to a central system.

[0047] The process of transmitting collected data from the data source to a distributed storage system. Ensuring data transmission integrity and security: During transmission, reliable transmission protocols (such as HTTPS, FTP, message queues, etc.) are used to ensure data integrity and security. HTTPS is used to transmit real-time financial transaction data from the trading platform to distributed storage nodes. Message queue systems (such as Kafka) are used to transmit status data from IoT devices to a central storage system.

[0048] A distributed storage system is a system composed of multiple storage nodes that work together to store and manage large-scale data, providing high availability and fault tolerance. Collected data is transferred to multiple storage nodes to ensure high availability and reliability. For example, by setting up storage nodes in three different data centers, collected user behavior data can be distributed and stored across these nodes to guarantee data redundancy and disaster recovery capabilities.

[0049] In this invention, by determining the data collector based on the data source and collection frequency, digital economy data can be collected and transmitted to distributed storage nodes in real time. The specific process includes identifying the data source, determining the collection frequency, configuring the data collector, ensuring the integrity and security of data transmission, and efficiently storing the data in the distributed storage system. In this way, the system can efficiently and reliably manage and process massive amounts of digital economy data, providing a solid data foundation for subsequent analysis and decision-making.

[0050] Data storage monitoring module: Writes data from the buffer to the target storage system through each distributed storage node, assigns a unique version number to each data record, evaluates the consistency of data versions by comparing the data hash values ​​of each node, monitors the update status of data during data updates, judges the latency fluctuation of data transmission between nodes, and evaluates the real-time performance of data updates.

[0051] In a distributed storage system, after initial data collection and temporary storage in a buffer, data needs to be written to the final target storage system. To ensure data consistency and track updates, each data record is assigned a unique version number. Using this version number, the system can analyze and ensure data consistency across different nodes, thereby detecting and resolving potential data conflicts or inconsistencies.

[0052] A buffer is a region for temporary data storage, typically used to smooth out bursts of data during data transfer and ensure no data loss. The target storage system refers to the final location where data is stored, usually the individual nodes in a distributed storage system.

[0053] Data is collected from various sources (such as sensors, user input, system logs, etc.) and temporarily stored in a buffer. Data is then transferred from the buffer to various nodes in the distributed storage system. Specifically, the data collector temporarily stores data in the buffer. According to the set transmission strategy, data is periodically or in real-time transferred from the buffer to each storage node. Reliable transmission protocols (such as HTTP, FTP, message queues) are used to ensure the integrity and accuracy of data transmission.

[0054] A unique identifier is assigned to each data record to track its creation and updates. A unique version number is generated for each data record when it is written to the target storage system. A new version number is generated each time the data is updated. The version number can be based on a timestamp, sequence number, or other unique identifier. The version number is appended to the data record as its metadata. The data record with the version number is written to the target storage system. Data version consistency is analyzed by comparing the version numbers of the same data record on different storage nodes to check data consistency and ensure that the data is synchronized across all nodes.

[0055] Consistency analysis is performed on the data hash values ​​of each node to generate a hash value anomaly index. The consistency of data versions is then evaluated based on the hash value anomaly index. The method for obtaining the hash value anomaly index is as follows:

[0056] Obtain the hash value of the same data record on each node, treat each hash value as a string, calculate the frequency of each character, and calculate the entropy of each hash value using the information entropy formula. The specific calculation expression is as follows: Where H is the entropy value, pi is the frequency of the i-th character, and n is the total number of characters (usually 64 characters for SHA-256 hashes); the hash anomaly index is calculated by comparing the entropy of each node's hash value, with entropy values ​​H1, H2, ..., Hm; the formula for calculating the hash anomaly index ES is:

[0057] Where ES is the hash anomaly index, and H is the hash value anomaly index. j is the entropy value of the j-th node, and TH is the average entropy value of all nodes.

[0058] The obtained hash value anomaly index is compared with a pre-set hash value anomaly index reference threshold. If the hash value anomaly index is greater than or equal to the pre-set hash value anomaly index reference threshold, it indicates that the data version has low consistency, and a data hash value anomaly signal is generated. If the hash value anomaly index is less than the pre-set hash value anomaly index reference threshold, it indicates that the data version has high consistency, and a data hash value normal signal is generated.

[0059] A higher hash anomaly index indicates poorer data version consistency. Specifically, the hash anomaly index measures the consistency of data content across nodes, reflecting the degree of inconsistency by calculating the entropy difference in hash values ​​across nodes. A large hash anomaly index indicates significant differences in data content across different nodes, potentially caused by data synchronization failures, data corruption, or failure to update in a timely manner.

[0060] First, a higher hash anomaly index means a greater difference in entropy values ​​for the same data record across different nodes. This difference indicates inconsistency in data content across different nodes, reflecting potential changes or errors during data transmission or storage. Since entropy is a measure of the randomness and uncertainty of information, a large difference in entropy values ​​usually signifies significant differences in data content, leading to decreased consistency in data versions.

[0061] Secondly, a high hash value anomaly index prompts system administrators to immediately check and address data consistency issues. Data consistency within a system is fundamental to ensuring normal business operations and accurate decision-making. When the hash value anomaly index is excessively high, it can lead to biased data analysis results, business logic errors, and a degraded user experience. Therefore, timely detection and resolution of data consistency issues are crucial for maintaining system stability and reliability.

[0062] Finally, by monitoring the hash value anomaly index, the system can dynamically adjust data synchronization strategies and mechanisms to improve data consistency. For situations with a high anomaly index, the system should strengthen data verification and automated repair mechanisms to ensure data synchronization and consistency across nodes. This can effectively reduce the probability of data consistency problems and improve the overall system performance and reliability.

[0063] During data updates, the update status is monitored, and the real-time tracking and recording of data updates within the system are maintained, including update start time, update completion time, and success or failure. When a data update request is issued, its timestamp is recorded. The progress of data updates at each node is tracked, recording the timestamps of each node receiving the update request and completing the update. After all nodes have completed the data update, the final completion time is recorded.

[0064] A transmission delay fluctuation index is generated based on the latency fluctuations of data transmission between nodes to evaluate the real-time performance of data updates. The method for obtaining the transmission delay fluctuation index is as follows:

[0065] Obtain the data transmission delay between nodes within the time period s, establish the corresponding time series, determine the sliding time window k, and calculate the moving average delay time within each time window. The formula for calculating the moving average is: In the formula, MK s Q is the moving average over time s, k is the size of the time window, and Q is the moving average over time s. x This is the delay time of the x-th data point; the transmission delay fluctuation index is calculated using a moving average, and the specific calculation formula is as follows:

[0066] Where MA is the transmission delay fluctuation index, n is the total number of data points, and MK x is the moving average at the x-th time point, and EMK is the average of all moving averages.

[0067] The obtained transmission delay fluctuation index is compared with a preset transmission delay fluctuation index reference threshold. If the transmission delay fluctuation index is greater than or equal to the preset transmission delay fluctuation index reference threshold, the more severe the fluctuation of the data transmission delay time between nodes, the lower the real-time performance of data updates, and a data transmission time abnormal signal is generated. If the transmission delay fluctuation index is less than the preset transmission delay fluctuation index reference threshold, the more slight the fluctuation of the data transmission delay time between nodes, the higher the real-time performance of data updates, and a data transmission time normal signal is generated.

[0068] A larger transmission delay fluctuation index indicates greater fluctuations in data transmission delay between nodes. This implies significant differences in data transmission delays between different nodes, potentially caused by network congestion, uneven hardware performance, or other system issues. Large fluctuations suggest a lack of stability in the system during data transmission.

[0069] A larger fluctuation in transmission latency also means poorer real-time data updates. Due to the large fluctuations in latency, data updates are completed at inconsistent times on different nodes, causing data on some nodes to lag behind that on others. This situation can lead to data inconsistency issues, affecting the overall system performance and user experience.

[0070] Data synchronization quality assessment module: Based on the consistency of data versions and the real-time nature of data updates, the data synchronization quality is assessed, and based on the assessment results, the data synchronization quality is divided into high-quality synchronization and low-quality synchronization.

[0071] The hash value anomaly index and transmission delay fluctuation index are normalized, and the data synchronization quality assessment coefficient is calculated using the normalized hash value anomaly index and transmission delay fluctuation index.

[0072] For example, the present invention can use the following formula to calculate the data synchronization quality assessment coefficient, the calculation expression being: In the formula, MP is the data synchronization quality assessment coefficient, ES is the hash value anomaly index, MA is the transmission delay fluctuation index, and a1 and a2 are the ratio coefficients of the hash value anomaly index and the transmission delay fluctuation index, and a2 > a1 > 0.

[0073] The obtained data synchronization quality evaluation coefficient is compared with the preset data synchronization quality evaluation coefficient reference threshold. If the data synchronization quality evaluation coefficient is greater than or equal to the preset data synchronization quality evaluation coefficient reference threshold, it indicates that the data synchronization quality is high, and it is classified as high-quality synchronization, and a high-quality synchronization signal is generated. If the data synchronization quality evaluation coefficient is less than the preset data synchronization quality evaluation coefficient reference threshold, it indicates that the data synchronization quality is low, and it is classified as low-quality synchronization, and a low-quality synchronization signal is generated.

[0074] High-quality data processing module: Further processes high-quality synchronized data to generate real-time reports and visualization charts, and visualizes data trends and economic dynamics.

[0075] Further processing of high-quality synchronized data requires initial data cleaning and preprocessing to ensure accuracy and consistency. Cleaning steps include removing duplicate and erroneous data, standardizing data formats, and classifying and summarizing data according to analytical needs. This ensures that the underlying data for subsequent analysis and visualization is reliable and consistent.

[0076] In-depth analysis of data using statistical analysis and machine learning techniques extracts valuable information and insights. This includes calculating basic statistical indicators (such as averages, maximums, and minimums), performing time series analysis and trend forecasting, and applying methods such as regression analysis to identify patterns and relationships within the data. These analytical results provide detailed data support for real-time reporting and visualization charts.

[0077] Utilize data visualization tools to generate real-time reports and charts, visually presenting analysis results. Employ various visualization formats such as dashboards, line charts, bar charts, and scatter plots to display data trends and economic dynamics. Real-time reports and visualizations not only help users quickly understand data changes and the economic situation but also support decision-makers in making data-driven strategic decisions, improving the scientific rigor and effectiveness of business operations.

[0078] Synchronization optimization module: For low-quality synchronized data, an automatic repair mechanism is triggered to resynchronize the data. The accuracy of the resynchronized data is further analyzed, and the data synchronization strategy is optimized based on the analysis results. The synchronization frequency and synchronization method are dynamically adjusted to improve the quality of data synchronization.

[0079] Before repairing the data, back up the current data state and record all repair operations for future auditing and backtracking. Back up the currently low-quality synchronized data to ensure no data loss during the repair process. Record all steps of the repair process, including the trigger time, the specific data being repaired, and the operators (if applicable). Identify nodes with high data consistency and accuracy for resynchronization. Based on data version number, update timestamp, and node health status, determine the most reliable node as the data source. Verify the integrity and accuracy of the data on the selected primary node to ensure it is up-to-date and error-free. Resynchronize the high-quality data from the primary node to the low-quality node to ensure data consistency across all nodes.

[0080] By performing a hash value consistency check on the resynchronized data, the consistency of the data across all nodes is ensured. The hash value of the data on each node is calculated to ensure the consistency of the data content. The specific calculation expression is: Hz = hash(Dz); where Hz is the hash value of the z-th node and Dz is the data of the z-th node.

[0081] Further analysis of the accuracy of the resynchronized data is conducted, and the accuracy assessment formula is as follows: Where A is the data accuracy evaluation index, and N is the total number of nodes.

[0082] The result of comparing the hash value of the z-th node with that of the reference node (usually the primary node) is 1 if they are equal and 0 otherwise. The accuracy index of the data of all nodes is calculated. If the data is consistent across all nodes, the accuracy index is 1.

[0083] Based on the results of data accuracy analysis, the frequency and method of data synchronization are dynamically adjusted. The new synchronization frequency Fnew is calculated using the formula Fnew=Fbase×(1+α(Atarget-Acurrent)). Here, the base synchronization frequency Fbase is a preset value, the adjustment coefficient α determines the sensitivity of the frequency adjustment, Atarget is the target accuracy indicator, and Acurrent is the current accuracy indicator. When the current accuracy is less than the target value, the synchronization frequency increases; conversely, it decreases.

[0084] Synchronization optimization is achieved through formulas The system defines the synchronization method as follows: Mnew is the new synchronization method, Mfull is the full synchronization method, Mincremental is the incremental synchronization method, and Athreshold is the accuracy threshold. When the current accuracy is less than the accuracy threshold, the system uses the full synchronization method to ensure that all data is updated. When the accuracy is greater than or equal to the accuracy threshold, the system uses the incremental synchronization method to improve synchronization efficiency.

[0085] In this embodiment, a data collector is determined based on the data source and collection frequency to collect digital economic data in real time. The collected economic data is then transmitted from the data source to several distributed storage nodes. Each distributed storage node writes data from its buffer to the target storage system, assigning a unique version number to each data record. The consistency of data versions is evaluated by comparing the data hash values ​​of each node. During data updates, the update status is monitored to determine the latency fluctuations in data transmission between nodes and assess the real-time performance of data updates. Based on the consistency of data versions and the real-time performance of data updates, the data synchronization quality is evaluated, and based on the evaluation results, the data synchronization quality is divided into high-quality synchronization and low-quality synchronization. High-quality synchronized data undergoes further processing to generate real-time reports and visualization charts, visually displaying data trends and economic dynamics. For low-quality synchronized data, an automated repair mechanism is triggered to resynchronize the data. The accuracy of the resynchronized data is further analyzed, and based on the analysis results, the data synchronization strategy is optimized, dynamically adjusting the synchronization frequency and method to improve the quality of data synchronization. It can achieve efficient and accurate data synchronization and management, ensuring data consistency and real-time performance across nodes, and improving data reliability and availability. For high-quality synchronized data, it can quickly generate real-time reports and visualization charts, providing valuable economic trends and dynamic analysis. For low-quality synchronized data, it improves the quality and efficiency of data synchronization through automated repair mechanisms and dynamic optimization strategies, thereby supporting decision-makers to make more scientific and timely decisions and enhancing the overall performance and user experience of the system.

[0086] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0087] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0088] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0089] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0092] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A big data-based digital economic information management system, characterized in that: It includes data collection module, data storage monitoring module, data synchronization quality evaluation module, high-quality data processing module and synchronization optimization module. The data collection module determines the data collector according to the data source and the collection frequency, is used for collecting the digital economic data in real time, and transmits the collected economic data from the data source to a plurality of distributed storage nodes. The data storage monitoring module writes data from the buffer to the target storage system through each distributed storage node, assigns a unique version number to each data record, evaluates the consistency of the data version by comparing the data hash values of each node, monitors the update status of the data when the data is updated, judges the delay time fluctuation of the data transmission between nodes, and evaluates the real-time performance of the data update. The data synchronization quality evaluation module evaluates the synchronization quality of the data according to the consistency of the data version and the real-time performance of the data update, and divides the synchronization quality of the data into high-quality synchronization and low-quality synchronization according to the evaluation results. The high-quality data processing module further processes the high-quality synchronization data to generate real-time reports and visual charts, and visually displays the data trends and economic dynamics. The synchronization optimization module triggers an automatic repair mechanism for low-quality synchronization data, resynchronizes the data, and further analyzes the accuracy of the resynchronized data, optimizes the data synchronization strategy according to the analysis results, and dynamically adjusts the synchronization frequency and synchronization mode to improve the quality of data synchronization.

2. The big data-based digital economy information management system according to claim 1, characterized in that: In the data storage monitoring module, the consistency of the data version is evaluated by generating a hash value abnormality index based on the consistency analysis of the data hash values of each node. The hash value of the same data record on each node is obtained, each hash value is regarded as a string, the frequency of each character is calculated, and the entropy of each hash value is calculated using an information entropy formula, and the specific calculation expression is: Wherein, H is the entropy value, pi is the frequency of the i th character, and n is the total number of characters; the hash value anomaly index is calculated by comparing the entropies of the hash values of the nodes, and the entropies are H1, H2,..., Hm respectively; and the calculation formula of the hash value anomaly index ES is: Wherein, ES is the hash value anomaly index, H j is the entropy value of the j th node, and TH is the average value of the entropies of all nodes.

3. The big data-based digital economy information management system according to claim 2, characterized in that: The hash value abnormality index is compared with the pre-set hash value abnormality index reference threshold, and if the hash value abnormality index is greater than or equal to the pre-set hash value abnormality index reference threshold, the consistency of the data version is low, and a data hash value abnormality signal is generated.

4. The big data-based digital economy information management system according to claim 3, characterized in that: If the hash value abnormality index is less than the pre-set hash value abnormality index reference threshold, the consistency of the data version is high, and a data hash value normal signal is generated. The data transmission delay time between each node in the s time period is obtained, and a corresponding time sequence is established. A sliding time window k is determined, and the moving average value of the delay time in each time window is calculated. The calculation formula of the moving average value is: MK s is the moving average value of time s, k is the size of the time window, Q x is the delay time of the xth data point; the transmission delay fluctuation index is calculated using the moving average value. The specific calculation formula is: where MA is the transmission delay fluctuation index, n is the total number of data points, MK x is the moving average at the xth time point, and EMK is the average of all moving averages.

5. The big data-based digital economy information management system according to claim 4, characterized in that: In the data storage monitoring module, a transmission delay fluctuation index is generated based on the delay time fluctuation of the data transmission between nodes to evaluate the real-time performance of the data update. The transmission delay fluctuation index is compared with the pre-set transmission delay fluctuation index reference threshold, and if the transmission delay fluctuation index is greater than or equal to the pre-set transmission delay fluctuation index reference threshold, the real-time performance of the data update is low, and a data transmission time abnormal signal is generated.

6. The big data-based digital economy information management system according to claim 5, characterized in that: If the transmission delay fluctuation index is less than the pre-set transmission delay fluctuation index reference threshold, the real-time performance of the data update is high, and a data transmission time normal signal is generated. In the data synchronization quality evaluation module, the hash value abnormality index and the transmission delay fluctuation index are normalized, and the data synchronization quality evaluation coefficient is calculated based on the normalized hash value abnormality index and the transmission delay fluctuation index.

7. The big data-based digital economy information management system according to claim 6, characterized in that: The obtained data synchronization quality evaluation coefficient is compared with a preset data synchronization quality evaluation coefficient reference threshold value, if the data synchronization quality evaluation coefficient is greater than or equal to the preset data synchronization quality evaluation coefficient reference threshold value, the data is classified as high-quality synchronization, and a high-quality synchronization signal is generated; if the data synchronization quality evaluation coefficient is less than the preset data synchronization quality evaluation coefficient reference threshold value, the data is classified as low-quality synchronization, and a low-quality synchronization signal is generated.

8. The big data-based digital economy information management system according to claim 7, characterized in that: In the synchronization optimization module, the consistency of the data on each node is ensured by performing a hash value consistency check on the re-synchronized data, and the hash value of the data of each node is calculated, and the specific calculation expression is Hz = hash (Dz); wherein Hz is the hash value of the zth node, and Dz is the data of the zth node; Further analysis is made on the accuracy of the re-synchronized data, and the accuracy evaluation formula is: wherein A is the data accuracy evaluation index, and N is the total number of nodes. For the comparison result of the zth node and the reference node hash value, if equal, it is 1, otherwise 0, the accuracy index of all node data is calculated, and if the data is consistent on all nodes, the accuracy index is 1.

9. The big data-based digital economy information management system according to claim 8, characterized in that: According to the result of the data accuracy analysis, the frequency and mode of data synchronization are dynamically adjusted, and the new synchronization frequency Fnew is calculated by the formula Fnew = Fbase x (1 + a (Atarget - Acurrent)); wherein the basic synchronization frequency Fbase is a preset value, the adjustment coefficient a determines the sensitivity of the frequency adjustment, Atarget is the target accuracy index, and Acurrent is the current accuracy index; when the current accuracy is less than the target value, the synchronization frequency is increased, and vice versa. The optimization of the synchronization mode is determined by the formula , where Mnew is the new synchronization mode, Mfull is the full-quantity synchronization mode, Mincremental is the incremental synchronization mode, and Athreshold is the accuracy threshold. When the current accuracy is less than the accuracy threshold, the system adopts the full-quantity synchronization mode to ensure that all data is updated; when the accuracy is greater than or equal to the accuracy threshold, the incremental synchronization mode is adopted to improve the synchronization efficiency.

Images