Digital economic data acquisition system and method based on 5G communication

Abstract

The invention discloses a digital economic data acquisition system and method based on 5G communication, and relates to the technical field of electronic digital data processing. The digital economic data acquisition system and method based on 5G communication comprises the following steps: S1, collecting digital economic related data; s2, preprocessing the digital economic related data stored in the database by using an automatic detection tool; s3, performing data standardization processing on the to-be-detected digital economic related data according to the data standardization detection value; S4, performing data fusion on the to-be-fused economic related data by using an Apache Kafka technology, and storing the to-be-fused digital economic related data after data stream integration by using a data lake technology of Apache Hadoop and Amazon S3; and S5, making corresponding optimization processing measures according to the data fusion detection value. The problem that in a traditional digital economic data acquisition system based on 5G communication, data is difficult to fuse due to diversity of data sources and heterogeneity of data formats and types is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and specifically to a digital economic data collection system and method based on 5G communication. Background Art

[0002] In today's rapidly developing digital economy, enterprises and organizations are increasingly relying on real-time, accurate data to drive decision-making and enhance business operations. This reliance has prompted the demand for efficient data processing technology, especially in data collection. With the emergence and popularization of 5G communication technology, its high-speed transmission, low-latency communication and extensive device connectivity provide unprecedented opportunities for processing large-scale data.

[0003] Although 5G technology has brought significant improvements in network performance, it is still a challenge to effectively collect, process and fuse data from diversified digital economy data sources to meet the requirements of real-time and high quality. Existing data collection and processing methods often show problems such as slow processing speed, lack of real-time performance and low data fusion efficiency when facing large amounts of data. These problems mainly stem from the failure of existing technologies to fully utilize the high-speed transmission capabilities of 5G, as well as the lack of effective automated tools and algorithms to ensure data quality and consistency during data preprocessing and fusion.

[0004] In addition, the data standardization and integration process lacks an effective monitoring and evaluation mechanism, making it difficult to provide real-time feedback and optimization of the data processing process. This not only affects the efficiency of data use, but also limits the quality of decision-making based on this data.

[0005] Therefore, in response to the above problems, there is an urgent need for a digital economy data collection system and method based on 5G communication. Summary of the invention

[0006] Technical issues solved

[0007] In view of the shortcomings of the prior art, the present invention provides a digital economy data collection system and method based on 5G communication, which solves the problem that data is difficult to integrate in traditional digital economy data collection systems based on 5G communication due to the diversity of data sources and the heterogeneity of data formats and types.

[0008] Technical Solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: a digital economy data collection system and method based on 5G communication, comprising the following steps: S1, collecting digital economy-related data, preliminarily formatting and cleaning the digital economy-related data and storing them in a database; S2, using an automated detection tool to preprocess the digital economy-related data stored in the database, marking the preprocessed digital economy-related data as digital economy-related data to be detected, and obtaining the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency; S3, obtaining a data standardization detection value by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency, performing data standardization processing on the digital economy-related data to be detected according to the data standardization detection value, and marking the digital economy-related data to be detected after data standardization processing as digital economy-related data to be integrated; S4, using Apache Kafka technology to perform data fusion on the economy-related data to be integrated, and using Apache Hadoop and Amazon S3's data lake technology stores the digital economy-related data to be integrated after the integration of data streams, and obtains the integrity, consistency, accuracy and richness of the fused data during the data fusion process; S5 obtains the data fusion detection value through comprehensive calculation of the integrity, consistency, accuracy and richness of the fused data, and makes corresponding optimization processing measures based on the data fusion detection value.

[0010] Furthermore, the specific process of collecting digital economy related data, preliminarily formatting and cleaning the digital economy related data and then storing them in a database is: formulating a standardized data interface protocol for each data source, collecting the digital economy related data in each data source according to the formulated data interface protocol, preliminarily formatting and cleaning the digital economy related data according to a preset data structure and format, and then storing them in the database.

[0011] Furthermore, the use of automated detection tools to pre-process the digital economy-related data stored in the database, marking the pre-processed digital economy-related data as digital economy-related data to be detected, and obtaining the number of invalid entries, the number of erroneous entries, the number of duplicate entries, data integrity and data consistency. The specific process is: using Apache NiFi to automatically manage and clean digital economy-related data, using Talend Data Quality to automatically identify and correct errors and inconsistencies in digital economy-related data, and recording the data integrity and data consistency of digital economy-related data; using regular expressions to verify and format string type data in digital economy-related data, and automatically correcting data entries that do not conform to the format in the digital economy-related data, recording the number of invalid entries, the number of erroneous entries, the number of duplicate entries and the total number of data entries in the digital economy-related data.

[0012] Furthermore, the specific process of obtaining the data standardization test value by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency is: obtaining the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency, and obtaining the data standardization test value by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency.

[0013] Furthermore, the specific calculation formula of the data standardization detection value is:

[0014]

[0015] In the formula, DSV represents data standardization detection value, C represents data integrity, ω c The weight factor that expresses data integrity, S represents data consistency, ω s represents the weight factor of data consistency, I represents the number of invalid entries, ω i represents the weight factor of the number of invalid entries, E represents the number of erroneous entries, ω e represents the weight factor for the number of erroneous entries, D represents the number of duplicate entries, ω d represents the weight factor of the number of duplicate entries, and T represents the total number of data entries.

[0016] Furthermore, the specific process of performing data standardization processing on the digital economy-related data to be detected according to the data standardization detection value is: comparing the data standardization detection value with the first threshold in real time, and when the data standardization detection value is greater than the first threshold, no processing is performed; when the data standardization detection value is less than or equal to the first threshold, the digital economy-related data to be detected is standardized based on the result of the standardized detection value, and the data standardization processing includes removing data entries that do not meet the format requirements, correcting erroneous data, deleting duplicate data records and converting all data into a consistent format until the data standardization detection value rises above the first threshold.

[0017] Furthermore, the Apache Kafka technology is used to perform data fusion on the economic data to be integrated, and the data lake technology of Apache Hadoop and Amazon S3 is used to store the digital economy-related data to be integrated after the integration of data streams, so as to obtain the integrity, consistency, accuracy and richness of the integrated data during the data fusion process. The specific process is as follows: the Apache Kafka technology is used to perform data fusion on the economic data to be integrated, and the Topics of Kafka are used to perform data classification management on the economic data to be integrated, and Streams is used to perform real-time data processing on the economic data to be integrated; the data lake technology of Apache Hadoop and Amazon S3 is used to store the digital economy-related data to be integrated after the integration of data streams; after data fusion, the data integrity of each field is evaluated, the fusion integrity is recorded, whether the records after fusion of different data sources are consistent, the fusion consistency is recorded, and the records are compared with known true values ​​and standard data sets, the fusion accuracy is recorded, and the dimension and depth of the integrated data are evaluated by calculating the number of fields in the data table before and after fusion, and the data richness is recorded.

[0018] Furthermore, the specific calculation formula of the data fusion detection value is:

[0019] DFVI=ω fc e FC +ω fs log(1+FS)+ω fa FA 2 +ω dr DR+ω fcfs FC×FS+ω fadr FA×DR; where DFVI represents the data fusion detection value, FC represents the fusion data integrity, ω fc represents the weight factor of fusion data integrity, FS represents fusion data consistency, ω fs represents the weight factor of fusion data consistency, FA represents fusion accuracy, ω farepresents the weight factor of fusion accuracy, DR represents data richness, ω dr A weighting factor representing the richness of the data.

[0020] Furthermore, the specific process of making corresponding optimization processing measures according to the data fusion detection value is: compare the data fusion detection value with the second threshold in real time, and when the data fusion detection value is greater than the second threshold, do not process it; when the data fusion detection value is less than or equal to the second threshold, repeat data standardization processing and data fusion until the data fusion detection value recovers to above the second threshold.

[0021] The second aspect of the present invention provides a digital economy data collection system based on 5G communication, which is applied to the above-mentioned digital economy data collection method based on 5G communication, including a data collection module, a preprocessing and cleaning module, a standardization detection module, a data fusion module and a data analysis and optimization module, wherein: the data collection module is used to collect digital economy-related data, and the digital economy-related data is preliminarily formatted and cleaned before being stored in a database; the preprocessing and cleaning module is used to use an automated detection tool to preprocess the digital economy-related data stored in the database, mark the preprocessed digital economy-related data as digital economy-related data to be detected, and obtain the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency; the standardization detection module is used to obtain a data standardization detection value through comprehensive calculation of the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency, perform data standardization processing on the digital economy-related data to be detected according to the data standardization detection value, and mark the digital economy-related data to be detected after data standardization processing as digital economy-related data to be fused; the data fusion module is used to use Apache Kafka technology to perform data fusion on the economy-related data to be fused, and use Apache Hadoop and Amazon S3's data lake technology stores the digital economy-related data to be integrated after the integration of data streams, and obtains the integrity, consistency, accuracy and richness of the fused data during the data fusion process; the data analysis and optimization module is used to obtain the data fusion detection value through comprehensive calculation of the integrity, consistency, accuracy and richness of the fused data, and make corresponding optimization processing measures based on the data fusion detection value.

[0022] Beneficial Effects

[0023] The present invention has the following beneficial effects:

[0024] (1) The present invention uses advanced data lake technologies such as Apache Hadoop and Amazon S3 to store the merged data. This method makes the storage of large-scale data more efficient and economical, thereby achieving high data accessibility and simplifying data management. This effectively solves the problems of high data storage costs and complex data access in the prior art.

[0025] (2) The present invention introduces automated tools to perform data preprocessing and standardization testing. The method can systematically identify and correct errors and inconsistencies in the data, thereby improving data quality and consistency, and effectively solving the problem of frequent data errors and data inconsistencies in the prior art.

[0026] (3) The present invention uses Apache Kafka and data lake technology to efficiently fuse data, and evaluates the fusion results through comprehensive data fusion detection values, thereby optimizing data fusion and effectively solving the problem of incomplete or inefficient data fusion in the prior art.

[0027] (4) The present invention provides real-time feedback of data standardization detection values ​​and data fusion detection values. The method can dynamically adjust the data processing flow, thereby achieving continuous optimization of the data processing flow, and effectively solving the problem of lack of flexibility and adaptability in the prior art.

[0028] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of a digital economic data collection method based on 5G communication according to the present invention;

[0030] Figure 2 This is a structural diagram of the digital economy data collection system based on 5G communication in the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] See also Figure 1, the embodiment of the present invention provides a technical solution: a digital economy data collection system and method based on 5G communication, comprising the following steps: S1, collecting digital economy related data, preliminarily formatting and cleaning the digital economy related data and storing them in a database; S2, using an automated detection tool to preprocess the digital economy related data stored in the database, marking the preprocessed digital economy related data as digital economy related data to be detected, and obtaining the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency; S3, obtaining a data standardization detection value by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency, performing data standardization processing on the digital economy related data to be detected according to the data standardization detection value, and marking the digital economy related data to be detected after data standardization processing as digital economy related data to be integrated; S4, using Apache Kafka technology to perform data integration on the economic related data to be integrated, using Apache Hadoop and Amazon S3's data lake technology stores the digital economy-related data to be integrated after the integration of data streams, and obtains the integrity, consistency, accuracy and richness of the fused data during the data fusion process; S5 obtains the data fusion detection value through comprehensive calculation of the integrity, consistency, accuracy and richness of the fused data, and makes corresponding optimization processing measures based on the data fusion detection value.

[0033] Specifically, the data related to the digital economy are collected, and the data related to the digital economy are initially formatted and cleaned before being stored in the database. The specific process is as follows: First, a standardized data interface protocol is formulated for each data source. These protocols define in detail the format, type, and transmission method of the data, thereby ensuring that the data collected from the data source conforms to unified standards. Based on these strict data interface protocols, the system can effectively collect digital economy-related data from each data source. Furthermore, in order to improve the availability and quality of the data, the collected data will undergo preliminary formatting and cleaning. This process includes key steps such as removing invalid or erroneous data, unifying the data timestamp format, and adjusting the data structure according to business needs. After completing these necessary preprocessing, the data will be securely stored in the database, providing accurate and timely information support for subsequent data analysis. This systematic data processing process not only improves the efficiency of data management, but also ensures data quality and security.

[0034] In this implementation scheme, by formulating standardized data interface protocols for each data source and implementing systematic data collection, formatting, and cleaning processes, this step significantly improves the quality and availability of the data. Specifically, the unified data interface protocol ensures that the information collected from the diverse data sources is consistent and complete, which is the basis for effective data analysis. The preliminary data formatting and cleaning process further removes invalid and erroneous information in the data set, unifies the data format, and makes the data stored in the database more accurate and reliable. These measures not only optimize the data storage structure and improve data processing efficiency, but also enhance the security and manageability of the data, providing a solid data support foundation for subsequent data analysis and business decision-making. In addition, the high degree of automation of the entire process greatly reduces the possibility of human error and ensures the efficiency and accuracy of the data processing process.

[0035] Specifically, the digital economy-related data stored in the database is preprocessed using automated detection tools, and the preprocessed digital economy-related data is marked as the digital economy-related data to be detected, and the number of invalid entries, the number of erroneous entries, the number of duplicate entries, data integrity and data consistency are obtained. The specific process is as follows: The system uses Apache NiFi to automatically manage and clean data, which not only improves processing efficiency, but also ensures the controllability and repeatability of the data processing process. Apache NiFi, with its powerful streaming data processing capabilities, can continuously process and accurately control data flows, thereby achieving real-time data cleaning and formatting. At the same time, the Talend Data Quality tool is used to automatically identify and correct errors and inconsistencies in the data, including detecting data format errors, logical errors or inconsistent data records, thereby improving the overall quality of the data. In addition, by writing special regular expressions, the system can verify and format string type data, and automatically correct those data entries that do not conform to the predetermined format, such as date format, phone number or email address. This step not only reduces the need for manual intervention, but also significantly improves the standardization of data. At the end of this process, the system will record various data quality indicators, including the number of invalid entries, the number of erroneous entries, the number of duplicate entries, data integrity and data consistency, which are key indicators for evaluating data quality. Such preprocessing not only ensures the high quality standards of the data, but also marks the preprocessed digital economy-related data as digital economy-related data to be tested, laying a solid foundation for the subsequent data standardization detection and fusion steps.

[0036] In this implementation plan, the digital economy-related data stored in the database is preprocessed by using automated detection tools. This step effectively improves the accuracy and consistency of the data and provides a high-quality data foundation for subsequent analysis and application. The use of Apache NiFi to achieve automated data management and cleaning, and Talend DataQuality for error identification and correction, greatly improves processing efficiency and data quality. In addition, the use of regular expressions further ensures the standardization of data formats and automatically corrects entries that do not conform to the format, thereby reducing human errors and inconsistencies in data processing. Through these tools, invalid, erroneous and duplicate data entries are effectively identified and processed, while accurately recording data integrity and consistency indicators. This not only optimizes the storage and management of data, but also enhances the transparency and traceability of data throughout the collection and analysis process, ensuring the efficiency and accuracy of the data processing process.

[0037] Specifically, the data standardization test value is obtained by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency. The specific process is as follows: First, the system needs to obtain and record key data quality indicators, including the number of invalid entries, the number of erroneous entries, the number of duplicate entries and the total number of data entries. These indicators provide an intuitive understanding of data quality and help identify the main problems in the data set. Next, the system evaluates the integrity and consistency of the data. The data integrity indicator reflects the proportion of non-null values ​​in the data set, focusing on whether the data loses information during collection or processing. Data consistency checks the logical relationship in the data set to ensure that there is no contradiction between data entries and complies with predefined business rules or data relationship models. The data standardization test value is obtained by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency.

[0038] In this implementation scheme, the data standardization test value is obtained by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency. This step effectively quantifies various aspects of data quality. This process not only helps to identify and quantify specific problems in the data set, but also makes data processing decisions more data-driven and accurate by providing a comprehensive data quality score. The calculated data standardization test value provides a clear indicator basis for subsequent data processing steps, guides data cleaning and improvement work, and ensures that the data meets the required quality standards before entering the analysis and fusion stage. This method greatly improves the transparency and efficiency of the data processing process, makes data quality control more systematic and automated, and provides a solid guarantee for high-quality data analysis.

[0039] Specifically, the specific calculation formula of the data standardization detection value is:

[0040]

[0041] Where DSV stands for data standardization value, which is a comprehensive indicator used to evaluate the overall quality and consistency of data. C stands for data integrity, which measures the proportion of non-null values ​​in the data. c The weight factor that describes data integrity usually ranges from 0 to 1. The specific value is determined according to the importance of data integrity in the overall data quality. S represents data consistency, which evaluates the logic and format consistency in the data. s The weight factor of data consistency is also between 0 and 1, reflecting the proportion of consistency in the overall data quality. I represents the number of invalid entries, ω i represents the weight factor of the number of invalid entries, E represents the number of erroneous entries, ω e represents the weight factor for the number of erroneous entries, D represents the number of duplicate entries, ω d Represents the weight factor for the number of duplicate entries, ω i ,ω e ,ω d The setting of these weight factors depends on the severity of the impact of various errors on data quality, and is generally set in the range of 0 to 1. The specific values ​​of the weight factors are usually determined through historical data analysis, expert evaluation or data quality feedback mechanism to ensure that they can accurately reflect the severity of different types of data quality problems. T represents the total number of data items and is the basis for evaluating data quality. This formula calculates a standardized test value that can intuitively reflect the data quality status by combining various quality indicators and corresponding weights, so that data processing personnel can identify the main problems with the data and take corresponding optimization measures. This method not only enhances the scientificity and accuracy of data quality control, but also provides a practical tool for continuously improving and maintaining the quality level of the data processing process.

[0042] In this embodiment, by applying the calculation formula of the data standardization test value, this step effectively quantifies the various dimensions of data quality, including the integrity and consistency of the data, and the proportion of invalid, erroneous and duplicate entries in the data. By introducing weight factors, the method can accurately adjust the contribution of different error types to the overall data quality assessment according to their severity and impact, thereby providing a comprehensive and dynamic perspective to monitor and evaluate data quality. The flexible setting of the weight factor allows the method to be adjusted according to changes in actual application scenarios and business needs to achieve optimal data quality control. In addition, the calculation results of the data standardization test value provide a scientific basis for data processing and subsequent steps, guide the formulation of data cleaning, correction and optimization strategies, and thus achieve continuous improvement of the data processing process, effectively solving the problem that data quality is difficult to comprehensively evaluate and optimize in the prior art.

[0043] Specifically, the specific process of data standardization processing of the digital economy-related data to be detected according to the data standardization detection value is as follows: first, the data standardization detection value is compared with the set first threshold in real time. When the data standardization detection value is greater than this threshold, it indicates that the quality of the data has met the predetermined standard, so no additional processing is performed; when the data standardization detection value is less than or equal to the first threshold, the automatic data standardization processing procedure is triggered.

[0044] At this point, based on the results of the standardized test values, the system begins to perform a series of standardized processing measures on the digital economy-related data to be tested. These measures include removing data entries that do not meet format requirements, such as incorrect date formats or non-standard numerical representations; correcting data with logical errors or input errors to ensure that all data entries are accurate; deleting duplicate records in the data, which not only improves data accuracy, but also optimizes storage usage and processing efficiency; and converting all data into a consistent format, such as a unified currency unit or standardized address format, to enhance data availability and consistency.

[0045] These standardization processes continue until the standardized detection value of the data rises above the first threshold again. In this way, the system can ensure that the data continues to meet high quality standards throughout its life cycle. At the same time, the real-time monitoring and dynamic response mechanism of data quality effectively reduces the need for manual intervention and improves the automation and intelligence level of the data processing process.

[0046] In this embodiment, by real-time monitoring of the data standardization test value and comparing it with a preset threshold value, this step effectively realizes continuous automatic monitoring and control of data quality. When the data standardization test value is lower than the first threshold value, the automatically started data standardization process ensures that data errors are corrected in time, the format is unified, and duplicate entries are removed. This not only greatly improves the accuracy and consistency of the data, but also optimizes the efficiency of data storage. In addition, the automation of this processing flow reduces the need for manual intervention and improves the efficiency and reliability of the entire data processing system. Through this dynamic data quality management method, it can be ensured that all data meet the highest quality standards before entering the analysis and decision-making stage, thereby effectively solving the problem that data quality in the prior art is difficult to monitor in real time and optimize in time.

[0047] Specifically, Apache Kafka technology is used to fuse the economic data to be fused, and Apache Hadoop and Amazon S3's data lake technology are used to store the digital economic data to be fused after the integration of data streams, so as to obtain the integrity, consistency, accuracy and richness of the fused data during the data fusion process. The specific process is: Use Apache Kafka to effectively fuse the economic data to be fused. Through Kafka's Topics function, effective classification management of the data to be fused is realized, and Kafka Streams is used to process the real-time data of these data streams to ensure the real-time and accuracy of the data during integration.

[0048] Next, we use data lake technologies such as Apache Hadoop and Amazon S3, which not only provide scalable storage solutions but also support the processing of large-scale data sets, thus ensuring the efficiency of the data fusion process and the security of data storage. Data lake technology allows the storage of structured and unstructured data processed by Kafka, providing strong data support for subsequent data analysis and mining.

[0049] After the data fusion is completed, this method continues to evaluate the data quality, including the detection of the integrity, consistency, accuracy and richness of the fused data. First, evaluate the data integrity of each field and record whether the data is lost or damaged during the fusion process; second, check whether the records after fusion of different data sources are consistent to ensure data consistency; third, verify the accuracy of the fused data by comparing with known true values ​​and standard data sets; finally, by calculating the number of fields in the data table before and after fusion, evaluate whether the dimension and depth of the data have increased, thereby reflecting the change in data richness. This whole process ensures the high quality and wide applicability of data after fusion, providing enterprises with a reliable data analysis foundation.

[0050] In this implementation, the efficiency and flexibility of data processing are greatly improved by using Apache Kafka for data fusion and Apache Hadoop and Amazon S3's data lake technology for data storage. Kafka's high throughput and real-time processing capabilities ensure the timeliness and accuracy of data during the fusion process, while data lake technology provides powerful storage capabilities and high scalability, making the storage and management of large-scale data sets more efficient. In addition, by systematically evaluating the integrity, consistency, accuracy, and richness of the fused data, the high quality of the fused data is ensured, thus providing a solid foundation for subsequent data analysis and decision-making. This not only optimizes the entire data processing process and improves data utilization, but also enhances the security and reliability of data, providing a strong data support system for enterprises.

[0051] Specifically, the specific calculation formula of the data fusion detection value is:

[0052] DFVI=ω fc e FC +ω fs log(1+FS)+ω fa FA 2 +ω dr DR+ω fcfs FC×FS+ω fadr FA×DR; where DFVI represents the data fusion detection value, which is obtained by weighted summation of multiple key performance indicators, FC represents the fusion data integrity, and its influence in data fusion is emphasized by the exponential function, ω fc represents the weight factor of fused data integrity, FS represents fused data consistency, and its impact is expressed through a logarithmic function to emphasize the decreasing marginal benefit of consistency in the high value range, ω fs represents the weight factor of fusion data consistency, FA represents fusion accuracy, and a square function is used to amplify the impact of accuracy issues, ω fa Represents the weight factor of fusion accuracy, DR represents data richness, and directly expresses its influence linearly, ω dr The weight factor representing the data richness, ω fc ,ω fs ,ω fa ,ω drThese weight factors are set according to their importance in data fusion. The setting of weight factors is usually based on historical data analysis, expert experience evaluation or automatic adjustment through optimization algorithms to ensure the rationality and accuracy of each factor in the evaluation. The value range of weight factors is usually between 0 and 1. This is done to standardize the influence of each factor and keep the total detection value normalized. In addition, the formula also includes the interaction terms FC×FS and FA×DR, which are respectively determined by the weight factors ω fcfs and ω fadr Adjustment, these interaction terms take into account the impact of the interaction between different data quality dimensions on the fusion effect. This calculation method provides a means to comprehensively evaluate the quality of data fusion, which can accurately point out the advantages and disadvantages of the data fusion process, help data managers better understand the quality status after data fusion, and make optimization adjustments accordingly.

[0053] In this implementation scheme, by implementing the calculation formula of the data fusion detection value, this step effectively conducts a comprehensive evaluation of the data fusion process and clearly quantifies the integrity, consistency, accuracy and richness of the fused data. This evaluation method not only adopts the basic weighted sum model, but also introduces interaction terms to consider the interaction between different data quality dimensions, thereby more comprehensively reflecting the quality of data fusion. The careful setting of the weight factors ensures that each quality indicator is properly considered according to its importance in the overall data quality, so that the data fusion detection value can provide a detailed and accurate snapshot of data quality. This method not only helps the data processing team identify and solve problems in data fusion, but also provides a solid quality assurance foundation for subsequent data use and analysis decisions, effectively optimizing the entire data fusion process and improving the practical value of data.

[0054] Specifically, the corresponding optimization measures are made according to the data fusion detection value. The specific process is: compare the data fusion detection value with the preset second threshold in real time. This method ensures the continuity and quality control of data fusion. When the data fusion detection value is greater than this threshold, it indicates that the quality of data fusion has reached or exceeded the expected standard, so no further processing is required. In this case, the data has shown a high degree of integrity, consistency, accuracy and richness, which meets the requirements of subsequent analysis and application.

[0055] On the contrary, when the data fusion detection value is lower than or equal to the second threshold, this indicates that the result of data fusion has not yet reached the predetermined quality standard and additional measures are needed to improve the data quality. In this case, the system automatically triggers a loop processing process to repeat the data standardization and data fusion steps. This includes re-evaluating errors, inconsistencies, repeatability issues, and lack of integrity in the data, and performing necessary cleaning, correction, and re-fusion of the data until the data fusion detection value is raised above the second threshold. This dynamic adjustment process not only ensures the final quality of data fusion, but also ensures the adaptability and flexibility of the entire data processing process, so that the data can continue to meet higher standards.

[0056] In this implementation scheme, by real-time monitoring of the data fusion detection value and comparing it with the set second threshold, this step achieves continuous quality control and optimization of the data fusion process. When the data fusion detection value exceeds the second threshold, it indicates that the data quality has reached or exceeded the predetermined standard and no further processing is required, thereby maintaining processing efficiency. Conversely, when the data fusion detection value is lower than the second threshold, the system will automatically repeat the data standardization and fusion process until the data quality is improved to meet or exceed the threshold, ensuring the high quality of the data fusion results. This dynamic adjustment mechanism not only enhances the reliability of data fusion, but also optimizes the data processing process to ensure that the data reaches the optimal quality before use.

[0057] Reference Figure 2As shown, the second aspect of the present invention provides a digital economy data acquisition system based on 5G communication, including a data acquisition module, a preprocessing and cleaning module, a standardization detection module, a data fusion module and a data analysis and optimization module, wherein: the data acquisition module is used to collect digital economy-related data, and the digital economy-related data is preliminarily formatted and cleaned before being stored in a database; the preprocessing and cleaning module is used to use an automated detection tool to preprocess the digital economy-related data stored in the database, mark the preprocessed digital economy-related data as digital economy-related data to be detected, and obtain the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency; the standardization detection module is used to obtain a data standardization detection value through comprehensive calculation of the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency, perform data standardization processing on the digital economy-related data to be detected according to the data standardization detection value, and mark the digital economy-related data to be detected after data standardization processing as digital economy-related data to be fused; the data fusion module is used to use Apache Kafka technology to perform data fusion on the economy-related data to be fused, and use Apache Hadoop and Amazon S3's data lake technology stores the digital economy-related data to be integrated after the integration of data streams, and obtains the integrity, consistency, accuracy and richness of the fused data during the data fusion process; the data analysis and optimization module is used to obtain the data fusion detection value through comprehensive calculation of the integrity, consistency, accuracy and richness of the fused data, and make corresponding optimization processing measures based on the data fusion detection value.

[0058] In this implementation plan, the system is carefully designed to cover all key stages of data processing, thereby achieving full process optimization from data collection to analysis. The system consists of five main modules: data collection module, preprocessing and cleaning module, standardization detection module, data fusion module and data analysis and optimization module. The data collection module is responsible for collecting various types of digital economy-related data and performing preliminary formatting and cleaning to ensure that the data quality meets basic standards before entering the database. The preprocessing and cleaning module uses automated detection tools to further preprocess the data in the database, identifies data items that need special attention, such as invalid, erroneous or duplicate entries, and obtains key data quality indicators to provide a basis for subsequent processing; the standardization detection module comprehensively calculates the data standardization detection value based on the data quality indicators obtained during the preprocessing process, and performs necessary standardization processing on the data to ensure that the data meets high quality requirements before entering the fusion stage; the data fusion module uses Apache Kafka technology for efficient data fusion processing, and uses Apache Hadoop and Amazon S3's data lake technology for data storage to ensure that the data remains complete and consistent after fusion; the data analysis and optimization module implements corresponding optimization processing measures based on the data fusion detection value calculated by the quality indicators after data fusion to improve the analysis accuracy and practical value of the data; the design of the entire system makes full use of the high speed and low latency characteristics of 5G communication to ensure the efficiency and real-time performance of the data processing process. Through the organic coordination and continuous operation of these modules, the system not only improves the automation and intelligence level of data processing, but also ensures that the data meets the highest quality standards at each stage, and ultimately supports the generation of high-quality analysis results, providing reliable data support for decision makers.

[0059] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0060] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A digital economic data collection method based on 5G communication, characterized in that: The following steps are involved: S1, collect digital economy related data, format and clean the digital economy related data preliminarily and store them in the database; S2, using an automated detection tool to preprocess the digital economy-related data stored in the database, marking the preprocessed digital economy-related data as digital economy-related data to be detected, and obtaining the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity, and data consistency; S3, obtains a data standardization test value by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency, performs data standardization processing on the digital economy-related data to be tested according to the data standardization test value, and marks the digital economy-related data to be tested that has undergone data standardization processing as digital economy-related data to be integrated; S4, using Apache Kafka technology to fuse the economic data to be fused, using Apache Hadoop and Amazon S3's data lake technology to store the digital economy-related data to be fused after the integration of data streams, and obtaining the integrity, consistency, accuracy and richness of the fused data during the data fusion process; S5, obtains the data fusion detection value through comprehensive calculation of fusion data integrity, fusion data consistency, fusion accuracy and data richness, and makes corresponding optimization processing measures according to the data fusion detection value.

2. According to claim 1, a digital economic data collection method based on 5G communication is characterized in that: The specific process of collecting digital economy related data, preliminarily formatting and cleaning the digital economy related data and storing them in the database is as follows: Develop a standardized data interface protocol for each data source, collect digital economy-related data from each data source based on the developed data interface protocol, perform preliminary formatting and cleansing of the digital economy-related data according to the preset data structure and format, and then store them in the database.

3. According to a method for collecting digital economic data based on 5G communication according to claim 1, it is characterized in that: The specific process of using the automated detection tool to preprocess the digital economy-related data stored in the database, marking the preprocessed digital economy-related data as the digital economy-related data to be detected, and obtaining the number of invalid entries, the number of erroneous entries, the number of duplicate entries, data integrity and data consistency is as follows: Use Apache NiFi to automate the management and cleansing of digital economy-related data, use Talend DataQuality to automatically identify and correct errors and inconsistencies in digital economy-related data, and record the data integrity and consistency of digital economy-related data; Use regular expressions to verify and format string-type data in digital economy-related data, automatically correct data entries that do not conform to the format in digital economy-related data, and record the number of invalid entries, erroneous entries, duplicate entries, and the total number of data entries in the digital economy-related data.

4. According to a method for collecting digital economic data based on 5G communication according to claim 1, it is characterized in that: The specific process of obtaining the data standardization test value by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency is as follows: The number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency are obtained, and the data standardization detection value is obtained by comprehensive calculation of the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency.

5. According to claim 4, a digital economic data collection method based on 5G communication is characterized in that: The specific calculation formula of the data standardization detection value is: In the formula, DSV represents data standardization detection value, C represents data integrity, ω c The weight factor that expresses data integrity, S represents data consistency, ω s represents the weight factor of data consistency, I represents the number of invalid entries, ω i represents the weight factor of the number of invalid entries, E represents the number of erroneous entries, ω e represents the weight factor for the number of erroneous entries, D represents the number of duplicate entries, ω d represents the weight factor of the number of duplicate entries, and T represents the total number of data entries.

6. The digital economic data collection method based on 5G communication according to claim 1 is characterized in that: The specific process of performing data standardization processing on the digital economy related data to be detected according to the data standardization detection value is as follows: Compare the data standardization detection value with the first threshold in real time, and do not process when the data standardization detection value is greater than the first threshold; When the data standardization detection value is less than or equal to the first threshold, the digital economy-related data to be detected is standardized based on the results of the standardized detection value. The data standardization includes removing data entries that do not meet the format requirements, correcting erroneous data, deleting duplicate data records and converting all data into a consistent format until the data standardization detection value rises above the first threshold.

7. The digital economic data collection method based on 5G communication according to claim 1 is characterized in that: The specific process of using Apache Kafka technology to fuse the economic data to be fused, using Apache Hadoop and Amazon S3's data lake technology to store the digital economic data to be fused after the integration of data streams, and obtaining the integrity, consistency, accuracy and richness of the fused data during the data fusion process is as follows: Use Apache Kafka technology to integrate economic data to be integrated, use Kafka's Topics to classify and manage economic data to be integrated, and use Streams to process economic data in real time; Use Apache Hadoop and Amazon S3's data lake technology to store the digital economy-related data to be integrated after the data streams are integrated; After data fusion, evaluate the data integrity of each field and record the fusion integrity. Check whether the records after fusion of different data sources are consistent and record the fusion consistency. Compare with known true values ​​and standard data sets and record the fusion accuracy. Evaluate whether the dimension and depth of the fused data have increased by calculating the number of fields in the data table before and after fusion and record the data richness.

8. The digital economic data collection method based on 5G communication according to claim 1 is characterized in that: The specific calculation formula of the data fusion detection value is: DFVI=ω fc e FC +oh fs log(1+FS)+ω fa FA 2 +oh dr DR+ω fcfs FC×FS+ω fadr FA×DR; In the formula, DFVI represents the data fusion detection value, FC represents the fusion data integrity, ω fc represents the weight factor of fusion data integrity, FS represents fusion data consistency, ω fs represents the weight factor of fusion data consistency, FA represents fusion accuracy, ω fa represents the weight factor of fusion accuracy, DR represents data richness, ω dr A weighting factor representing the richness of the data.

9. The digital economic data collection method based on 5G communication according to claim 1 is characterized in that: The specific process of making corresponding optimization measures according to the data fusion detection value is as follows: Compare the data fusion detection value with the second threshold in real time, and when the data fusion detection value is greater than the second threshold, no processing is performed; When the data fusion detection value is less than or equal to the second threshold, data normalization processing and data fusion are repeatedly performed until the data fusion detection value is restored to be above the second threshold.

10. A digital economic data collection system based on 5G communication, characterized in that: A digital economy data collection method based on 5G communication as described in any one of claims 1 to 9, comprising a data collection module, a preprocessing and cleaning module, a standardization detection module, a data fusion module and a data analysis and optimization module, wherein: The data collection module is used to collect digital economy-related data, and store the digital economy-related data in the database after preliminary formatting and cleaning; A preprocessing and cleaning module is used to use an automated detection tool to preprocess the digital economy-related data stored in the database, mark the preprocessed digital economy-related data as digital economy-related data to be detected, and obtain the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity, and data consistency; A standardization detection module is used to obtain a data standardization detection value by comprehensively calculating the number of invalid entries, the number of erroneous entries, the number of duplicate entries, the total number of data entries, data integrity and data consistency, perform data standardization processing on the digital economy-related data to be detected according to the data standardization detection value, and mark the digital economy-related data to be detected that has undergone data standardization processing as digital economy-related data to be integrated; The data fusion module is used to use Apache Kafka technology to fuse the economic data to be fused, and use Apache Hadoop and Amazon S3's data lake technology to store the digital economy-related data to be fused after the integration of data streams, and obtain the integrity, consistency, accuracy and richness of the fused data during the data fusion process; The data analysis and optimization module is used to obtain the data fusion detection value through comprehensive calculation of fusion data integrity, fusion data consistency, fusion accuracy and data richness, and make corresponding optimization processing measures according to the data fusion detection value.

Images