Data processing method and system based on big data collection

Abstract

This invention discloses a data processing method and system based on big data collection, relating to the field of big data collection technology. The method includes the following steps: aggregating relevant data from multiple data sources, cleaning the aggregated data, and inserting the cleaned data into a central data warehouse to form a unified data foundation; based on the central data warehouse, configuring the relationships and collection conditions between various data sources according to preset regulatory themes through a data collection adaptation program, establishing a data lineage diagram, and determining the logical relationships and field ranges of the target data to be collected. This invention establishes a unified central data warehouse through data aggregation and cleaning, achieving standardized fusion of multi-source data; it collects data on demand based on the data lineage diagram and constructs a thematic library, improving query efficiency; it achieves multi-dimensional analysis and flexible expansion through a dynamically associated indicator library; and it introduces incremental updates and visualization, constructing a real-time, efficient, and scalable data processing closed loop.

Description

Technical Field

[0001] This invention relates to the field of big data acquisition technology, and more specifically to data processing methods and systems based on big data acquisition. Background Technology

[0002] With the continuous development of information systems, data statistical analysis has become an important component of various business management platforms. During daily operation, systems generate a large amount of multi-source, heterogeneous data. This data needs to be cleaned, aggregated, and statistically analyzed to provide decision support and operational monitoring for managers. Currently, mainstream data statistical methods mainly rely on direct queries to the system database or the construction of intermediate tables. In the direct query method, the system executes complex statistical logic to perform multi-table joins and calculations on business tables in the database to obtain the required statistical results. In the intermediate table method, the system typically establishes a dedicated statistical table or cache table outside the main database, synchronizing business data to the intermediate table before performing statistical operations.

[0003] The existing technology has the following shortcomings: Existing data statistical analysis methods generally suffer from performance and efficiency bottlenecks. Direct querying, due to its large data volume, complex logic, and numerous inter-table relationships, results in slow query speeds and significantly impacts overall system performance. While using intermediate tables can partially alleviate performance pressure, its statistical results have long update cycles and can only meet relatively fixed indicator requirements. When statistical needs change frequently, dimensions diversify, or cross-domain fusion analysis is involved, traditional methods struggle to adapt flexibly, leading to delayed statistical results, poor scalability, and an inability to effectively support multi-level, multi-dimensional data analysis needs.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a data processing method and system based on big data collection to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a data processing method based on big data collection, comprising the following steps: Step 1: Aggregate relevant data from multiple data sources, clean the aggregated data, and insert the cleaned data into the central data warehouse to form a unified data foundation; Step 2: Based on the main data warehouse, according to the preset regulatory themes, configure the relationship and collection conditions between various data sources through the data collection adaptation program, establish a data lineage diagram, and determine the logical relationship and field range of the target data to be collected; Step 3: Based on the data lineage diagram and collection conditions, perform data collection operations from the main data warehouse to generate corresponding thematic data sets. Store the collected data in the cache unit and database tables to build a data theme library corresponding to the regulatory theme. Step 4: Based on the data theme library, associate and combine the data items that need to be statistically analyzed with the preset indicator library and algorithm model to generate a data analysis engine that can be used for statistical analysis; Step 5: Perform data statistical analysis operations using the data analysis engine, visualize the analysis results, and export the data to complete dynamic data processing and statistical analysis based on the topic library.

[0007] Preferably, the process of aggregating relevant data includes: when collecting raw data from multiple source systems, performing unified format conversion and structural standardization on data from different sources, in different formats, and with different time identifiers; executing field type mapping and format unification operations through preset data import rules; removing duplicate information, filling in missing information, and correcting outliers in the collected raw data to ensure data consistency and integrity; after cleaning, inserting the processed data into a central data warehouse according to unified field standards. The central data warehouse is used to centrally store standardized multi-source fused data and achieves cross-source fusion using a unified data structure definition method, so that data from different sources can be uniformly managed and accessed, thereby providing a consistent data foundation and traceable data sources for subsequent data collection, theme construction, and analysis.

[0008] Preferably, the process of establishing a data lineage diagram includes: configuring the logical relationships, field dependencies, and collection conditions between various data sources through a data acquisition adaptation program; defining the mapping relationships and data retrieval logic between data items according to the needs of different business themes; and forming a lineage structure diagram to describe the data source path and transmission relationship. This data lineage diagram provides logical guidance during data collection, clarifies the data source, dependency, and flow of each field, and automatically determines the composition and scope of the target data set based on this diagram when performing collection operations. In this way, the logical structure between different data sources is clearly expressed, and the collection scope and granularity can be flexibly adjusted according to the business theme, thereby achieving on-demand collection and precise extraction, effectively reducing redundant data access and improving the efficiency and accuracy of data collection.

[0009] Preferably, the process of generating corresponding thematic data sets includes: when performing on-demand data collection, extracting data items and records related to the target theme from the main data warehouse based on field mappings and logical dependencies in the data lineage diagram; organizing and classifying the extracted data by logical theme to form data sets corresponding to each regulatory theme; allocating the data sets to cache units to support fast access, and simultaneously writing them into database tables for long-term storage and historical analysis; through thematic data organization, the collected data exists logically independently and has inherent consistency, and cross-referencing between different thematic data can be achieved through indicator logic; this process constructs a hierarchical data organization structure, providing an efficient data foundation for subsequent data analysis engine generation and multi-theme parallel analysis.

[0010] Preferably, the process of constructing a data theme library corresponding to the regulatory theme includes: after data collection, the target data sets corresponding to each business theme are collected and structured; field mapping and aggregation are performed according to predefined field relationships and logical dependencies; and the data is stored hierarchically in cache units and database tables to achieve both fast access and persistent storage. Each theme library establishes logical boundaries around an independent business theme and supports independent invocation and parallel processing in subsequent analysis. The internal structure of the theme library includes two types of data: basic business fields and derived auxiliary fields, to support statistical and multidimensional analysis needs. This theme-based organization method reduces cross-table access while ensuring data integrity, enabling the system to have higher query efficiency and flexible expansion capabilities when facing complex multidimensional data statistics.

[0011] Preferably, the process of associating and combining the data items to be statistically analyzed with a preset indicator library and algorithm model includes: selecting the data items required for analysis from the theme library based on the theme structure and data hierarchy defined in the theme library; matching the selected data items with the corresponding statistical dimensions and indicator definitions in the indicator library; and generating the data set to be analyzed through logical combination. The indicator library includes multi-dimensional indicator definition information, which is used to parametrically describe the statistical scope, aggregation method, and analysis perspective of different theme data. Through a dynamic association mechanism, the analysis dimensions and indicator content can be flexibly expanded according to actual needs, and new analysis combination logic can be formed without modifying the structure of the data theme library. This process makes data statistics and analysis flexible and scalable, and can adapt to the dynamic analysis needs of multiple themes and multiple scenarios.

[0012] Preferably, the data statistical analysis operation includes: performing multi-dimensional data statistics based on the generated data analysis engine, generating corresponding statistical result sets by aggregating, classifying, and performing correlation analysis on the data in the subject library; the statistical results can be output in the form of charts, tables, or visualizations according to the dimensions defined in the indicator library, so that the differences, trends, and distribution of the data results can be presented in an intuitive way; after the analysis is completed, the result data can be exported in the form of structured data files through the export function, so as to generate reports or be called by external systems; this process, through a unified display and export mechanism, enables the statistical results to be reused in various business scenarios, realizes the traceability and visualization of the entire process from data collection to output, and improves the operability and decision support capabilities of the system.

[0013] Preferably, after the data analysis engine is generated, the synchronization between the data subject library and the main data warehouse is periodically maintained by introducing timed incremental update rules to achieve dynamic data updates. When the system detects new data being written to the main data warehouse, it automatically identifies the added or changed data according to preset update rules and updates the changed content to the corresponding data subject library to ensure that the data on which the analysis is based remains real-time and continuous. The update process is carried out without affecting the normal access to the subject library, realizing continuous iteration of data processing. This design improves the system's responsiveness in dynamic data environments, ensuring that statistical analysis is always based on the latest data status and meeting the needs of real-time analysis and continuous decision-making.

[0014] Preferably, dynamic data processing and statistical analysis based on subject libraries includes: in multi-subject parallel scenarios, the system can simultaneously call multiple data subject libraries according to different business needs, and perform cross-subject data integration and joint statistics through the data analysis engine; the multi-subject joint analysis uses indicator logic as a link to associate and match data items in different subject libraries, realizing cross-business dimension data fusion and trend analysis; through this parallel analysis mechanism, the system can realize synchronous analysis of multiple subjects, multiple indicators, and multiple dimensions within a unified platform, avoiding the problem of repeated collection and calculation caused by data fragmentation in traditional statistical methods, thereby improving the efficiency of analysis and processing and data utilization, and constructing a scalable dynamic analysis architecture.

[0015] A data processing system based on big data collection includes a data aggregation and cleaning module, a data association and modeling module, a thematic data construction module, a data analysis and generation module, and a result display and export module. Data aggregation and cleaning module: Aggregates relevant data from multiple data sources, cleans the aggregated data, and inserts the cleaned data into the central data warehouse to form a unified data foundation; Data association modeling module: Based on the main data warehouse, according to the preset regulatory themes, the module configures the association relationships and collection conditions between various data sources through the data collection adaptation program, establishes a data lineage diagram, and determines the logical relationships and field ranges of the target data to be collected. Thematic data construction module: Based on the data lineage diagram and collection conditions, it performs data collection operations from the main data warehouse, generates corresponding thematic data sets, stores the collected data in the cache unit and database tables, and constructs a data thematic library corresponding to the regulatory themes; Data analysis generation module: Based on the data theme library, the data items that need to be statistically analyzed are associated and combined with the preset indicator library and algorithm model to generate a data analysis engine that can be used for statistical analysis; Results Display and Export Module: Performs data statistical analysis operations based on the data analysis engine, visualizes the analysis results, and exports the data, completing dynamic data processing and statistical analysis based on the topic library.

[0016] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention establishes a unified data aggregation and cleaning process before data collection, creating a central data warehouse to achieve standardized fusion of multi-source heterogeneous data. Through format conversion, field standardization, and anomaly removal, data from different sources achieves a unified structure, avoiding collection and analysis errors caused by inconsistent formats. This solution enables centralized management across systems without changing the data source, improving data consistency and traceability. Compared to traditional direct query methods, this invention reduces cross-table operations and redundant accesses through pre-cleaning and centralized aggregation, significantly improving data integration efficiency and system response speed.

[0017] This invention establishes a data lineage graph through a data acquisition adaptation program and performs on-demand data acquisition accordingly. This allows for flexible adjustment of the acquisition scope and granularity, avoiding the performance burden caused by traditional full-volume scanning. Through a hierarchical structure of thematic datasets and thematic data libraries, data is organized and managed by theme, enabling independent access and parallel analysis of different thematic data. This structure shortens access paths, reduces I / O consumption, and improves query efficiency in multidimensional statistics, providing technical support for high-performance parallel analysis in complex data environments.

[0018] This invention, based on a data theme library, dynamically associates and combines data items with a preset indicator library to form an analysis engine adaptable to multiple dimensions and indicators. Analysis dimensions and indicators can be expanded as needed, and new analysis logic can be quickly generated without adjusting the underlying structure. This solution breaks through the limitations of fixed statistical calibers in traditional intermediate tables, enabling customized, multi-theme cross-analysis and fusion analysis, significantly enhancing the flexibility and scalability of data analysis, and continuously expanding analytical capabilities according to business changes.

[0019] This invention introduces a timed incremental update rule to dynamically maintain synchronization between the main data warehouse and the subject database. This allows the system to automatically update the subject database content when new data is written, ensuring data real-time performance and continuity. Analysis results are generated based on the latest data and are visualized and exported through charts, tables, and other formats, intuitively reflecting data trends. This process constructs an automated closed loop for data collection, organization, analysis, and presentation, significantly improving the timeliness of data processing and decision support capabilities. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0021] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention.

[0022] Figure 2 This is a flowchart of the method in Embodiment 2 of the present invention.

[0023] Figure 3 This is a flowchart of the data processing method based on big data collection according to the present invention.

[0024] Figure 4 This is a schematic diagram of the data processing system based on big data acquisition according to the present invention; Figure 5 This is a comparison chart of the system response performance under different concurrent pressures according to the present invention; Figure 6 Extend the demand response efficiency comparison distribution chart to include complex metrics. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0026] This invention provides, for example Figures 1 to 3 The data processing method based on big data collection shown includes the following steps: Step 1: Aggregate relevant data from multiple data sources, clean the aggregated data, and insert the cleaned data into the central data warehouse to form a unified data foundation for subsequent data collection; Step 2: Based on the main data warehouse, according to the preset regulatory themes, configure the relationship and collection conditions between various data sources through the data collection adaptation program, and establish a data lineage diagram to determine the logical relationship and field range of the target data to be collected; Step 3: Based on the data lineage diagram and collection conditions, perform data collection operations from the main data warehouse to generate corresponding thematic data sets. Store the collected data in cache units and database tables to build a data theme library corresponding to the regulatory themes. Step 4: Based on the data theme library, the data items that need to be statistically analyzed are associated and combined with the preset indicator library and algorithm model to generate a data analysis engine that can be used for statistical analysis, so as to realize multi-dimensional statistics and analysis. Step 5: Perform data statistical analysis operations based on the data analysis engine, visualize the analysis results and export the data to complete dynamic data processing and statistical analysis based on the topic library.

[0027] Example 1: This embodiment provides a data processing method based on big data acquisition, used to achieve efficient data integration, on-demand acquisition, and flexible analysis in a multi-source heterogeneous data environment, to meet the needs of multi-dimensional data statistics and dynamic analysis. The method mainly includes the following sequential and interrelated steps: This process aggregates various types of data related to the target business from multiple data sources, which can include data sets from different business scenarios or in different formats. During data aggregation, the raw data is uniformly formatted according to preset data import rules. Duplicates, missing items, and format anomalies in each data source are identified and removed. Field standardization, time format unification, and data type conversion are performed during the cleaning process. This ensures that the aggregated data has consistent structured characteristics, facilitating subsequent operations. The cleaned data is then inserted into a central data warehouse, which serves as a unified centralized data carrier to store the standardized, multi-source fused data. Through this step, data originally scattered across different data sources is centrally integrated, forming a complete and unified data infrastructure environment, providing reliable data support for subsequent data collection and analysis.

[0028] After establishing a unified data warehouse, based on pre-defined regulatory themes, the logical relationships between different data sources are defined through a data acquisition adaptation program, and theme-related data acquisition conditions are set. This process first determines the correspondence and dependencies between different data fields based on business logic. Then, the acquisition scope and granularity are determined by configuring acquisition rules and filtering conditions to generate a data lineage diagram describing the logical connections between data items. This data lineage diagram not only reflects the correlation between data but also clarifies the source path and data retrieval logic of each data field, thus ensuring the traceability and consistency of subsequent acquisition operations. Based on the generated data lineage diagram and corresponding acquisition conditions, on-demand acquisition operations are performed from the data warehouse to extract target data related to the regulatory themes. This on-demand acquisition method effectively reduces redundant data, improves acquisition efficiency, and forms a preliminary thematic data set, laying the foundation for building a data theme library.

[0029] Subsequently, based on the aforementioned collected thematic data sets, the target data is aggregated and structured for storage by logical themes, forming a thematic data library for specific themes. During this process, according to the field mappings and logical structures defined in the data lineage diagram, the collected data is organized and stored hierarchically, ensuring that related data under the same theme maintains inherent consistency and composability. The resulting thematic data library not only includes original business fields but also auxiliary and derived fields prepared for statistics and analysis, thereby improving the applicability of the data in the statistical analysis phase. The data in this thematic library is stored separately in cache units and database tables to enable rapid reading and access during data analysis. The thematic library established in this way can achieve independent data organization and management for different regulatory themes, ensuring both efficient data access and supporting the parallel data analysis needs of multiple themes, providing a stable data source for subsequent statistical and visualization steps.

[0030] Finally, based on the generated data theme library, the data items requiring statistical analysis are associated and combined with a pre-defined indicator library and algorithm model to construct a data analysis engine for performing the analysis. This engine can perform combined calculations and statistical summaries on various types of data from the data theme library according to different statistical dimensions and indicator definitions. During analysis, target data items are first selected from the theme library, aggregated and categorized according to the statistical dimensions defined in the indicator library, and then the analysis results are combined with a pre-defined output format to generate the final display data. The resulting analysis results can be presented through charts, tables, or other visualizations, allowing users to intuitively understand the data distribution and trends across different dimensions. Simultaneously, the analysis results can be exported in various formats for report generation or further processing. Through this step, the entire data processing process achieves a complete closed loop from multi-source aggregation, on-demand collection, theme construction to flexible analysis, significantly improving the efficiency and flexibility of data statistics and analysis while ensuring data consistency and scalability.

[0031] Through the above sequential steps, this implementation method organically integrates previously scattered data sources into a unified data foundation. Based on this foundation, a topic-oriented data acquisition and analysis mechanism is established, transforming the data processing process from a traditional full-data query model to an on-demand acquisition and hierarchical analysis model. This method ensures the controllability and consistency of the acquisition process through the logical guidance of a data lineage diagram; achieves structured data management and rapid access through the establishment of a data topic library; and completes multi-dimensional statistical and visualization analysis through the combination and execution of data analysis engines. This results in a data processing method that is efficient, flexible, and scalable in complex data environments. The overall process of this method has a clear hierarchical relationship and rigorous logical connections. Each step is interdependent and closely linked, enabling flexible application in different business scenarios and achieving high-performance data acquisition and analysis functions.

[0032] Example 2: This embodiment provides a data processing method based on big data acquisition. This method is an optimization of Embodiment 1, designed to adapt to the comprehensive needs of multi-source complex data environments for data acquisition flexibility, granular topic segmentation, and diverse statistical analysis. This method achieves efficient data acquisition and analysis processing in a big data environment through the orderly combination of sequential sub-steps: data aggregation, acquisition adaptation, topic library generation, and analysis execution. The specific steps are as follows: Raw data relevant to the target business is collected from multiple data sources. During the collection process, the data from different sources is formatted and standardized to ensure consistency in field names, types, and value formats across different sources. The collected data is then cleaned to remove duplicate information and abnormal records, and the time, category, and identifier fields of each data item are standardized to ensure all data conforms to preset format specifications. The cleaned data is centrally written into a data warehouse, which houses the diverse data from different sources. A unified data structure definition enables cross-source fusion, providing stable foundational data support for subsequent on-demand data collection and analysis. Through this step, data originally scattered across multiple sources is integrated into a unified and manageable dataset, forming a complete data infrastructure environment and providing data input sources for subsequent steps.

[0033] After the central data warehouse is established, data relationships and collection conditions are configured through a data acquisition adaptation program based on pre-defined regulatory themes. By analyzing business requirements, logical relationships and dependency paths between different data themes are defined, and a data lineage diagram is created accordingly. This diagram describes the logical connections between various data items and the data transmission relationships during collection and aggregation, accurately reflecting data sources, field mappings, and aggregation logic. Based on this diagram, data collection operations are performed from the central data warehouse to extract theme-related data as needed. The data collection operation extracts target data records from multiple data tables according to defined field ranges and collection conditions, generating a theme-based data set corresponding to the regulatory theme. This collection process is highly targeted and scalable, enabling data under different themes to be generated independently and with traceability.

[0034] Subsequently, based on the aforementioned collected thematic data sets, multiple independent and logically clear data thematic libraries were constructed to achieve hierarchical data management for different business areas or regulatory themes. Through field mapping relationships defined in the data lineage diagram, the collected data was organized into corresponding thematic data tables and divided into different logical sets according to themes, such as the notification and commitment information set, the simplified project information set, the regional achievement information set, and the joint application information set. Each thematic data set is stored in a cache unit after generation for quick retrieval and loading during subsequent access, while also being retained in the database table for long-term storage and historical analysis. The establishment of the data thematic libraries allows the large amount of data from the central data warehouse to be logically layered by theme. This eliminates the need to repeatedly access all the original data during data analysis; operations can be performed directly within the corresponding theme scope, effectively improving data processing efficiency. Through this process, thematic data resources corresponding to multiple business scenarios are formed, providing a structured foundation for the subsequent statistical analysis.

[0035] After each thematic library is constructed, the data items in the thematic libraries are associated and combined with pre-defined indicator libraries and algorithm models to form a data analysis engine for performing statistical analysis. The data analysis engine combines and classifies data items based on the data structure of each thematic library and the statistical dimensions defined in the indicator library, generating datasets suitable for different analytical objectives. During analysis, the corresponding datasets are selected from each thematic library, and data statistics, summarization, and correlation processing are performed according to the definitions of the indicator library to obtain results data matching the analytical objectives. Analysis results are output through charts, tables, or other forms of visualization, allowing data users to intuitively understand the data distribution and trends under different dimensions. Simultaneously, the analyzed results can be exported to various file formats for further use or archiving. Through this analysis process, data previously scattered across multiple thematic libraries is processed and displayed uniformly, thus completing a full closed-loop process from data collection to statistical analysis.

[0036] By executing the above steps sequentially, this implementation transforms multi-source heterogeneous data into a structured dataset that can be organized by topic, enabling multi-dimensional analysis and visualization. The entire process begins with data acquisition and sequentially completes data cleaning, relationship definition, on-demand data collection, topic construction, and result display. Each step is logically interconnected, forming a continuous technical chain from raw data to analytical results. This method not only reduces redundant data access through topical organization but also improves the responsiveness and flexibility of data analysis through on-demand data collection. It effectively adapts to different types of data statistics and decision analysis scenarios, providing an efficient and scalable implementation path for big data processing.

[0037] Example 3: This embodiment provides a data processing method based on big data collection. This method further improves upon Embodiments 1 and 2 to enhance the real-time performance and adaptability of data processing. Through four consecutive steps—multi-source data aggregation, on-demand data collection, topic library generation, and data analysis execution—this method achieves end-to-end data processing and dynamic analysis from raw data to statistical display. It is suitable for multi-topic, multi-dimensional big data scenarios. The specific implementation steps are as follows: The system aggregates various types of raw data from multiple source systems, and performs format standardization and structured organization on the data from different sources. Due to differences in data types, formats, timestamps, and field definitions across different sources, format conversion and field standardization are performed on the data from each data source during the aggregation phase to achieve consistent matching between data from different sources. Simultaneously, missing and outlier items are repaired and removed during the aggregation process to improve the overall integrity and usability of the data. This approach ensures that data from multiple sources is stored in the central data warehouse in a unified data structure. The central data warehouse, as the centralized carrier of multi-source data, stores the cleaned, integrated, and standardized data, providing a unified data foundation for subsequent data collection and theme construction. After data aggregation is completed, a unified data environment is formed in the system that can be accessed by different business themes, providing stable data input support for subsequent steps.

[0038] After the central data warehouse is established, collection conditions and field ranges are set according to different analytical themes through a data collection adaptation program. The key to this step is determining the required data items for each theme, along with their corresponding source tables, field mapping relationships, and data filtering logic, based on the business needs of each theme. By organizing and defining these relationships, a data lineage diagram is generated to describe the data flow and dependency paths. This diagram clarifies the reference relationships and collection order between different data items, enabling collection operations to be executed according to the preset path. Based on the generated data lineage diagram, on-demand collection operations are performed from the central data warehouse, extracting data that meets the collection conditions to form a thematic data set corresponding to the target theme. This graph-based collection method enables precise data extraction and efficient data utilization, thereby reducing unnecessary data reading and processing and improving overall collection efficiency. The collected thematic data sets logically correspond one-to-one with the regulatory themes, providing a basic data source for subsequent thematic data organization.

[0039] Subsequently, based on the aforementioned collected thematic data sets, the data is organized and stored in a thematic manner. By analyzing the logical relationships between themes and data lineage diagrams, the data related to each theme is organized and divided according to thematic dimensions, forming independent thematic data libraries. Each thematic data library revolves around a specific theme, such as regulatory information, project evaluation, and public resources, to ensure logical integrity and contextual consistency between thematic data. During the construction of the thematic libraries, the collected data is stored in cache units and database tables, enabling the thematic libraries to have both fast access and persistent storage capabilities. The design of the cache units enables rapid response to frequently accessed data, while the database tables are used for long-term data archiving and multiple analysis calls. Through this step, data from different themes is effectively layered and organized, making the entire data structure more flexible and maintainable, and allowing for rapid retrieval of relevant thematic data according to business needs. The establishment of this thematic library provides reliable data support for subsequent data analysis and visualization output.

[0040] After the data theme library is established, the data items in the theme library are associated with the preset indicator library, and the theme libraries are combined according to the analysis requirements to form a data analysis engine for performing analysis. Based on the structure of each theme library and the dimensional definitions in the indicator library, the data analysis engine combines and statistically processes data from different themes to achieve cross-theme, multi-dimensional data correlation analysis. To improve the timeliness of data updates and the real-time nature of analysis results, this implementation further introduces a timed incremental update rule to dynamically maintain the synchronization between the main data warehouse and the theme libraries. When new data is detected being written to the main data warehouse, the changed data is automatically identified according to the preset update rules, and the newly added or changed parts are updated to the corresponding theme library, thereby maintaining the timeliness and continuity of the theme library content. After completing the data update and theme integration, the data analysis engine performs statistical and analytical operations on the theme libraries, generating multi-dimensional analysis results. Finally, the statistical results are displayed in a visual chart format, and a data export function is provided to facilitate subsequent report generation or in-depth analysis. This step enables a closed-loop process for the entire data processing, from data aggregation and collection to theme construction and analysis output, giving the data processing continuous iteration and dynamic response capabilities.

[0041] By continuously executing the above steps, this implementation method, while inheriting the advantages of the aforementioned embodiments, further enhances the real-time updating and dynamic analysis capabilities of data processing. This method establishes a unified data infrastructure environment through data aggregation, achieves on-demand collection and precise extraction through data acquisition adaptation and lineage diagrams, realizes the structured and hierarchical organization of data through the establishment of a topic library, and finally completes cross-topic statistics and visualization through a data analysis engine. The entire process is logically clear, hierarchically distinct, and tightly integrated, capable of adapting to the needs of multi-topic parallel analysis in multi-source data environments, achieving an efficient, flexible, and sustainable data collection and statistical processing method, and providing a widely applicable technical implementation path for the field of big data analysis.

[0042] This invention establishes a complete data warehouse structure by introducing a unified data aggregation and cleaning process before data collection, fundamentally solving the problem of difficult integration of multi-source heterogeneous data. By performing standardization, format conversion, and anomaly removal operations on raw data from different sources and in different formats, all types of data have a unified structural definition and compatible data format before entering the data warehouse, thus avoiding the collection difficulties and analysis errors caused by inconsistent data formats in traditional systems. This technology can achieve cross-system data fusion and centralized management without changing the original data source, ensuring that the data has a consistent structural foundation and traceability in subsequent collection, statistics, and analysis stages. Compared with existing direct query modes, this invention, through pre-cleaning and centralized aggregation, makes the integration of multi-source data more flexible and efficient, reduces cross-table joins and repeated accesses in the statistical query stage, and improves overall data management efficiency and system response speed.

[0043] This invention establishes a data lineage graph through a data acquisition adaptation program and uses this graph as the logical basis for on-demand data acquisition operations, allowing for flexible configuration of the acquisition scope and granularity according to actual business needs. This on-demand acquisition mechanism avoids the traditional method of scanning the entire database in statistical analysis, significantly reducing data access volume and redundant computational burden. Simultaneously, this invention introduces a hierarchical organizational structure of "thematic data sets—data thematic libraries," managing acquired data hierarchically according to thematic logic. Data from different themes are independent yet interconnected, thus constructing a multi-theme data architecture that can be accessed in parallel. Through this organization, data access paths are significantly shortened, query efficiency is improved, and the system does not need to repeatedly read the underlying raw data when performing multi-dimensional statistics, reducing data I / O load and resource consumption. This achieves high performance and high responsiveness in data analysis, providing stable technical support for multi-theme parallel analysis in complex business environments.

[0044] This invention constructs an analysis engine adaptable to multiple dimensions and indicators by dynamically associating and combining data items required for statistical analysis with a pre-defined indicator library and algorithm model based on a data theme library. This design allows for flexible expansion of analysis dimensions and indicator content according to different business themes, enabling the rapid formation of new analysis logic without modifying the underlying data structure. This method effectively overcomes the shortcomings of traditional intermediate table statistical methods, which suffer from fixed statistical scope and limited analysis dimensions, allowing data statistical analysis to be customized and adjusted according to actual needs. Especially in multi-theme scenarios, data from multiple theme libraries can be cross-linked and fused according to indicator logic, supporting comprehensive statistics and trend analysis across domains and businesses. Through this approach, the invention not only improves the flexibility and versatility of analysis but also gives the system excellent scalability, enabling continuous expansion of analytical capabilities as business needs change, meeting the data analysis requirements of multiple scenarios and dimensions.

[0045] This invention further introduces a timed incremental update rule in the data analysis phase. By dynamically maintaining data synchronization between the main data warehouse and the subject database, the system can automatically update the subject database content when new data is detected, thus maintaining the real-time and continuous nature of the analyzed data. This design enables continuous iteration of the data processing process, ensuring that the analysis results are always based on the latest data state and avoiding the delay problems caused by traditional batch update methods. Simultaneously, this invention provides visualization and data export functions in the analysis output phase. Through charts, tables, and multiple output formats, the analysis results can be presented in an intuitive and dynamic form, facilitating managers to quickly understand data distribution, trends, and discrepancies, providing timely reference for decision-making. This process achieves a complete closed loop from data collection, organization, analysis to presentation, significantly improving the automation level and information utilization value of the data processing workflow, giving the system high timeliness, visualization, and strong decision support capabilities.

[0046] Example 4: To verify the technical effectiveness of the present invention in complex data processing scenarios, such as Figure 5As shown, this embodiment constructs a comparative simulation experiment. The experimental environment uses a high-performance server to simulate 10 million pieces of regulatory business data distributed across 5 related tables. The control group uses the traditional data warehouse direct multi-table real-time join query method, while the experimental group uses the dual-mode architecture described in this application, which automatically extracts and constructs thematic data sets based on data lineage graphs and stores the data hierarchically in Redis cache units and MySQL database tables; utilizing Apache The JMeter tool simulated 50 to 1000 concurrent users initiating "Enterprise Supervision Topic" query requests. The experimental results showed that under the high-pressure scenario of 500 concurrent users, the control group experienced an average response time exceeding 3500ms and connection timeouts due to excessive real-time computing overhead, with system CPU utilization reaching over 95%. In contrast, the experimental group, benefiting from the topic-based preprocessing organization and cache hit mechanism, achieved an average response time stable within 25ms, a system throughput increase of approximately 140 times, and a reduction of over 70% in CPU and disk I / O resource consumption. This strongly demonstrates that the technical solution of constructing independent logical topics through data lineage graphs and combining them with caching mechanisms can significantly overcome the performance bottleneck of traditional multi-table join queries, achieving millisecond-level response and high concurrency stability.

[0047] Example 5: To verify the technical effectiveness of this invention in achieving multidimensional analysis and flexible expansion through a dynamically correlated index library, such as... Figure 6 As shown, this embodiment designs a simulation experiment on the response efficiency of business indicator changes. The experiment simulates a scenario where three complex analysis dimensions are added to the existing regulatory topic library. The comparison results show that in the control group (traditional hard-coding method), adding new indicators requires two schema modifications to the underlying data table and rewriting more than 450 lines of ETL processing scripts, with a total process time of 36.5 hours. In the experimental group applying the solution described in claim 6 of this application, since the indicator library and the topic library are decoupled by parameters, technicians only need to update the parameter descriptions of statistical calibers and algorithm models on the front-end interface, with a configuration time of only 22 minutes. The system can then automatically generate new analysis engine logic through a dynamic association mechanism. Experimental data shows that the solution of this application shortens the demand response cycle from 2.5 days to less than 15 minutes without touching the underlying database structure, improving configuration efficiency by more than 30 times.

[0048] This invention provides, for example Figure 4 The data processing system based on big data collection shown includes a data aggregation and cleaning module, a data association and modeling module, a thematic data construction module, a data analysis and generation module, and a result display and export module. Data aggregation and cleaning module: Aggregates relevant data from multiple data sources, cleans the aggregated data, and inserts the cleaned data into the central data warehouse to form a unified data foundation; Data association modeling module: Based on the main data warehouse, according to the preset regulatory themes, the module configures the association relationships and collection conditions between various data sources through the data collection adaptation program, establishes a data lineage diagram, and determines the logical relationships and field ranges of the target data to be collected. Thematic data construction module: Based on the data lineage diagram and collection conditions, it performs data collection operations from the main data warehouse, generates corresponding thematic data sets, stores the collected data in the cache unit and database tables, and constructs a data thematic library corresponding to the regulatory themes; Data analysis generation module: Based on the data theme library, the data items that need to be statistically analyzed are associated and combined with the preset indicator library and algorithm model to generate a data analysis engine that can be used for statistical analysis; Results Display and Export Module: Performs data statistical analysis operations based on the data analysis engine, visualizes the analysis results, and exports the data, completing dynamic data processing and statistical analysis based on the topic library.

[0049] The data processing method based on big data acquisition provided in this embodiment of the invention is implemented through the aforementioned data processing system based on big data acquisition. For details of the specific methods and processes of the data processing system based on big data acquisition, please refer to the embodiments of the data processing method based on big data acquisition described above, which will not be repeated here.

[0050] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A data processing method based on big data collection, characterized in that, Includes the following steps: Step 1: Aggregate relevant data from multiple data sources, clean the aggregated data, and insert the cleaned data into the central data warehouse to form a unified data foundation; Step 2: Based on the main data warehouse, according to the preset regulatory themes, configure the relationship and collection conditions between various data sources through the data collection adaptation program, establish a data lineage diagram, and determine the logical relationship and field range of the target data to be collected; Step 3: Based on the data lineage diagram and collection conditions, perform data collection operations from the main data warehouse to generate corresponding thematic data sets. Store the collected data in the cache unit and database tables to build a data theme library corresponding to the regulatory theme. Step 4: Based on the data theme library, associate and combine the data items that need to be statistically analyzed with the preset indicator library and algorithm model to generate a data analysis engine that can be used for statistical analysis; Step 5: Perform data statistical analysis operations using the data analysis engine, visualize the analysis results, and export the data to complete dynamic data processing and statistical analysis based on the topic library.

2. The data processing method based on big data collection according to claim 1, characterized in that, The process of aggregating relevant data includes: when collecting raw data from multiple source systems, performing unified format conversion and structural standardization on data from different sources, in different formats, and with different time stamps; executing field type mapping and format unification operations through preset data import rules; removing duplicate information, filling in missing information, and correcting outliers in the collected raw data; after cleaning, inserting the processed data into the central data warehouse according to unified field standards. The central data warehouse is used to centrally store standardized multi-source fusion data and achieves cross-source fusion using a unified data structure definition method, so that data from different sources can be managed and accessed in a unified manner.

3. The data processing method based on big data collection according to claim 2, characterized in that, The process of establishing a data lineage diagram includes: configuring the logical relationships, field dependencies, and collection conditions between various data sources through a data acquisition and adaptation program; defining the mapping relationships and data retrieval logic between data items according to the needs of different business themes; and forming a lineage structure diagram to describe the data source path and transmission relationship.

4. The data processing method based on big data collection according to claim 3, characterized in that, The process of generating corresponding thematic datasets includes: when performing on-demand data collection, extracting data items and records related to the target theme from the main data warehouse based on field mappings and logical dependencies in the data lineage diagram; organizing and classifying the extracted data by logical theme to form datasets corresponding to each regulatory theme; allocating the datasets to cache units to support fast access, while simultaneously writing them to database tables for long-term storage and historical analysis; and ensuring that the collected data are logically independent and inherently consistent through thematic data organization, allowing cross-referencing between different thematic data through indicator logic.

5. The data processing method based on big data collection according to claim 1, characterized in that, The process of building a data theme library corresponding to regulatory themes includes: after data collection, the target data sets corresponding to each business theme are collected and structured; field mapping and aggregation are performed according to predefined field relationships and logical dependencies; and the data is stored hierarchically in cache units and database tables. Each theme library establishes logical boundaries around an independent business theme and supports independent invocation and parallel processing in subsequent analysis. The internal structure of the theme library includes two types of data: basic business fields and derived auxiliary fields, to support statistical and multidimensional analysis needs.

6. The data processing method based on big data acquisition according to claim 5, characterized in that, The process of associating and combining data items requiring statistical analysis with a pre-defined indicator library and algorithm model includes: selecting data items needed for analysis from the theme library based on the theme structure and data hierarchy defined in the theme library; matching the selected data items with the corresponding statistical dimensions and indicator definitions in the indicator library; and generating the data set to be analyzed through logical combination. The indicator library includes multi-dimensional indicator definition information, used to parametrically describe the statistical scope, aggregation method, and analytical perspective of different theme data. Through a dynamic association mechanism, the analytical dimensions and indicator content can be flexibly expanded according to actual needs, forming new analytical combination logic without modifying the structure of the theme library.

7. The data processing method based on big data acquisition according to claim 5, characterized in that, Data statistical analysis operations include: performing multi-dimensional data statistics based on the generated data analysis engine; generating corresponding statistical result sets by aggregating, classifying, and performing correlation analysis on the data in the subject library; the statistical results can be output in the form of charts, tables, or visualizations according to the dimensions defined in the indicator library, so that the differences, trends, and distribution of the data results can be presented in an intuitive way; after the analysis is completed, the result data can be exported in the form of structured data files through the export function.

8. The data processing method based on big data acquisition according to claim 7, characterized in that, After the data analysis engine is generated, the synchronization between the data subject library and the main data warehouse is periodically maintained. Dynamic updates of data are achieved by introducing timed incremental update rules. When the system detects new data being written to the main data warehouse, it automatically identifies the new or changed data according to the preset update rules and updates the changed content to the corresponding data subject library.

9. The data processing method based on big data acquisition according to claim 8, characterized in that, Dynamic data processing and statistical analysis based on subject libraries include: in multi-subject parallel scenarios, the system can call multiple data subject libraries simultaneously according to different business needs, and perform cross-subject data integration and joint statistics through the data analysis engine; multi-subject joint analysis uses indicator logic as a link to associate and match data items in different subject libraries.

10. A data processing system based on big data acquisition, used to implement the data processing method based on big data acquisition as described in any one of claims 1-9, characterized in that, It includes modules for data aggregation and cleaning, data association and modeling, thematic data construction, data analysis and generation, and results display and export. Data aggregation and cleaning module: Aggregates relevant data from multiple data sources, cleans the aggregated data, and inserts the cleaned data into the central data warehouse to form a unified data foundation; Data association modeling module: Based on the main data warehouse, according to the preset regulatory themes, the module configures the association relationships and collection conditions between various data sources through the data collection adaptation program, establishes a data lineage diagram, and determines the logical relationships and field ranges of the target data to be collected. Thematic data construction module: Based on the data lineage diagram and collection conditions, it performs data collection operations from the main data warehouse, generates corresponding thematic data sets, stores the collected data in the cache unit and database tables, and constructs a data thematic library corresponding to the regulatory themes; Data analysis generation module: Based on the data theme library, the data items that need to be statistically analyzed are associated and combined with the preset indicator library and algorithm model to generate a data analysis engine that can be used for statistical analysis; Results Display and Export Module: Performs data statistical analysis operations based on the data analysis engine, visualizes the analysis results, and exports the data, completing dynamic data processing and statistical analysis based on the topic library.

Images