Data aggregation system and method based on message middleware

Abstract

The invention belongs to the technical field of data processing, and particularly discloses a data aggregation system based on message-oriented middleware, which comprises a data access module, a data reading module, a data processing module, a data writing module and a global monitoring module, the data reading module is used for reading data from different message queues in batches, the data processing module is used for decrypting, verifying, sorting, de-duplicating and aggregating the read data, the data writing module is used for generating database writing statements, and the global monitoring module is used for recording data processing results and triggering an exception handling mechanism. In order to solve the problem of difficulty in integration of current multi-source data, data from different sources are uniformly converged to message middleware, so that a data island is broken, and the problems that a traditional data convergence method can only face a single service scene and does not reserve a universal interface, so that interfaces of all service systems are incompatible, and the data convergence efficiency is high are solved. Therefore, the problem of difficult data integration is solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a data aggregation system and method based on message middleware. Background Technology

[0002] With the rapid development of information technology, various industries have gradually built diverse information systems to meet the diverse needs of different business departments. These systems operate independently, with data from different systems and data sources isolated, lacking effective communication and collaboration mechanisms. For example, in a large enterprise, the sales department's Customer Relationship Management (CRM) system records basic customer information, purchase records, and sales data; while the finance department's financial management system focuses on processing data such as financial transactions, account details, and financial statements. Because these two systems lack a bridge for data sharing and interaction, sales data cannot be fed back to the financial system in a timely and accurate manner, resulting in the finance department lacking comprehensive and accurate sales data support for financial analysis and decision-making. Similarly, cost data and profit data in the financial system are also of significant reference value for the sales department in formulating sales strategies and evaluating sales performance, but due to the existence of data silos, the sales department struggles to access this crucial data.

[0003] Besides the problem of data silos, traditional data aggregation methods also suffer from data quality issues, such as missing, incorrect, and duplicate data, which severely impact data usability and value. In enterprise databases, due to negligence by data entry personnel or malfunctions in data collection equipment, some data may be missing or entered incorrectly, which will affect the company's evaluation of a business unit and the formulation of market strategies. Duplicate data also negatively impacts data aggregation and subsequent analysis. Storing the same data repeatedly not only wastes a lot of storage space but also leads to data inconsistency problems.

[0004] Furthermore, incompatibility between interfaces of various business systems makes data integration difficult, hindering comprehensive and accurate support for decision-making. This results in a lack of data-driven decision-making processes, severely impeding the company's data analysis and decision-making capabilities. The company is unable to conduct comprehensive data analysis from a holistic perspective, limiting itself to partial analysis of data within individual departments. This leads to biased and inaccurate analysis results, failing to provide strong support for strategic decision-making. Moreover, the problem of data silos increases the company's data management costs and maintenance difficulties. Different systems use different data formats, storage methods, and data standards, making data integration and unified management challenging.

[0005] Therefore, it is necessary to design a data aggregation system and method based on message middleware to solve the problem that traditional data aggregation methods can only be applied to a single business scenario and do not reserve universal interfaces, resulting in incompatibility between interfaces of various business systems and thus making data integration difficult. Summary of the Invention

[0006] The purpose of this invention is to provide a data aggregation system and method based on message middleware, so as to solve the problem that traditional data aggregation methods can only be applied to a single business scenario and do not reserve a universal interface, resulting in incompatibility between interfaces of various business systems and thus making data integration difficult.

[0007] To achieve the above objectives, the basic solution provided by this invention is as follows: a data aggregation system based on message middleware, comprising a data access module, a data reading module, a data processing module, a data writing module, and a global monitoring module. The data access module receives data from HTTP protocols and message queues. The data reading module reads data in batches from different message queues and maps the data to entity class objects using configuration files and Java reflection technology. The data processing module decrypts, verifies, sorts, deduplicates, and aggregates the read data. The data writing module dynamically generates database write statements based on annotations in the entity classes. The global monitoring module records the processing results of the batch of data upon successful or abnormal data writing and triggers a global exception handling mechanism.

[0008] The beneficial effects of the present invention are as follows: (1) The present invention unifies the data from different sources such as IoT sensors, business system interfaces, and files into the message middleware to form standardized data, breaking the data silos and solving the problems of complex access to multi-source data and protocol incompatibility for enterprises; and uses configuration files and Java reflection technology to dynamically map data to entity class objects, realizing automatic field conversion and format unification, greatly reducing manual coding, and using a data sorting mechanism to identify qualified and unqualified data, ensuring data quality in subsequent processing; (2) Through the layered design of data reading, data processing and data writing interfaces, it supports the addition of message middleware and databases without rebuilding the core process, greatly improving the scalability of the system. The data processing chain also supports plug-in development, and can customize business rules such as decryption, verification, and aggregation, quickly responding to changes in business needs. The modular design also improves the maintainability of the system; (3) Through the global monitoring mechanism, batch operation and exception recovery strategy, a stable and reliable data processing link is constructed. The global monitoring module based on AOP can record the data processing batch ID, processing time and data processing volume in real time, enhancing the observability of the system.

[0009] Option 2, a data aggregation method based on message middleware, includes the following steps: S1: Data reception and forwarding: First, the HTTP server and MQ message queue are started through the data reception module to receive raw data and parse the data. Then, the parsed data is preliminarily checked. S2: Data Reading: Then, the data reading module reads data from the MQ message queue and defines data mapping rules through the configuration file. Then, Java reflection technology is used to convert the data into entity class objects and send them to the data processing module. S3: Data Processing: The encrypted entity objects are first decrypted by the data processing module, then the integrity of the data and the legality of the business rules are checked, and finally duplicate data is removed and aggregated. S4: Data writing: Then, the data writing module generates database SQL statements based on the entity class annotations of the data, and finally uses the database connection pool to execute the SQL statements to store the aggregated data into the database in batches. By leveraging configuration files and Java reflection technology, manual coding is reduced, improving development efficiency and system adaptability. At the same time, it can cover multi-source data access, batch data processing, and batch writing, meeting data aggregation needs.

[0010] Option 3, which is the preferred option of Option 2, involves using S1 to obtain raw data from interface requests, files, message queues, and third-party systems.

[0011] Option 4, which is the preferred option of Option 2, involves parsing the data from the HTTP request in S1 and then authenticating it. Once the authentication is successful, the data is sent to the MQ message queue through the message middleware, and a source identifier and timestamp are added to the data during the sending process. Authenticating the data from the HTTP request ensures that the data source is trustworthy.

[0012] Option 5, which is the preferred option of Option 2, involves preliminary data checks in S1, including checks on data format, required fields, and field types. Abnormal or incomplete data is marked or discarded after the checks. Checking data format, required fields, and field types before reading data can prevent invalid data from entering subsequent processes, thereby ensuring system stability and data reliability.

[0013] Option 6, which is an alternative to Option 2, involves sending entity objects in batches by configuring the quantity or time in S2; batch processing improves data processing efficiency.

[0014] Option 7, an optimal choice from Option 2, involves sorting the data in S3 after checking its integrity. The data is divided into qualified and unqualified data. Qualified data undergoes deduplication, while unqualified data is returned to the corresponding topic via message middleware. The data sender subscribes to this topic to obtain the unqualified data, modifies it, and resubmits it. This separation of qualified and unqualified data ensures data quality in the core data aggregation process and allows unqualified data to be returned to the source, supporting data correction and reprocessing, thus improving the system's fault tolerance.

[0015] Option 8, an optimal choice of Option 2, involves the following steps in S4: After successful data storage, the data writing module records the batch ID, processing time, and data volume, and synchronizes this information to the global monitoring module. If an interruption occurs during data writing, the interrupted data is captured by the global monitoring module's AOP and stored in a log file, along with the processing status of these files. Once the data writing module recovers, the data in the log file is rewritten to the database, and the processing status is updated in the log file. Through batch recording and AOP exception capture, the traceability of data processing status is ensured. Furthermore, by storing interrupted data in the log file and retrying the writing process after system recovery, eventual data consistency is guaranteed. Attached Figure Description

[0016] Figure 1 This is a flowchart of a data aggregation method based on message middleware according to the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below through specific embodiments: Example A data aggregation system based on message middleware includes a data access module, a data reading module, a data processing module, a data writing module, and a global monitoring module. The data access module receives data via HTTP protocol and message queues. The data reading module reads data in batches from different message queues and maps the data to entity class objects using configuration files and Java reflection. The data processing module decrypts, verifies, sorts, deduplicates, and aggregates the read data. The data writing module dynamically generates database write statements based on annotations in entity classes, supporting batch writing of data to different types of databases. The global monitoring module records the processing results of the batch of data upon successful or abnormal data writing and triggers a global exception handling mechanism.

[0018] Based on the aforementioned data aggregation system, a data aggregation method based on message middleware is provided, such as... Figure 1As shown, the specific steps include: S1: Data Reception and Forwarding: First, the data reception module starts the HTTP server and MQ message queue to receive raw data from interface requests, files, message queues, and third-party systems. When an HTTP request arrives, the HTTP server first reads the JSON formatted data from the HTTP request and converts it into a format that the MQ message queue can process. Then, it verifies the data. If the verification is successful, the data is sent to the MQ message queue through the message middleware, and a source identifier and timestamp are added to the data during the sending process. When an MQ message arrives, it is parsed into JSON data. Then, the data in the message queue is initially checked, including data format, required fields, and field types. Finally, abnormal or incomplete data is marked or discarded. S2: Data Reading: The data reading module then reads data from the MQ message queue and defines data mapping rules through the configuration file. For example, the "temp" field in the JSON data is mapped to the temperature attribute of the entity class. Then, Java reflection technology is used to convert the message data into entity class objects. The entity class objects are then sent to the data processing module in batches according to quantity or time. For example, all data is configured to be sent every 100 messages. When there are fewer than 100 messages, the time of the last send is used as the reference, and the remaining data is sent after 5 minutes. S3: Data Processing: The encrypted data is then decrypted through the data processing module. Next, the integrity of the data and the legality of the business rules are checked. The checked data is then divided into qualified data and unqualified data. Unqualified data is returned to the corresponding topic through the message middleware. The data sender obtains the unqualified data by subscribing to the topic, modifies the unqualified data, and resubmits it. Qualified data is deduplicated and then summarized, statistically analyzed, and combined according to the rules. S4: Data Writing: The data writing module then generates database SQL statements based on the entity class annotations of the data. Finally, the database connection pool is used to execute the SQL statements to batch store the summarized data into the database. After successful data storage, the data writing module records the batch ID, processing time, and data volume of the data and synchronizes this information to the global listening module. If an abnormal interruption occurs during the data writing process, the global listening module captures the interrupted data through AOP and stores this data in the log file while recording the processing status of these files. When the data writing module recovers, the data in the log file is rewritten into the database, and the data processing status in the log file is updated.

[0019] In summary, by constructing a method based on message middleware for multi-protocol data access, a three-layer abstract interface, and a global listening mechanism, not only is efficient aggregation and standardized processing of multi-source heterogeneous data achieved, but also dynamic mapping of data to entity class objects is realized using configuration files and Java reflection technology, reducing manual coding. Furthermore, the unified sorting and batch processing mechanisms can classify data into qualified and unqualified categories and support batch data writing, improving data processing efficiency and facilitating use by business systems. The global listening mechanism can uniformly record the processing results of all batches of data, providing global anomaly capture and hot value recording, enhancing system stability and observability, and effectively solving the problem of difficult multi-source data integration.

[0020] The above descriptions are merely embodiments of the present invention, and common knowledge regarding specific structures and characteristics is not elaborated upon here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the structure of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A data aggregation system based on message middleware, characterized in that, It includes a data access module, a data reading module, a data processing module, a data writing module, and a global monitoring module. The data access module is used to receive data from HTTP protocol and message queues. The data reading module is used to read data in batches from different message queues and map the data to entity class objects using configuration files and Java reflection technology. The data processing module is used to decrypt, verify, sort, deduplicate, and aggregate the read data. The data writing module is used to dynamically generate database write statements based on the annotations of entity classes. The global monitoring module is used to uniformly record the processing results of the batch of data when the data is written successfully or an exception occurs, and trigger the global exception handling mechanism.

2. A data aggregation method based on message middleware, characterized in that, Includes the following steps: S1: Data reception and forwarding: First, the HTTP server and MQ message queue are started through the data reception module to receive raw data and parse the data. Then, the parsed data is preliminarily checked. S2: Data Reading: Then, the data reading module reads data from the MQ message queue and defines data mapping rules through the configuration file. Then, Java reflection technology is used to convert the data into entity class objects and send them to the data processing module. S3: Data Processing: The encrypted entity objects are first decrypted by the data processing module, then the integrity of the data and the legality of the business rules are checked, and finally duplicate data is removed and aggregated. S4: Data Writing: Then, the data writing module generates database SQL statements based on the entity class annotations of the data. Finally, the database connection pool is used to execute the SQL statements to store the summarized data into the database in batches.

3. The data aggregation method based on message middleware according to claim 2, characterized in that, In S1, raw data comes from API requests, files, message queues, and third-party systems.

4. The data aggregation method based on message middleware according to claim 2, characterized in that, In S1, after the data from the HTTP request is parsed, it needs to be authenticated. Once the authentication is successful, the data is sent to the MQ message queue through the message middleware, and a source identifier and timestamp are added to the data during the sending process.

5. A data aggregation method based on message middleware according to claim 2, characterized in that, In S1, the initial data check includes checking the data format, required fields, and field types. Abnormal or incomplete data found after the check is marked or discarded.

6. A data aggregation method based on message middleware according to claim 2, characterized in that, In S2, entity objects are sent in batches based on the configured quantity or time.

7. A data aggregation method based on message middleware according to claim 2, characterized in that, In S3, after checking data integrity, the data needs to be sorted into qualified and unqualified data. Qualified data is then deduplicated, while unqualified data is returned to the corresponding topic through the message middleware. The data sender obtains the unqualified data by subscribing to the topic, then modifies the unqualified data and resubmits it.

8. A data aggregation method based on message middleware according to claim 2, characterized in that, In S4, after data is successfully stored, the data writing module records the batch ID, processing time, and data volume of the data and synchronizes this information to the global listening module. If an abnormal interruption occurs during the data writing process, the AOP of the global listening module captures the interrupted data and stores it in the log file, while also recording the processing status of these files. After the data writing module recovers, the data in the log file is rewritten to the database, and the processing status of the data in the log file is updated.

Images