Multi-message consumption mode multi-storage processing method and device

Abstract

Multi-message consumption mode multi-storage processing method and device. The present application relates to the field of data processing, and in particular to a message consumption mode multi-storage processing method, which comprises the following steps: receiving transaction messages of a device interface, and bridging the transaction messages to a kafka platform; platform log messages of the device interface are connected with the kafka platform, and the platform log messages are transmitted to the kafka platform; according to a strategy, a transaction message cleaner consumes transaction messages in the kafka platform; a transaction service cluster consumes the cleaned transaction messages issued by the transaction message cleaner, and saves the data of the transaction messages to a relational database according to business; while the platform log messages are connected with the kafka platform, a platform log cleaner consumes the platform log messages in a message router, and stores them to a hierarchical database; while the transaction message cleaner consumes the transaction messages in the kafka platform; through this multi-level storage architecture, the problem of disordered data can be effectively solved, and data can be efficiently and on-demand retrieved in the later stage.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and apparatus for processing multiple message consumption modes and multiple storage. Background Technology

[0002] With the increasing number of production devices, the original solution involved devices uploading messages to Kafka, where a dedicated message processing service consumed and stored the messages. However, with the continuous increase in transaction messages, especially during peak periods, data errors frequently occurred. This was mainly due to the low processing capacity of a single service receiving transaction messages from multiple devices, coupled with the presence of mixed business logic, leading to frequent message congestion. The message storage also utilized a common relational database, resulting in excessive data storage and inefficient business retrieval. Furthermore, with the increasing number of customers, significant time was spent manually processing abnormal data, impacting customer experience. Secondly, the growing number of customers also increased their data demands, making platform logs crucial. Handling the large volume of operation logs became a challenge, making a transformation of the message processing system imperative. Summary of the Invention

[0003] To address the issue of poor customer experience caused by processing large amounts of operation logs and massive business data in existing technologies, a multi-message consumption mode and multi-storage processing method is provided.

[0004] In a first aspect, the present invention provides a multi-message consumption mode and multi-storage processing method, the steps of which are as follows:

[0005] Receive transaction messages from the device interface and bridge the transaction messages to the Kafka platform;

[0006] The device-side interface connects with the Kafka platform to send platform log messages to the Kafka platform.

[0007] According to the strategy, the transaction message cleaner consumes transaction messages from the Kafka platform;

[0008] The transaction service cluster consumes the cleaned transaction messages issued by the transaction message cleaner, and then shards and partitions the transaction message data according to the business, saving it to a relational database.

[0009] While the platform log messages are being integrated with the Kafka platform, the platform log cleaner consumes the platform log messages in the message router and stores them in a hierarchical database.

[0010] While the transaction message cleaner consumes transaction messages from the Kafka platform, a third-party platform connects to and consumes platform log messages from the Kafka platform.

[0011] The preferred technical solution involves a transaction message cleaner consuming transaction messages from a Kafka platform, which includes processing the transaction messages from the Kafka platform into a logical data stream and the Kafka platform splitting, merging, completing, and cleaning the consumed information stream.

[0012] In the preferred technical solution, the transaction service cluster consumes transaction messages from the Kafka platform through the transaction message cleaner and stores them in a relational database according to business-specific tables.

[0013] The preferred technical solution involves integrating the service consumption information in the transaction message with Kafka to perform data transformation, and then saving the message to a relational database for later retrieval.

[0014] The preferred technical solution is that the data warehouse service stores the transaction messages consumed by the transaction message cleaner from the Kafka platform into a hierarchical database according to aggregation and hierarchical rules.

[0015] In the preferred technical solution, the platform log cleaner consumes platform log messages, and the steps are as follows:

[0016] Define a launcher;

[0017] Start several tasks, each task can process multiple types of messages, and each message can specify a logging policy;

[0018] Based on different log strategies, messages are cleaned and stratified, and then written to a stratified data warehouse.

[0019] The preferred technical solution is that the log strategy includes describing the message cleaning steps, cleaning principles, calling the corresponding service to obtain data completion for the cleaned data, different message definitions, describing the message cleaning process, and saving it to the corresponding database.

[0020] In the preferred technical solution, the data warehouse server consumes the cleaned transaction messages issued by the transaction message cleaner, aggregates the data of the transaction messages according to dimensions, and stores the transaction messages in a hierarchical data warehouse.

[0021] The preferred technical solution involves collecting platform log messages and uploading them to Kafka, supplementing key attributes, cleaning the data, and then writing the logs to a layered data warehouse.

[0022] Firstly, this solution adopts a multi-consumption model, defining multiple consumers to consume different messages, thus solving the message congestion problem. It also defines multiple message cleaners to clean and layer the data when receiving messages. At the same time, it defines a multi-layered data structure according to business needs, saving the cleaned data to the corresponding layers, achieving efficient storage and efficient data retrieval. Secondly, platform log messages are processed using a dedicated server, isolating the impact of platform logs on business operations.

[0023] Secondly, the present invention provides a multi-message consumption mode multi-storage processing device, comprising:

[0024] Transaction equipment: Equipment for uploading transaction information;

[0025] Message Router: Receives transaction messages from device interfaces and EMQ transaction messages, and bridges the messages to the platform log cleaner and Kafka platform;

[0026] EMQ: Maintains a connection with the trading device and is used to upload trading messages;

[0027] Kafka platform: used to transmit transaction messages and platform log messages;

[0028] Transaction message cleaner: Collects transaction data, calls internal basic data services to complete and clean the data, and distributes messages according to business needs;

[0029] Transaction service cluster: collects transaction data and stores it in a relational database;

[0030] Data warehouse services: data aggregation and tiered storage;

[0031] Platform log collection server: records some key operations of the platform;

[0032] The platform's log message processing service cleans log data using a log cleaner.

[0033] Layered data warehouse: a multi-layered structure, divided according to the level of data cleaning;

[0034] Relational database: It is divided into databases and tables according to business functions to store the main transaction data.

[0035] The beneficial effects achieved by this invention are as follows: It solves several key challenges: First, the uneven distribution of transaction messages generated by devices prevents message backlog during peak periods through concurrent consumption by multiple consumers. Second, the large volume of transaction message data poses a storage challenge; this multi-layered storage architecture effectively addresses data clutter, enabling efficient, on-demand data retrieval. Third, multiple message cleaners are defined to handle data cleaning, writing data to the aforementioned multi-layered storage database based on different log strategies. Fourth, it isolates platform log data, dedicating a specific service to third-party integration, reducing platform issues caused by external factors. Attached Figure Description

[0036] Figure 1 This is a logical schematic diagram of the multi-message consumption mode and multi-storage processing method of the present invention. Detailed Implementation

[0037] To facilitate understanding of the present invention by those skilled in the art, specific embodiments of the present invention will be described below with reference to the accompanying drawings.

[0038] like Figure 1 As shown, this invention provides a multi-message consumption mode multi-storage processing method, the steps of which are as follows:

[0039] Receive transaction messages from the device interface and bridge the transaction messages to the Kafka platform;

[0040] The device-side interface connects with the Kafka platform to send platform log messages to the Kafka platform.

[0041] According to the strategy, the transaction message cleaner consumes transaction messages from the Kafka platform;

[0042] The transaction service cluster consumes the cleaned transaction messages issued by the transaction message cleaner, and then shards and partitions the transaction message data according to the business, saving it to a relational database.

[0043] While the platform log messages are being integrated with the Kafka platform, the platform log cleaner consumes the platform log messages in the message router and stores them in a hierarchical database.

[0044] While the transaction message cleaner consumes transaction messages from the Kafka platform, a third-party platform connects to and consumes platform log messages from the Kafka platform.

[0045] It should be noted that: the platform log messages of the device interface are connected to the Kafka platform, and the platform log messages are passed to the Kafka platform; when the platform log collection server collects platform log messages and uploads them to Kafka, it is necessary to complete key attributes and perform data cleaning; key attributes refer to some attributes used by the backend system for page queries, which are often stored in multiple different places in the database design. The messages uploaded by the device do not have these attributes, so they need to be completed. Data cleaning refers to the transformation of messages, which can be deletion or aggregation.

[0046] The original solution involved devices uploading messages to Kafka, where a dedicated message processing service consumed and stored the messages. However, with the increasing number of transaction messages, especially during peak periods, data errors frequently occurred. This was mainly due to the low processing capacity of a single service receiving transaction messages from multiple devices, coupled with the presence of business logic, which often caused message congestion. Furthermore, the message storage used a common relational database, resulting in excessive data storage and very low efficiency when performing business retrievals. As the number of customers continued to increase, a significant amount of time was often spent manually processing abnormal data, impacting the customer experience.

[0047] For the reasons mentioned above, the messages received by the device are now divided into log messages and business transaction messages. Transaction messages are bridged to the Kafka platform for processing, while log messages are processed by the platform's log cleaner. This separates the processing of log messages and transaction messages, enabling non-blocking data processing on the existing equipment during peak message processing periods. This allows relevant business personnel to quickly retrieve relevant information when using business-related transaction messages.

[0048] To enable rapid retrieval and use of log messages and business transactions, log messages are stored in a hierarchical database, while business transaction messages are stored in a relational database. When administrators or business personnel need to access log messages or business transaction messages, the server selects and calls either the hierarchical database or the relational database based on the requirements, thereby shortening data retrieval time and reducing system load.

[0049] Secondly, as the number of customers continues to increase, their demand for data also increases, making platform logs very important. Therefore, in addition to associating or storing business transaction messages, the Kafka platform also needs to store log messages to meet customers' data needs.

[0050] Because Kafka platforms contain not only transaction messages but also log messages, this reduces the platform's capacity for data storage, retrieval, and processing. Therefore, it's necessary to consume and process the data within the Kafka platform. The question then becomes: how can we better and more effectively consume and process the data from the Kafka platform?

[0051] First, a strategy needs to be specified, and the data will be consumed and processed according to the corresponding strategy.

[0052] The strategies presented here include: first, a strategy for processing log messages; second, a strategy for processing business transaction messages; and third, a strategy for processing both log messages and business transaction messages. For example, the first strategy involves determining whether a message is a log message based on its definition. If it is a log message, it is processed according to predefined consumption rules (description of log cleaning rules, log cleaning steps, which services to call for data completion, how to save the data to the database, and whether to save it to a relational database or a hierarchical data warehouse). The second and third strategies have the same functionality as the first strategy.

[0053] Secondly, according to the corresponding strategy, the transaction message cleaner consumes and processes the data in the Kafka platform; for example, splitting, merging, completing, and cleaning; the transaction message cleaner then transmits the consumed and processed data stream to the Kafka platform.

[0054] It should be noted that: Cleaning: Based on the message topic, a defined strategy is found, and data with missing key attributes is removed according to the strategy; Completion: The microservice calls the basic data service to query the necessary business attributes and assemble the message; Splitting: Similar messages are converted into multiple sub-messages; Merging: Multiple types of messages are merged into one type of message.

[0055] Depending on the strategy (multiple consumption methods), when different types of material requisition messages are received, the messages are merged. For similar messages, such as work order material requisitions and ordinary machine material requisition messages, business-related splitting is performed. Missing business fields in the messages are completed, cleaned, and filled with default values ​​through internal basic service calls. The original messages and cleaned messages are then passed down through Kafka.

[0056] Then, the transaction service cluster stores the data in a relational database by partitioning the data according to the information from the Kafka platform; the service consumption information in the transaction message is integrated with Kafka to perform data transformation, and the message is saved in the relational database for later retrieval; the later retrieval is performed using SQL.

[0057] This data is transformed into, for example, transaction messages are converted into inventory, material requisition, and replenishment messages, enabling database sharding and table partitioning for easier management, smaller data granularity, and higher query performance.

[0058] Finally, the data warehouse service aggregates, layers, and stores the information from the Kafka platform into a layered database.

[0059] To address the issue of low processing capacity when a single service receives transaction messages from multiple devices, coupled with the presence of mixed business logic that frequently causes message blocking, the transaction message cleaner needs to process these messages into a logical data stream. This ensures data consistency in subsequent data processing, invocation, and merging processes on the Kafka platform, improving transaction message processing capabilities and preventing message blocking.

[0060] The platform log cleaner consumes platform log messages in the following steps:

[0061] Define a launcher;

[0062] Start several tasks, each task can process multiple types of messages, and each message can specify a logging policy;

[0063] Based on different logging strategies, messages are cleaned and stratified, and then written to a tiered data warehouse. It's important to note that the logging strategy, based on different message definitions, describes how messages are cleaned, the cleaning steps, how services are invoked to retrieve data for completion, and how they are saved to the tiered data warehouse.

[0064] While the transaction message cleaner consumes transaction messages from the Kafka platform, a third-party platform connects to and consumes platform log messages from the Kafka platform.

[0065] Firstly, this solution adopts a multi-consumption model, defining multiple consumers to consume different messages, thus solving the message congestion problem. It also defines multiple message cleaners to clean and layer the data when receiving messages. At the same time, it defines a multi-layered data structure according to business needs, saving the cleaned data to the corresponding layers, achieving efficient storage and efficient data retrieval. Secondly, platform log messages are processed using a dedicated server, isolating the impact of platform logs on business operations.

[0066] Secondly, the present invention provides a multi-message consumption mode multi-storage processing device, comprising:

[0067] Transaction equipment: Equipment for uploading transaction information;

[0068] Message Router: Receives transaction messages from device interfaces and EMQ transaction messages, and bridges the messages to the platform log cleaner and Kafka platform;

[0069] EMQ: Maintains a connection with the transaction device to upload transaction messages, receive messages from the device, and transmit the corresponding messages to the platform log cleaner and Kafka platform.

[0070] It should be noted that EMQ is an open-source MQTT message server based on the high-concurrency Erlang / OTP language platform, supporting millions of connections and distributed clusters, and using a publish-subscribe model.

[0071] Kafka platform: used for delivering transaction messages and platform log messages; Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action stream data from consumers on a website.

[0072] Transaction message cleaner: Collects transaction data, calls internal basic data services to complete and clean the data, and distributes messages according to business needs;

[0073] Transaction service cluster: Composed of multiple services (such as replenishment, inventory, and material requisition), it receives cleaned transaction data, shards and tables according to business functions, and stores the data in different relational databases. Its main function is to collect transaction data and store it in relational databases. Each service consumes cleaned data from the transaction message cleaner, shards and tables according to different business functions, and saves the data to a relational database. For example, the replenishment service consumes replenishment messages, and the inventory service consumes inventory messages.

[0074] Data warehouse: A multi-layered structure, divided according to the levels of data cleaning, with excellent message processing capabilities. Databases are designed for transactions; data warehouses are designed for topics. Databases typically store online transaction data; data warehouses typically store historical data. Databases are designed to capture data, while data warehouses are designed to analyze data. The basic element of a database is the fact table, while the basic element of a data warehouse is the dimension table. Data warehouses are designed for data analysis and decision-making, serving senior management decision-makers within enterprises.

[0075] Data warehouse service: Consumes transaction messages from the transaction message cleaner, aggregates data according to different dimensions, and stores transaction messages in layers; data aggregation and layered storage;

[0076] Platform log collection server: Records some key operations of the platform, such as login logs, API sample logs, data modification records, etc.

[0077] The platform log cleaner is a platform log message processing service. It prevents excessive residual logs from wasting disk space and impacting service performance. It is responsible for consuming platform log messages. The process is as follows: a startup engine is defined, and multiple tasks are started. Each task can process various types of messages, such as storing raw messages in the hierarchical data warehouse ODS data layer, specifying a strategy for each message, and performing cleaning, completion, and transformation on the data according to different strategies. Similar messages are merged, and the cleaned data is stored in the hierarchical data warehouse DWD data layer.

[0078] Layered data warehouse: a multi-layered structure, divided according to the level of data cleaning; the level of data cleaning is determined by business, and the original data is continuously transformed according to business rules. For example, the original data is one layer, and the data after cleaning dirty data is another layer. For example, transaction details are one layer, and the data is summarized and saved to another layer according to transaction type.

[0079] Relational database: It is divided into databases and tables according to business needs, stores popular transaction data, and has excellent data retrieval capabilities for later report queries.

[0080] Message cleaner: Collects transaction data, calls internal basic data services to complete and clean the data, and distributes messages according to business needs; simple configuration can handle multiple different messages simultaneously, and new message consumption can be added by adding strategies. Compared with commonly used logical message processing, no logic needs to be modified, reducing modification costs and not affecting existing messages.

[0081] This approach saves service resources by enabling layered data storage with minimal overhead through multi-tasking within the same service. Compared to processing a single message with a single service, this reduces server costs. Furthermore, the multiple tasks operate independently, preventing service unavailability due to message anomalies. The finer granularity of the data allows for high-performance retrieval in later business operations.

[0082] It achieves isolation between platform logs and business data, preventing them from affecting each other and ensuring that an anomaly in one database will not impact other business operations, thus improving system stability.

[0083] The beneficial effects achieved by this invention are as follows: It solves several key challenges: First, the uneven distribution of transaction messages generated by devices prevents message backlog during peak periods through concurrent consumption by multiple consumers. Second, the large volume of transaction message data poses a storage challenge; this multi-layered storage architecture effectively addresses data clutter, enabling efficient, on-demand data retrieval. Third, multiple message cleaners are defined to handle data cleaning, writing data to the aforementioned multi-layered storage database based on different log strategies. Fourth, it isolates platform log data, dedicating a specific service to third-party integration, reducing platform issues caused by external factors.

[0084] The embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A multi-message consumption mode and multi-storage processing method, characterized in that, The steps are as follows: Receive transaction messages from the device interface and bridge the transaction messages to the Kafka platform; The device-side interface connects with the Kafka platform to send platform log messages to the Kafka platform. According to the strategy, the transaction message cleaner consumes transaction messages from the Kafka platform; The transaction service cluster consumes the cleaned transaction messages issued by the transaction message cleaner, and then shards and partitions the transaction message data according to the business, saving it to a relational database. While the platform log messages are being integrated with the Kafka platform, the platform log cleaner consumes the platform log messages in the message router and stores them in a hierarchical database. While the transaction message cleaner consumes transaction messages from the Kafka platform, a third-party platform connects to and consumes platform log messages from the Kafka platform.

2. The multi-message consumption mode multi-storage processing method according to claim 1, characterized in that, The transaction message cleaner consumes transaction messages from the Kafka platform, including processing the transaction messages into logical data streams and splitting, merging, completing, and cleaning the consumed information streams on the Kafka platform.

3. The multi-message consumption mode multi-storage processing method according to claim 2, characterized in that, The transaction service cluster consumes transaction messages from the Kafka platform through the transaction message cleaner and stores them in a relational database according to business-specific tables.

4. The multi-message consumption mode multi-storage processing method according to claim 3, characterized in that, The service consumption information in the transaction message is integrated with Kafka to perform data transformation, and the message is saved to a relational database for later retrieval.

5. The multi-message consumption mode multi-storage processing method according to claim 3, characterized in that, The data warehouse service consumes transaction messages from the Kafka platform through the transaction message cleaner and stores them in a hierarchical database according to aggregation and hierarchical rules.

6. The multi-message consumption mode multi-storage processing method according to claim 1, characterized in that, The platform log cleaner consumes platform log messages in the following steps: Define a launcher; Start several tasks, each task can process multiple types of messages, and each message can specify a logging policy; Based on different log strategies, messages are cleaned and stratified, and then written to a stratified data warehouse.

7. The multi-message consumption mode multi-storage processing method according to claim 6, characterized in that, The log strategy includes describing the message cleaning steps, cleaning principles, calling the corresponding service to obtain data completion for cleaned data, different message definitions, describing the message cleaning process, and saving it to the corresponding database.

8. The multi-message consumption mode multi-storage processing method according to claim 1, characterized in that, The data warehouse server consumes the cleaned transaction messages issued by the transaction message cleaner, aggregates the data of the transaction messages according to dimensions, and stores the transaction messages in a hierarchical data warehouse.

9. The multi-message consumption mode multi-storage processing method according to claim 1, characterized in that, Collect platform log messages and upload them to Kafka, complete key attributes, perform data cleaning, and then write the logs to a layered data warehouse.

10. A multi-message consumption mode multi-storage processing device, characterized in that, The method for processing multiple message consumption modes and multiple storage modes according to any one of claims 1-9 includes: Transaction equipment: Equipment for uploading transaction information; Message Router: Receives transaction messages from device interfaces and EMQ transaction messages, and bridges the messages to the platform log cleaner and Kafka platform; EMQ: Maintains a connection with the trading device and is used to upload trading messages; Kafka platform: used to transmit transaction messages and platform log messages; Transaction message cleaner: Collects transaction data, calls internal basic data services to complete and clean the data, and distributes messages according to business needs; Transaction service cluster: collects transaction data and stores it in a relational database; Data warehouse services: data aggregation and tiered storage; Platform log collection server: records some key operations of the platform; The platform log message processing service cleans log data using a log cleaner. Layered data warehouse: a multi-layered structure, divided according to the level of data cleaning; Relational database: It is divided into databases and tables according to business functions to store the main transaction data.

Images