Data routing method and apparatus, terminal device, and storage medium

Abstract

The application discloses a data routing method and device, a terminal equipment and a storage medium, and the method comprises the following steps: receiving to-be-processed data of a source system through a message access layer of a message middleware; processing the to-be-processed data through a pre-configured message processing cluster configuration template in the message middleware; and distributing the processed data to a corresponding target system through the message push layer according to a preset distribution rule. The application effectively solves the problem of data connection of heterogeneous systems based on the distributed message middleware and the template configuration, realizes decoupling between systems or within a system, avoids mutual influence between upstream and downstream systems, improves the system availability, and reduces the data loss risk.

Description

Technical Field

[0001] This application relates to the field of data routing technology, and in particular to a template-based data routing method, apparatus, terminal device, and storage medium. Background Technology

[0002] As business systems evolve, enterprises design, develop, and apply different business subsystems according to varying business needs. This leads to differences between business systems and data discrepancies between upstream and downstream systems. Business development and changes necessitate the sharing and integration of data between these systems.

[0003] However, the data interaction between upstream and downstream subsystems is becoming increasingly complex and manual maintenance is becoming increasingly difficult. In actual production environments, there are problems such as difficulty in data interaction between heterogeneous systems, mutual coupling and influence between upstream and downstream systems, and difficulty in data reuse. Summary of the Invention

[0004] The main purpose of this application is to provide a data routing method, apparatus, terminal device, and storage medium, which aims to decouple upstream and downstream systems, avoid mutual interference between upstream and downstream systems, and improve system availability.

[0005] To achieve the above objectives, this application provides a data routing method, the data routing method comprising:

[0006] The message access layer receives data to be processed from the source system.

[0007] The data to be processed is processed using the message processing cluster configuration template in the pre-configured message middleware.

[0008] According to the preset distribution rules, the processed data is distributed to the corresponding target system through the message push layer.

[0009] Optionally, after the step of receiving the data to be processed from the source system through the message access layer, the method further includes:

[0010] The correctness of the data source is determined by the data source rules of the cluster configuration template in the message middleware.

[0011] If the data source is correct, proceed to the following steps: process the data to be processed using the pre-configured message processing cluster configuration template in the message middleware.

[0012] Optionally, before the step of processing the data to be processed through the cluster configuration template in the message middleware, the method further includes:

[0013] Configure the message processing cluster configuration template in the message middleware.

[0014] Optionally, the step of configuring the message processing cluster configuration template in the message middleware includes:

[0015] The template engine is obtained through the front-end configuration platform of the message processing platform;

[0016] Based on the template engine, a preset configuration rule is invoked to generate a message processing cluster configuration template;

[0017] The message processing cluster configuration template is published and written to the configuration library so that the message processing cluster can subscribe to or query the corresponding message processing cluster configuration template.

[0018] Optionally, the configuration rules include one or more of the following: data source configuration, filtering rules, deduplication rules, content conversion rules, message field rules, rate limiting configuration, analysis and statistics, comparison and statistics, monitoring and display, data storage, batch retries, and retry configuration refresh information.

[0019] Optionally, the step of processing the data to be processed using a pre-configured message processing cluster configuration template in the message middleware includes:

[0020] The system calls the message processing cluster configuration template configured by the front-end configuration platform through the corresponding message processing cluster, performs hot compilation of the rules of the message processing cluster configuration template, and processes the data to be processed based on the hot-compiled message processing cluster configuration template.

[0021] Optionally, the step of processing the data to be processed based on the hot-compiled message processing cluster configuration template includes:

[0022] If the message processing cluster is an access cluster, then the configuration template of the front-end configuration platform is called to filter the data, convert the content, and select message fields.

[0023] If the message processing cluster is a sending cluster, then the configuration template of the front-end configuration platform is invoked to perform deduplication and rate limiting on the data;

[0024] If the message processing cluster is a monitoring cluster, then the configuration template of the front-end configuration platform is invoked to perform data analysis, statistics comparison, and monitoring display processing.

[0025] If the message processing cluster is an exception handling cluster, the configuration template of the front-end configuration platform is invoked to store the data, perform batch retries, and refresh the retry configuration.

[0026] Optionally, before the step of distributing the processed data to the corresponding target system through the message push layer according to the preset distribution rules, the method further includes:

[0027] The deduplication rule configuration template of the front-end configuration platform is invoked to deduplicat the data distributed by the message push layer to the corresponding target system;

[0028] The rate limiting configuration template of the front-end configuration platform is invoked to limit the data rate distributed by the message push layer to the corresponding target system.

[0029] Optionally, after the step of distributing the processed data to the corresponding target system through the message push layer according to the preset distribution rules, the method further includes:

[0030] The preset error code handling mechanism is invoked, and when a data transmission error occurs, the target system returns a configured special error code.

[0031] The data is resent based on the configured special error code.

[0032] Optionally, the step of retrying to send the data based on the configured special error code includes:

[0033] The preset batch retry mechanism is invoked to retry the transmission for different data types, specifying whether to retry, the retry interval, and the number of retries.

[0034] This application also proposes a data routing device, the data routing device comprising:

[0035] The receiving module is used to receive data to be processed from the source system through the message access layer;

[0036] The processing module is used to process the data to be processed through the message processing cluster configuration template in the pre-configured message middleware;

[0037] The data push module is used to distribute processed data to the corresponding target system through the message push layer according to preset distribution rules.

[0038] This application also proposes a terminal device, which includes a memory, a processor, and a data routing program stored in the memory and executable on the processor. When the data routing program is executed by the processor, it implements the steps of the data routing method described above.

[0039] This application also proposes a computer-readable storage medium storing a data routing program, which, when executed by a processor, implements the steps of the data routing method described above.

[0040] The data routing method, apparatus, terminal device, and storage medium proposed in this application receive data to be processed from the source system through the message access layer; process the data to be processed through a pre-configured message processing cluster configuration template in the message middleware; and distribute the processed data to the corresponding target system through the message push layer according to preset distribution rules. This solution, based on distributed message middleware and template configuration, effectively solves the data integration problem of heterogeneous systems, achieves decoupling between or within systems, avoids mutual interference between upstream and downstream systems, improves system availability, and reduces the risk of data loss. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the functional modules of the terminal equipment to which the data routing device of this application belongs;

[0042] Figure 2 This is a flowchart illustrating a first exemplary embodiment of the data routing method of this application;

[0043] Figure 3 This is a schematic diagram of the two-layer architecture of the message processing platform in this application;

[0044] Figure 4 This is a flowchart illustrating a second exemplary embodiment of the data routing method of this application;

[0045] Figure 5 This is a flowchart illustrating a third exemplary embodiment of the data routing method of this application;

[0046] Figure 6 This is a flowchart illustrating a fourth exemplary embodiment of the data routing method of this application;

[0047] Figure 7 This application provides a schematic diagram illustrating the configuration hot update process for the message processing platform.

[0048] Figure 8 This is a flowchart illustrating a fifth exemplary embodiment of the data routing method of this application;

[0049] Figure 9 This is a schematic diagram of the message processing platform system architecture for this application;

[0050] Figure 10 This is a flowchart illustrating the sixth exemplary embodiment of the data routing method of this application;

[0051] Figure 11 This is a flowchart illustrating the seventh exemplary embodiment of the data routing method of this application;

[0052] Figure 12 This is a flowchart illustrating the eighth exemplary embodiment of the data routing method of this application;

[0053] Figure 13 This is a flowchart illustrating the ninth exemplary embodiment of the data routing method of this application.

[0054] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application. The main solution of this application's embodiments is: receiving data to be processed from the source system through the message access layer; processing the data to be processed through a pre-configured message processing cluster configuration template in the message middleware; and distributing the processed data to the corresponding target system through the message push layer according to preset distribution rules. This solution achieves unified reception, parsing, storage, and forwarding of various types of data, effectively solving the data integration problem of heterogeneous systems, decoupling upstream and downstream systems, avoiding mutual interference between upstream and downstream systems, and improving system availability; a dual retry mechanism reduces the risk of data loss.

[0056] Technical terms used in this application:

[0057] Distributed message middleware: a software system used to transmit and process data messages in a distributed computing environment.

[0058] Configuration template: is a predefined message format used to describe information such as the various fields, data types, and lengths in a message message.

[0059] A message processing cluster is a distributed system consisting of multiple nodes, specifically designed for message transmission and routing.

[0060] Publish-subscribe mechanism: This is a common messaging pattern where applications can publish messages to a topic, and all applications that have subscribed to that topic can receive the corresponding messages.

[0061] Hot script compilation is a technique that updates script code within an application without stopping the application.

[0062] The embodiments of this application take into account that the data interaction between upstream and downstream subsystems in the relevant technical solutions is becoming increasingly complex and manual maintenance is becoming increasingly difficult. Therefore, data interaction between upstream and downstream subsystems is difficult, and the coupling and mutual influence between upstream and downstream systems make data reuse difficult.

[0063] Based on this, this application proposes a solution that enables data connection between upstream and downstream systems, and allows for configurable and personalized data distribution and orchestration between upstream and downstream systems, thereby achieving decoupling between them.

[0064] Specifically, refer to Figure 1 , Figure 1 This is a functional module diagram of the terminal device to which the data routing device of this application belongs. The data routing device can be a data processing device independent of the terminal device, and it can be carried on the terminal device in hardware or software form. The terminal device can be a smart mobile terminal with a touchscreen, such as a mobile phone or tablet computer, or a fixed terminal with a touchscreen. This embodiment uses a mobile phone as an example.

[0065] In this embodiment, the terminal device to which the data routing device belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.

[0066] The memory 130 stores the operating system and data routing program. The data routing device can store received and processed data, as well as configuration templates for the message processing cluster, in the memory 130. The output module 110 can be a display screen, speaker, etc. The communication module 140 can include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0067] When the data routing program in memory 130 is executed by the processor, it performs the following steps:

[0068] The message access layer receives data to be processed from the source system.

[0069] The data to be processed is processed using the message processing cluster configuration template in the pre-configured message middleware.

[0070] According to the preset distribution rules, the processed data is distributed to the corresponding target system through the message push layer.

[0071] Furthermore, when the data routing program in memory 130 is executed by the processor, it also performs the following steps:

[0072] The correctness of the data source is determined by the data source rules of the cluster configuration template in the message middleware.

[0073] If the data source is correct, proceed to the following steps: process the data to be processed using the pre-configured message processing cluster configuration template in the message middleware.

[0074] Furthermore, when the data routing program in memory 130 is executed by the processor, it also performs the following steps:

[0075] Configure the message processing cluster configuration template in the message middleware.

[0076] Furthermore, when the data routing program in memory 130 is executed by the processor, it also performs the following steps:

[0077] The template engine is obtained through the front-end configuration platform of the message processing platform;

[0078] Based on the template engine, a preset configuration rule is invoked to generate a message processing cluster configuration template;

[0079] The message processing cluster configuration template is published and written to the configuration library so that the message processing cluster can subscribe to or query the corresponding message processing cluster configuration template.

[0080] Furthermore, when the data routing program in memory 130 is executed by the processor, it also performs the following steps:

[0081] The system calls the message processing cluster configuration template configured by the front-end configuration platform through the corresponding message processing cluster, performs hot compilation of the rules of the message processing cluster configuration template, and processes the data to be processed based on the hot-compiled message processing cluster configuration template.

[0082] Furthermore, when the data routing program in memory 130 is executed by the processor, it also performs the following steps:

[0083] If the message processing cluster is an access cluster, then the configuration template of the front-end configuration platform is called to filter the data, convert the content, and select message fields.

[0084] If the message processing cluster is a sending cluster, then the configuration template of the front-end configuration platform is invoked to perform deduplication and rate limiting on the data;

[0085] If the message processing cluster is a monitoring cluster, then the configuration template of the front-end configuration platform is invoked to perform data analysis, statistics comparison, and monitoring display processing.

[0086] If the message processing cluster is an exception handling cluster, the configuration template of the front-end configuration platform is invoked to store the data, perform batch retries, and refresh the retry configuration.

[0087] Furthermore, when the data routing program in memory 130 is executed by the processor, it also performs the following steps:

[0088] The deduplication rule configuration template of the front-end configuration platform is invoked to deduplicat the data distributed by the message push layer to the corresponding target system;

[0089] The rate limiting configuration template of the front-end configuration platform is invoked to limit the data rate distributed by the message push layer to the corresponding target system.

[0090] Furthermore, when the data routing program in memory 130 is executed by the processor, it also performs the following steps:

[0091] The preset error code handling mechanism is invoked, and when a data transmission error occurs, the target system returns a configured special error code.

[0092] The data is resent based on the configured special error code.

[0093] Furthermore, when the data routing program in memory 130 is executed by the processor, it also performs the following steps:

[0094] The preset batch retry mechanism is invoked to retry the transmission for different data types, specifying whether to retry, the retry interval, and the number of retries.

[0095] This embodiment, through the above-described scheme, specifically receives data to be processed from the source system via the message access layer; processes the data using a pre-configured message processing cluster template in the message middleware; and distributes the processed data to the corresponding target system via the message push layer according to preset distribution rules. Based on the distributed message middleware and template configuration provided by the data routing program, the problems of difficult data interaction between heterogeneous systems, mutual coupling and interference between upstream and downstream systems, and difficulty in data reuse in actual production environments can be solved. Based on this scheme, starting from the fact that existing data routing technologies cannot solve the problems of upstream and downstream system coupling and difficult data interaction, a data routing method is designed by combining script hot compilation and publish-subscribe mechanisms. The effectiveness of the data routing method of this invention is verified in data routing, and finally, the reusability of data routing is significantly improved by the method of this invention.

[0096] Based on, but not limited to, the terminal device architecture described above, embodiments of the method of the present invention are proposed. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here. The executing entity of the method in this embodiment can be a data routing device, a data routing terminal device, or a server. This embodiment uses a data routing device as an example, which can be integrated into a terminal device with data processing capabilities.

[0097] Reference Figure 2 , Figure 2 This is a flowchart illustrating a first exemplary embodiment of the data routing method of this application.

[0098] An embodiment of the present invention provides a data routing method applied to a message processing platform, the message processing platform including: a message middleware, the message middleware including a message access layer and a message push layer, the method including:

[0099] Step S10: Receive the data to be processed from the source system through the message access layer.

[0100] In order to facilitate data interaction between upstream and downstream systems and achieve decoupling between them, this embodiment designs a corresponding data routing program.

[0101] One implementation method involves receiving data from the data source system via the message access layer of a message middleware, enabling further data processing. This facilitates data preparation for further processing and simplifies the data processing workflow.

[0102] The message processing platform is built on the distributed message middleware Kafka. It hides the underlying technical details and provides a lightweight, general-purpose communication interface to achieve unified reception, parsing, storage, and forwarding of various types of data. This effectively solves the problem of data connection between heterogeneous systems and achieves decoupling between or within systems.

[0103] Step S20: Process the data to be processed using the pre-configured message processing cluster configuration template in the message middleware.

[0104] The pre-configured message middleware contains message processing cluster configuration templates that contain rule information for different message processing clusters to process data differently, enabling the data to be processed accordingly based on the rule information.

[0105] To enable data reuse between upstream and downstream systems and facilitate smooth data interaction, this solution uses a message processing cluster configuration template in the message middleware to adapt the format of upstream system data to that of downstream system data.

[0106] The message processing cluster configuration template in the message middleware is based on StringTemplate message message configuration, enabling customization of data sent by downstream systems. This customized access point can achieve an effective combination of functions such as filtering, deduplication, content conversion, field selection, message sending, and rate limiting through configuration without requiring development, thus enabling configurable and personalized orchestration of message distribution across heterogeneous systems.

[0107] Step S30: Distribute the processed data to the corresponding target system through the message push layer according to the preset distribution rules.

[0108] Specifically, based on preset data distribution rules, the further processed data is distributed to the target system through the message push layer of the message middleware.

[0109] To ensure the accuracy of the data processing flow, thereby simplifying the message processing flow and shortening the message processing time, this embodiment controls the distribution of data according to preset data distribution rules to achieve the distribution of the corresponding data to the correct target system.

[0110] like Figure 3 As shown, after data is accessed from the data source, it first needs to undergo data source verification according to the configured rules. After the data passes the verification, it will enter the business data message queue for further processing and distribution. Different topics represent different types of data or implement concurrent data. When the data is distributed to the data forwarding, it will perform data processing such as field transformation, field selection, field rate limiting, data deduplication, and exception retries according to the rule configuration. The processed data will be distributed to the corresponding application system after data forwarding.

[0111] The above solution implements a platform-based multi-layered processing architecture, achieving business isolation and system decoupling. The platform architecture primarily consists of two layers: a message access layer and a message push layer.

[0112] The message access layer receives data from different sources, formats, and characteristics into the system, and processes it to convert it into formatted data that meets the system requirements.

[0113] The message push layer distributes specific data to the target system according to distribution rules. This achieves both system decoupling and simplifies the message processing flow, reducing message processing time.

[0114] This embodiment, through the above-described scheme, specifically receives data to be processed from the source system via the message access layer, processes the data using a pre-configured message processing cluster template in the message middleware, and distributes the processed data to the corresponding target system via the message push layer according to preset distribution rules. This enables data interaction between upstream and downstream systems, decoupling them, and achieving data reuse between systems.

[0115] Reference Figure 4 , Figure 4 This is a flowchart illustrating an exemplary embodiment of the data routing method of this application.

[0116] Based on the first embodiment, a second embodiment of this application is proposed. The difference between the second embodiment and the first embodiment is as follows:

[0117] In this embodiment, after the step of receiving the data to be processed from the source system through the message access layer, the method further includes:

[0118] Step S101: Determine whether the data source is correct by using the data source rules of the cluster configuration template in the message middleware.

[0119] After receiving data from the data source, the data source of the data received by the message access layer is determined by using the cluster configuration template in the message middleware to determine whether the data source conforms to the data source rules for data interaction of the target system.

[0120] In order to ultimately distribute the corresponding data to the correct target system, the cluster configuration template in the message middleware sets data source rules, thereby enabling smooth data interaction between upstream and downstream systems.

[0121] Step S102: If the data source is correct, then execute the following step: process the data to be processed using the message processing cluster configuration template in the pre-configured message middleware.

[0122] If the data is verified correctly through the data source, the following steps are executed: the data to be processed is processed using the message processing cluster configuration template in the pre-configured message middleware.

[0123] This embodiment improves the efficiency of data interaction by specifically verifying the data source as a basis for determining whether to proceed to the next step.

[0124] Reference Figure 5 , Figure 5 This is a flowchart illustrating an exemplary embodiment of the data routing method of this application.

[0125] Based on the first embodiment, a third embodiment of this application is proposed. The difference between the third embodiment and the first embodiment is as follows:

[0126] In this embodiment, before processing the data to be processed using the pre-configured message processing cluster configuration template in the message middleware in step S20, the method further includes:

[0127] Step S201: Configure the message processing cluster configuration template in the message middleware.

[0128] Configure the message processing cluster configuration template for the message middleware and set the processing rules for the data to be processed.

[0129] Based on template configuration, the access party can achieve an effective combination of a series of functions such as filtering, deduplication, content conversion, field selection, message sending, and rate limiting without development, thereby realizing the configurable and personalized orchestration of message distribution to heterogeneous systems.

[0130] This embodiment improves the usability of the data processing platform by configuring the data processing rules, as described above.

[0131] Reference Figure 6 , Figure 6This is a flowchart illustrating an exemplary embodiment of the data routing method of this application.

[0132] Based on the third embodiment, a fourth embodiment of this application is proposed. The difference between the fourth embodiment and the third embodiment is as follows:

[0133] In this embodiment, the step of configuring the message processing cluster configuration template in the message middleware includes:

[0134] Step S2011: Obtain the template engine through the front-end configuration platform of the message processing platform.

[0135] The front-end configuration platform of the message processing platform obtains the StringTemplate template engine it uses.

[0136] To achieve full-process data processing configuration without requiring additional platform development, this embodiment utilizes the StringTemplate template engine to customize the data sent to the target system.

[0137] Among them, the StringTemplate template engine can fill data into a specific text template according to a certain format, which is suitable for various scenarios that require data formatting.

[0138] Step S2012: Based on the template engine, call the preset configuration rules to generate a message processing cluster configuration template.

[0139] Based on the preset configuration rules in the StringTemplate template engine of the front-end configuration platform, a configuration template for data processing cluster is generated.

[0140] Step S2013: Publish the message processing cluster configuration template and write it into the configuration library so that the message processing cluster can subscribe to or query the corresponding message processing cluster configuration template.

[0141] After the configuration template is distributed to ZooKeeper and written to the configuration database, the message processing cluster subscribes to ZooKeeper and queries the configuration database for the corresponding message processing cluster configuration template.

[0142] To ensure that the message processing cluster can update the rules in real time when the front-end configuration platform of the message processing platform updates the configuration rules, thereby improving the efficiency and accuracy of data processing, and ensuring that the configuration rules can be updated in real time according to the needs of the users.

[0143] The publish-subscribe mechanism of ZooKeeper can listen for specific events in ZooKeeper, such as the update of configuration rules in this invention. When the data changes, the message processing cluster will trigger an action to query the configuration library.

[0144] like Figure 7 As shown, the configuration rules for sending data to the target system are customized by obtaining the StringTemplate template engine from the front-end configuration platform of the message processing platform. Based on the customized configuration rules, a configuration template for data processing by the message processing cluster is generated in the template engine. The generated message processing cluster configuration template is published and written to the configuration library, enabling the message processing cluster to subscribe to or query the corresponding configuration template. By listening to specific events through the publish-subscribe mechanism of ZooKeeper, when the configuration rules are updated, the message processing cluster will trigger a query action to the configuration library, thereby realizing the real-time update of the configuration rules. After reading the rules, the message processing cluster performs hot compilation of the rules based on Groovy to make the configuration effective.

[0145] For the message processing platform, a full-process configurable architecture is adopted to achieve real-time hot updates of configurations. The platform's core functions support configurability, enabling flexible customization of downstream message sending based on StringTemplate message configuration; and enabling flexible changes to filtering and transformation rules based on Groovy hot compilation. Configuration allows for an organic combination of message processing functions such as filtering, deduplication, content transformation, field selection, field name conversion, sending, and rate limiting. Finally, real-time hot updates of configurations are achieved through ZooKeeper's publish-subscribe mechanism. Hot deployment of configurations using ZooKeeper + StringTemplate + Groovy enables rapid response to business needs and significantly reduces development and maintenance costs.

[0146] This embodiment uses the above-described solution, specifically zookeeper + StringTemplate + groovy, to achieve hot deployment of configurations, thereby improving data processing efficiency and reducing development and usage costs.

[0147] Reference Figure 8 , Figure 8 This is a flowchart illustrating an exemplary embodiment of the data routing method of this application.

[0148] Based on the first embodiment, a fifth embodiment of this application is proposed. The difference between the fifth embodiment and the first embodiment is as follows:

[0149] In this embodiment, the processing of the data to be processed through step S20, via the pre-configured message processing cluster configuration template in the message middleware, includes:

[0150] Step S202: Through the corresponding message processing cluster, call the message processing cluster configuration template configured by the front-end configuration platform, hot compile the rules of the message processing cluster configuration template, and process the data to be processed based on the hot-compiled message processing cluster configuration template.

[0151] like Figure 9 As shown, the upstream system distributes the data to be processed to the message processing platform. After the data is received, it enters the message processing cluster of the message processing platform, and the front-end configuration platform... Figure 9 The messaging platform portal system contains configuration rules for data source management, access configuration management, operation management, downstream consumption management, sending configuration management, retry configuration, data source field configuration, failure data query, environment variable management, audit management, and single message tracking. After the user configures the rules, the messaging platform portal system publishes them through ZooKeeper events and stores the configuration rules in MySQL. The message processing cluster obtains the corresponding configuration template by querying the configuration storage, performs hot compilation of the rules in the configuration template, and then processes the data accordingly.

[0152] The data processing mechanism involves several steps: First, the access cluster performs data verification, comparison and statistics, primary / backup failover, traffic distribution, lossless publishing, configuration updates, and message conversion. Second, the sending cluster handles data field filtering, comparison and statistics, data assembly, data screening, configuration updates, sending rate limiting, message conversion, and real-time retries. Third, the monitoring cluster analyzes, compares, and monitors messages in the distribution chain and failure messages from the sending cluster. Finally, the exception handling cluster handles data storage, batch retries, primary / backup failover, and retry configuration updates. After processing by these message processing clusters, error-free data is distributed by the sending cluster to the corresponding downstream systems according to preset distribution rules.

[0153] In this embodiment, the message processing platform adopts a layered distributed high-availability architecture with flexible and scalable clusters. Each server node on the platform is stateless; the message itself records its state. When a server receives a message, it retrieves the current processing state from the message record and continues processing, thus enabling flexible scaling and failover. This embodiment, through its layered distributed architecture, achieves multi-dimensional decoupling of business data and systems, significantly reducing the impact of inter-system correlation risks. Simultaneously, it accelerates the landing of business system data and simplifies the cost of using that data.

[0154] This embodiment, through the above-described scheme, specifically through the message processing cluster configuration template, processes the data to be processed according to the corresponding configuration rules, thereby achieving efficient, fast, and accurate processing of the data to be processed.

[0155] Reference Figure 10 , Figure 10 This is a flowchart illustrating an exemplary embodiment of the data routing method of this application.

[0156] Based on the above embodiments, a sixth embodiment of this application is proposed. The difference between the sixth embodiment and the first embodiment is as follows:

[0157] In this embodiment, the steps of processing the data to be processed based on the hot-compiled message processing cluster configuration template through the corresponding message processing cluster include:

[0158] Step S2021: If the message processing cluster is an access cluster, then the configuration template of the front-end configuration platform is called to filter the data, convert the content, and select message fields.

[0159] When the message processing cluster that processes the data is an access cluster, it calls the configuration template of the front-end configuration platform and performs data filtering, content transformation, and message field selection processing according to the corresponding configuration rules.

[0160] To ensure that the access cluster can correctly identify and process data of different formats and sources, and convert them into data formats suitable for subsequent processing and target systems.

[0161] Step S2022: If the message processing cluster is a sending cluster, then the configuration template of the front-end configuration platform is called to perform deduplication and rate limiting on the data.

[0162] When the message processing cluster that processes data is a sending cluster, the configuration template of the front-end configuration platform is called to perform data deduplication and rate limiting according to the corresponding configuration rules.

[0163] To avoid duplicate data transmission or exceeding system processing capacity, and to ensure the stability and reliability of the sending cluster, the resource load on the target system receiving data should be reduced.

[0164] Step S2023: If the message processing cluster is a monitoring cluster, then the configuration template of the front-end configuration platform is called to perform data analysis, statistics, comparison and statistics, and monitoring and display processing.

[0165] When the message processing cluster that processes data is a monitoring cluster, it calls the configuration template of the front-end configuration platform and performs data analysis, statistics, comparison and display according to the corresponding configuration rules.

[0166] In order to achieve comprehensive monitoring and analysis of the system's operating status, statistical analysis of data types and field content is conducted to improve the system's availability and performance.

[0167] Step S2024: If the message processing cluster is an exception handling cluster, then the configuration template of the front-end configuration platform is called to perform data storage, batch retries, and retry configuration refresh processing.

[0168] When the message processing cluster that processes data is an exception handling cluster, the configuration template of the front-end configuration platform is called, and the data storage, batch retries, and retry configuration refresh are processed according to the corresponding configuration rules.

[0169] To address various anomalies that may occur in the message processing cluster and ensure system stability and data reliability.

[0170] This embodiment, through the above-described scheme, specifically through the configuration template of the front-end configuration platform, can be customized according to the processing needs and processes of different types of message processing clusters, so as to achieve efficient, stable and reliable operation of the system.

[0171] Reference Figure 11 , Figure 11 This is a flowchart illustrating an exemplary embodiment of the data routing method of this application.

[0172] Based on the above embodiments, a seventh embodiment of this application is proposed. The difference between the seventh embodiment and the first embodiment is as follows:

[0173] In this embodiment, before the step of distributing the processed data to the corresponding target system through the message push layer according to the preset distribution rules, the method further includes:

[0174] Step S301: Call the deduplication rule configuration template of the front-end configuration platform to deduplicatize the data distributed by the message push layer to the corresponding target system.

[0175] By calling the deduplication rule configuration template of the front-end configuration platform, the data distributed by the message push layer to the corresponding target system is deduplicated according to the corresponding deduplication rules.

[0176] To avoid generating duplicate data, reduce the system load, and prevent data inaccuracies or system crashes caused by conflicts during message processing.

[0177] The deduplication rule configuration template is a set of rules based on the business requirements of the target system. For example, it can deduplication based on key information such as data fields and timestamps to ensure that the same data will only be pushed once.

[0178] To address duplicate data during distribution, the front-end configuration platform is equipped with a corresponding deduplication mechanism. Whether the message source is duplicated or the message duplication is due to reliance on Kafka middleware, the platform provides a Redis deduplication append function at the sending interface to shield downstream systems from the impact of message duplication.

[0179] Step S302: Call the rate limiting configuration template of the front-end configuration platform to rate limit the data distributed by the message push layer to the corresponding target system.

[0180] By calling the rate limiting rule configuration template of the front-end configuration platform, the data distributed by the message push layer to the corresponding target system is rate-limited according to the corresponding rate limiting rules.

[0181] To address traffic fluctuations during data interaction, the front-end configuration platform implements corresponding rate limiting mechanisms. To further improve message transmission success rates, the platform provides flexible rate limiting services for downstream systems. The platform implements rate limiting functionality for each business scenario, allowing downstream systems to set appropriate rate limits based on their specific needs, ensuring stable operation.

[0182] This embodiment, through the above-described scheme, specifically through data deduplication and rate limiting, can improve potential problems in the data processing process, enhance the stability and reliability of the system, and provide support for the continuous use of data in the target system.

[0183] Reference Figure 12 , Figure 12 This is a flowchart illustrating an exemplary embodiment of the data routing method of this application.

[0184] Based on the above embodiments, an eighth embodiment of this application is proposed. The difference between the eighth embodiment and the first embodiment is as follows:

[0185] In this embodiment, after the step of distributing the processed data to the corresponding target system through the message push layer according to the preset distribution rules, the method further includes:

[0186] Step S303: Invoke the preset error code handling mechanism to obtain the special error code returned by the target system when the data transmission error occurs.

[0187] By invoking a preset error code handling mechanism, the special error codes returned by the target system can be captured.

[0188] To identify specific errors, malfunctions, or anomalies that occur during data transmission.

[0189] The pre-defined, customized error code handling mechanism can handle sudden failures in downstream systems. The platform provides retry functionality for specific error codes. Downstream systems can be configured with special error codes, and when a data dependency is not ready, the platform will return the configured error code and retry the data after a certain period of time.

[0190] Step S304: Retry sending the data according to the configured special error code.

[0191] Based on the configured special error codes, data that has encountered errors can be resent.

[0192] For special error codes, the message processing platform's dual retry mechanism ensures data delivery. The platform provides a robust dual exception retry mechanism. When transmission errors occur due to network jitter or downstream system instability, the platform will retry three times. If all retries fail, the message will be temporarily stored in the Elasticsearch + MySQL master-slave database, and then a scheduled task will execute batch retries to ensure data delivery to the downstream system.

[0193] A robust data consistency mechanism is implemented, with refined data transmission granularity to accurately assess the impact of abnormal retransmissions and avoid unnecessary data duplication. Building upon this, a dual retry mechanism for real-time and asynchronous batch processing is optimized to mitigate the risk of uncontrollable data loss due to environmental jitter. Simultaneously, a customized error code handling mechanism is provided to address scenarios where downstream systems experience sudden transmission interruptions, enabling data recovery afterward.

[0194] This embodiment, through the above-described scheme, specifically by having the target system return a special error code when a data transmission error occurs, and retrying the data transmission based on the special error code, better handles various possible errors and abnormal situations during the data transmission process, avoids data loss and transmission failure, and further improves the stability and reliability of the system.

[0195] Reference Figure 13 , Figure 13 This is a flowchart illustrating an exemplary embodiment of the data routing method of this application.

[0196] Based on the eighth embodiment, a ninth embodiment of this application is proposed. The difference between the ninth embodiment and the eighth embodiment is as follows:

[0197] In this embodiment, the step of retrying to send the data based on the configured special error code includes:

[0198] Step S3041: Invoke the preset batch retry mechanism to retry the transmission of different data types, including whether to retry, the retry interval, and the number of retries.

[0199] Based on different data types, determine whether retrying is necessary. If retrying is required, invoke the preset batch retry mechanism, determine the retry interval and number of retries according to the preset retry strategy, and send the data that needs to be retried in batches.

[0200] To avoid retries for all data and excessive retries that could block system data interaction, we limit whether data types should be retried, the retry interval, and the number of retries, thereby saving system resources and improving the efficiency of data interaction.

[0201] The platform provides a configurable batch retry mechanism, which allows for flexible adjustment of whether to retry for different data types, the retry interval, and the number of retries. This can compensate for data loss caused by long-term system anomalies and greatly reduce the frequency of manual intervention.

[0202] This embodiment, through the above-described scheme, specifically uses a configurable batch retry mechanism to limit whether data types should be retried, the retry interval, and the number of retries, thereby avoiding data loss caused by prolonged system anomalies and improving the stability and reliability of the system.

[0203] Furthermore, embodiments of this application also propose a data routing device, the data routing device comprising:

[0204] The receiving module is used to receive data to be processed from the source system through the message access layer;

[0205] The processing module is used to process the data to be processed through the message processing cluster configuration template in the pre-configured message middleware;

[0206] The data push module is used to distribute processed data to the corresponding target system through the message push layer according to preset distribution rules.

[0207] The principle and implementation process of data routing in this embodiment can be found in the above embodiments, and will not be repeated here.

[0208] Furthermore, this application also proposes a terminal device, which includes a memory, a processor, and a data routing program stored in the memory and executable on the processor. When the data routing program is executed by the processor, it implements the steps of the data routing method described above.

[0209] Since this data routing program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be elaborated here.

[0210] Furthermore, embodiments of this application also propose a computer-readable storage medium storing a data routing program, which, when executed by a processor, implements the steps of the data routing method described above.

[0211] Since this data routing program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be elaborated here.

[0212] Compared to existing technologies, the data routing method, apparatus, terminal device, and storage medium proposed in this application receive data to be processed from the source system through the message access layer; process the data to be processed through a pre-configured message processing cluster configuration template in the message middleware; and distribute the processed data to the corresponding target system through the message push layer according to preset distribution rules. This invention's technical solution, based on distributed message middleware, effectively solves the data integration problem between heterogeneous systems, achieving decoupling between or within systems.

[0213] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0214] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0215] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.

[0216] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data routing method, characterized in that, The method is applied to a message processing platform, which includes a message middleware, comprising a message access layer and a message push layer. The method includes: The message access layer receives data to be processed from the source system. The data to be processed is processed using the message processing cluster configuration template in the pre-configured message middleware. The step of processing the data to be processed using the pre-configured message processing cluster configuration template in the message middleware includes: The system calls the message processing cluster configuration template configured by the front-end configuration platform through the corresponding message processing cluster, performs hot compilation of the rules of the message processing cluster configuration template based on Groovy, and processes the data to be processed based on the hot-compiled message processing cluster configuration template; wherein, hot deployment of configuration is achieved through Zookeeper + StringTemplate + Groovy. According to the preset distribution rules, the processed data is distributed to the corresponding target system through the message push layer; Before the step of processing the data to be processed using the cluster configuration template in the message middleware, the method further includes: Configure the message processing cluster configuration template in the message middleware; The steps for configuring the message processing cluster configuration template in the message middleware include: The template engine is obtained through the front-end configuration platform of the message processing platform; Based on the template engine, a preset configuration rule is invoked to generate a message processing cluster configuration template; The message processing cluster configuration template is published and written to the configuration library so that the message processing cluster can subscribe to or query the corresponding message processing cluster configuration template. The steps for processing the data to be processed based on the hot-compiled message processing cluster configuration template include: If the message processing cluster is an access cluster, then the configuration template of the front-end configuration platform is called to filter the data, convert the content, and select message fields. If the message processing cluster is a sending cluster, then the configuration template of the front-end configuration platform is invoked to perform deduplication and rate limiting on the data; If the message processing cluster is a monitoring cluster, then the configuration template of the front-end configuration platform is invoked to perform data analysis, statistics comparison, and monitoring display processing. If the message processing cluster is an exception handling cluster, the configuration template of the front-end configuration platform is invoked to perform data storage, batch retries, and retry configuration refresh processing. Before the step of distributing the processed data to the corresponding target system through the message push layer according to the preset distribution rules, the method further includes: The deduplication rule configuration template of the front-end configuration platform is invoked to deduplicat the data distributed by the message push layer to the corresponding target system; The rate limiting configuration template of the front-end configuration platform is invoked to limit the data rate distributed by the message push layer to the corresponding target system.

2. The data routing method according to claim 1, characterized in that, Following the step of receiving the data to be processed from the source system through the message access layer, the method further includes: The correctness of the data source is determined by the data source rules of the cluster configuration template in the message middleware. If the data source is correct, proceed to the following steps: process the data to be processed using the pre-configured message processing cluster configuration template in the message middleware.

3. The data routing method according to claim 1, characterized in that, The configuration rules include one or more of the following: data source configuration, filtering rules, deduplication rules, content conversion rules, message field rules, rate limiting configuration, analysis and statistics, comparison and statistics, monitoring and display, data storage, batch retries, and retry configuration refresh information.

4. The data routing method according to claim 1, characterized in that, After the step of distributing the processed data to the corresponding target system through the message push layer according to the preset distribution rules, the method further includes: The preset error code handling mechanism is invoked, and when a transmission error occurs in the processed data, the special error code returned by the target system is obtained; Based on the specific error code, the data is resent.

5. The data routing method according to claim 4, characterized in that, The step of retrying to send data based on the special error code includes: The preset batch retry mechanism is invoked to retry the transmission based on whether to retry, the retry interval, and the number of retries for different data types.

6. A data routing device, characterized in that, The data routing device includes: The receiving module is used to receive data to be processed from the source system through the message access layer in the message middleware; The processing module is used to process the data to be processed through the message processing cluster configuration template in the pre-configured message middleware; The processing module is also used in the step of processing the data to be processed through the pre-configured message processing cluster configuration template in the message middleware, including: calling the message processing cluster configuration template configured by the front-end configuration platform through the corresponding message processing cluster, performing hot compilation of the rules of the message processing cluster configuration template based on Groovy, and processing the data to be processed based on the hot-compiled message processing cluster configuration template; wherein, hot deployment of configuration is achieved through Zookeeper+StringTemplate+Groovy. The data push module is used to distribute the processed data to the corresponding target system through the message push layer in the message middleware according to the preset distribution rules. The configuration module is used to configure the message processing cluster configuration template in the message middleware; The configuration module is further configured to obtain a template engine through the front-end configuration platform of the message processing platform; generate a message processing cluster configuration template by calling preset configuration rules based on the template engine; and publish the message processing cluster configuration template and write it into the configuration library so that the message processing cluster can subscribe to or query the corresponding message processing cluster configuration template. The processing module is further configured to: if the message processing cluster is an access cluster, call the configuration template of the front-end configuration platform to filter, convert, and select message fields for the data; if the message processing cluster is a sending cluster, call the configuration template of the front-end configuration platform to perform deduplication and rate limiting on the data; if the message processing cluster is a monitoring cluster, call the configuration template of the front-end configuration platform to perform data analysis, statistics, comparison, and monitoring display; and if the message processing cluster is an exception handling cluster, call the configuration template of the front-end configuration platform to perform data storage, batch retries, and retry configuration refresh. The calling module is used to call the deduplication rule configuration template of the front-end configuration platform to deduplicatize the data distributed by the message push layer to the corresponding target system; and to call the rate limiting configuration template of the front-end configuration platform to rate limit the data distributed by the message push layer to the corresponding target system.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a data routing program stored in the memory and executable on the processor. When the data routing program is executed by the processor, it implements the steps of the data routing method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data routing program, which, when executed by a processor, implements the steps of the data routing method as described in any one of claims 1-5.

Images