Data gateway system based on message queue
By designing a data gateway system based on message queues, the problem of missing message system verification and conversion in message queues is solved, data consistency in the message sending and receiving process is achieved, and communication and development costs between systems are reduced.
Patent Information
- Application Number
- CN202411679668.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-02-18
AI Technical Summary
When using the message queue intermediate key for message delivery, the lack of checksum conversion of the message body, resulting in unverified data of the upstream system being passed to the downstream system, resulting in data inconsistency and increasing development and maintenance costs.
Design a data gateway system based on message queue, and dynamically complete message sending and receiving through message forwarding configuration module, message control module and data quality feedback module, and perform message verification, processing and recording in the process to reduce data inconsistency The situation.
By performing checksum processing during message sending and receiving, data inconsistency caused by failure of verification is reduced, system stability is improved, and communication and development costs are reduced.
Smart Images
Figure CN119484271A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of information management, in particular to a data gateway system based on a message queue. Background Art
[0002] One of the core functions of the data gateway is to forward messages and configure the sender and receiver information of the message into the system. The sender's upstream system and the receiver's downstream system are not directly connected, but read the configuration information in the data gateway system, and dynamically complete the message sending and receiving through the data gateway, and perform verification, control and recording in the process.
[0003] At present, the common data gateway in web development mainly refers to the traditional API gateway based on HTTP request, which forwards messages according to the request path and can perform field verification, field processing and request interception in the process. However, when using the message queue intermediate key for message transmission, the lack of verification and conversion of the message body causes some unverified data in the upstream system to be passed to the downstream subsystem, which ultimately leads to data inconsistency between the upstream and downstream systems, such as Figure 1 As shown in the figure, data quality is affected, communication costs between developers of upstream and downstream systems are increased, as well as system data maintenance costs. Summary of the invention
[0004] The purpose of the present invention is to provide a data gateway system based on a message queue, which dynamically completes message sending and message receiving through the data gateway according to the configuration information in the data gateway system, and verifies, controls and records the messages in the process, thereby reducing the data inconsistency caused by verification failure between systems. The present invention is implemented by the following technical solutions.
[0005] The present invention provides a data gateway system based on a message queue, comprising: The message forwarding configuration module is used to configure the sender information and receiver information of the message into the system; A message control module, used to control message routing during message sending based on configuration information in the message forwarding configuration module and to verify message compliance according to field verification rules; The data quality feedback module is used to record and count the messages that fail the compliance check in the message control module.
[0006] In actual applications, due to the lack of verification and conversion of the message body during the transmission of the message body from the upstream system to the downstream subsystem, some unverified data of the upstream system is transmitted to the downstream subsystem, which in turn leads to data inconsistency between the message sender and the message receiver. The present invention adopts a method in which the upstream system of the message body sender and the downstream subsystem of the message body receiver are not directly connected, but the data gateway system dynamically completes the sending and receiving of the message body according to the configuration information read in the data gateway system, and verifies, processes and records the message body in this process, thereby reducing the data inconsistency caused by verification failure between the upstream and downstream systems.
[0007] Optionally, the message forwarding configuration module includes a message body configuration unit, a data source configuration unit and a target configuration unit; the message forwarding configuration module is used to associate the message body with the data source and the target source, realize the configuration of the message body forwarding process, and send the message body of the specified structure from the data source to the target source; these three units are combined to describe the three elements in the message sending process: the structure of the message, the basic information of the sender, and the basic information of the receiver; The message body configuration unit is used to configure basic information and field information of the message; The data source configuration unit is used to configure the sender message queue server's IP, port, user name, password, and the topic or queue name to which the message is sent; The target configuration unit is used to configure the IP, port, user name, password, topic or queue name to be monitored of the recipient message queue server.
[0008] Optionally, the basic information includes a message name, a function module name, a forwarding method, and a message category, and the field information includes a field name, a field type, a field description, and a default value; The data source configuration unit is also used to configure the system name, function module name and remarks; The target configuration unit is also used to configure the system name, function module name and remarks.
[0009] Optionally, the message control module includes a serially connected data verification rule configuration unit, a verification unit and a field processing configuration unit; The data verification rule configuration unit is used to provide different verification rules for fields of different data types in each message body, describe the field verification rules in the message body through expressions, and then parse the expressions through an expression parser, and input the parsing results into the verification unit, input the parsing results that pass the verification into the field processing configuration unit, and input the parsing results that fail the verification into the data quality feedback module.
[0010] Optionally, different field verification rules are provided for fields of different data types in each message body, specifically including: For string type fields, non-blank, length limit and regular expression matching verification rules are provided. For integer type fields, it provides the maximum, minimum and multiple fields' four arithmetic operation result verification rules. For decimal type fields, maximum, minimum and decimal place verification rules are provided. For array type fields, aggregation operation verification rules for array fields are provided, including sum, average, maximum value, minimum value, and array length.
[0011] Optionally, the field processing configuration unit is used to configure field processing logic for each field of each message body that passes the verification, and complete the processing of some fields through the expression parser, including converting decimal values into percentages, converting fields according to expression formulas, assigning default values to fields, and adding additional fields and default values.
[0012] Optionally, the data quality feedback module includes an alarm unit and a data quality statistics unit connected in parallel; the alarm unit reflects the immediacy of message exception processing: when an exception occurs, an alarm behavior is triggered immediately. The data quality statistics unit performs statistical analysis on historical alarm records; The alarm unit is used to configure the notification mode when the trigger verification fails for each message body, and configure different notification strategies according to the importance of the message, the number of verification exceptions and the frequency. The data quality statistics unit is used to combine the number of verification anomalies, count the data quality in the business dimension and the time dimension, and output a data quality report.
[0013] Optionally, the notification method includes system notification, DingTalk notification, enterprise WeChat notification, SMS notification and phone notification.
[0014] Optionally, the business dimension includes message body, field, message type, and reason for verification failure; the time dimension includes daily report, weekly report and monthly report.
[0015] Optionally, a comparison function is also provided in the time dimension, including data quality statistics of today and yesterday, data quality statistics of this week and last week, and data quality statistics of this month and last month. Beneficial Effects
[0016] (1) The present invention reduces the data inconsistency between the sender and the receiver of the message body due to verification failure by adding a layer of data gateway system between the message sender and the message receiver; completes the field-based message compliance verification before the consumer receives the message, so that the problematic data is exposed and the transmission is terminated in advance, thereby improving the stability of the consumer system, and also reducing the communication cost and development cost caused by data quality problems, thereby reducing the impact on the consumer system; (2) The data quality feedback module designed in the present invention can record and count data quality problems: when a message fails verification, the module records relevant information, counts the number and frequency of occurrence of data quality problems in the business dimension and time dimension, and provides data support for improving data quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 The figure is a schematic diagram of a scenario in which data between upstream and downstream systems is inconsistent according to the present invention; Figure 2 The figure is a schematic diagram of the structure of a data gateway system based on a message queue in an embodiment of the present invention. DETAILED DESCRIPTION
[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0019] Embodiment 1, combination Figure 2 This embodiment introduces a data gateway system based on a message queue, including: The message forwarding configuration module is used to configure the sender information and receiver information of the message into the system; A message control module, used to control message routing during message sending based on configuration information in the message forwarding configuration module and to verify message compliance according to field verification rules; The data quality feedback module is used to record and count the messages that fail the compliance check in the message control module.
[0020] In actual applications, due to the lack of verification and conversion of the message body during the transmission of the message body from the upstream system to the downstream subsystem, some unverified data of the upstream system is transmitted to the downstream subsystem, which in turn leads to data inconsistency between the message sender and the message receiver. The present invention adopts a method in which the upstream system of the message body sender and the downstream subsystem of the message body receiver are not directly connected, but the data gateway system dynamically completes the sending and receiving of the message body according to the configuration information read in the data gateway system, and verifies, processes and records the message body in this process, thereby reducing the data inconsistency caused by verification failure between the upstream and downstream systems.
[0021] Example 2, based on Example 1, this example introduces the following contents of a data gateway system based on a message queue: The message forwarding configuration module includes the message body configuration unit, the data source configuration unit and the target configuration unit; the message body configuration unit is used to associate the message body with the data source and the target source, realize the configuration of the message body forwarding process, and send the message body of the specified structure from the data source to the target source. These three units are combined to describe the three elements in the message sending process: the structure of the message, the basic information of the sender, and the basic information of the receiver.
[0022] The message body configuration unit is used to configure the basic information and field information of the message; wherein the basic information includes the message name, function module name, forwarding method and message classification, and the field information includes the field name, field type, field description and default value.
[0023] The data source configuration unit is used to configure the sender message queue server's IP, port, user name, password, and the topic or queue name to which the message is sent; it is also used to configure the system name, function module name, and remarks. The target configuration unit is used to configure the IP, port, user name, password, and topic or queue name to be monitored of the receiving message queue server; it is also used to configure the system name, function module name, and remarks.
[0024] The message control module includes a serially connected data verification rule configuration unit, a verification unit and a field processing configuration unit.
[0025] The data verification rule configuration unit is used to provide different verification rules for fields of different data types in each message body. The field verification rules in the message body are described through expressions, and then the expressions are parsed by the expression parser. The parsing results are input into the verification unit, and the parsing results that pass the verification are input into the field processing configuration unit, and the parsing results that fail the verification are input into the data quality feedback module.
[0026] Different field validation rules are provided for fields of different data types in each message body, including: For string type fields, non-blank, length limit and regular expression matching verification rules are provided. For integer type fields, it provides the maximum, minimum and multiple fields' four arithmetic operation result verification rules. For decimal type fields, maximum, minimum and decimal place verification rules are provided. For array type fields, aggregation operation verification rules for array fields are provided, including sum, average, maximum value, minimum value, and array length.
[0027] The field processing configuration unit is used to configure the field processing logic for each field of each message body that has passed the verification. The expression parser is used to complete the processing of some fields, including converting decimal values into percentages, converting fields according to expression formulas, assigning default values to fields, and adding additional fields and default values.
[0028] The data quality feedback module includes an alarm unit and a data quality statistics unit connected in parallel. The alarm unit reflects the immediacy of message exception processing: when an exception occurs, an alarm is triggered immediately. The data quality statistics unit performs statistical analysis on historical alarm records. The alarm unit is used to configure the notification method for each message body when the trigger verification fails, and configure different notification strategies according to the importance of the message, the number of verification exceptions, and the frequency. Among them, the notification methods include system notification, DingTalk notification, enterprise WeChat notification, SMS notification, and phone notification.
[0029] The data quality statistics unit is used to combine the number of verification exceptions, count the data quality in the business dimension and time dimension, and output the data quality report. The business dimension includes the message body, field, message type, and reason for verification failure; the time dimension includes daily, weekly, and monthly reports. The time dimension also provides a comparison function, including data quality statistics for today and yesterday, data quality statistics for this week and last week, and data quality statistics for this month and last month.
[0030] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.
Claims
1. A data gateway system based on a message queue, characterized in that: include: The message forwarding configuration module is used to configure the sender information and receiver information of the message into the system; A message control module, used to control message routing during message sending based on configuration information in the message forwarding configuration module and to verify message compliance according to field verification rules; The data quality feedback module is used to record and count the messages that fail the compliance check in the message control module.
2. The data gateway system according to claim 1, characterized in that: The message forwarding configuration module includes a message body configuration unit, a data source configuration unit and a target configuration unit; the message forwarding configuration module is used to associate the message body with the data source and the target source, realize the configuration of the message body forwarding process, and send the message body of the specified structure from the data source to the target source; The message body configuration unit is used to configure basic information and field information of the message; The data source configuration unit is used to configure the sender message queue server's IP, port, user name, password, and the topic or queue name to which the message is sent; The target configuration unit is used to configure the IP, port, user name, password, topic or queue name to be monitored of the recipient message queue server.
3. The data gateway system according to claim 2, characterized in that: The basic information includes the message name, function module name, forwarding mode and message classification, and the field information includes the field name, field type, field description and default value; The data source configuration unit is also used to configure the system name, function module name and remarks; The target configuration unit is also used to configure the system name, function module name and remarks.
4. The data gateway system according to claim 1, characterized in that: The message control module includes a serially connected data verification rule configuration unit, a verification unit and a field processing configuration unit; The data verification rule configuration unit is used to provide different verification rules for fields of different data types in each message body, describe the field verification rules in the message body through expressions, and then parse the expressions through an expression parser, and input the parsing results into the verification unit, input the parsing results that pass the verification into the field processing configuration unit, and input the parsing results that fail the verification into the data quality feedback module.
5. The data gateway system according to claim 4, characterized in that: Different field verification rules are provided for fields of different data types in each message body, specifically including: For string type fields, non-blank, length limit and regular expression matching verification rules are provided. For integer type fields, it provides the maximum, minimum and multiple fields' four arithmetic operation result verification rules. For decimal type fields, maximum, minimum and decimal place verification rules are provided. For array type fields, aggregation operation verification rules for array fields are provided, including sum, average, maximum value, minimum value, and array length.
6. The data gateway system according to claim 5, characterized in that: The field processing configuration unit is used to configure the field processing logic for each field of each message body that has passed the verification, and complete the processing of some fields through the expression parser, including converting decimal values into percentages, converting fields according to expression formulas, assigning default values to fields, and adding additional fields and default values.
7. The data gateway system according to claim 6, characterized in that: The data quality feedback module includes an alarm unit and a data quality statistics unit connected in parallel; The alarm unit is used to configure the notification mode when the trigger verification fails for each message body, and configure different notification strategies according to the importance of the message, the number of verification exceptions and the frequency. The data quality statistics unit is used to combine the number of verification anomalies, count the data quality in the business dimension and the time dimension, and output a data quality report.
8. The data gateway system according to claim 7, characterized in that: The notification methods include in-system notifications, DingTalk notifications, enterprise WeChat notifications, SMS notifications and phone notifications.
9. The data gateway system according to claim 7, characterized in that: The business dimension includes message body, field, message type, and reason for verification failure; the time dimension includes daily report, weekly report, and monthly report.
10. The data gateway system according to claim 9, characterized in that: A comparison function is also provided in the time dimension, including data quality statistics for today and yesterday, data quality statistics for this week and last week, and data quality statistics for this month and last month.