FLINK job real-time message processing method and device based on distributed configuration center

Through the message processing method based on the distributed configuration center, dynamically configure and parse message information, the service interruption problem caused by rules changes during message processing in the existing technology is solved, and the flexibility of real-time message processing and logical calculation is realized. and efficient.

CN115665284BActive Publication Date: 2024-11-22SHANGHAI PUDONG DEVELOPMENT BANK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211333892.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-28
Publication Date
2024-11-22
Estimated Expiration
2042-10-28

AI Technical Summary

Technical Problem

In the process of message processing in the big data environment, the flow operation needs to be modified when the rule logic changes, resulting in service interruption. The connection between modules is tight, making it difficult to achieve flexible addition and modification, and cannot meet the needs of fast response and timely decision-making.

Method used

Using the message processing method based on the distributed configuration center, it uses the client configuration data to configure message analysis, adaptation and rule matching information, listens to configuration directory updates in real time, and dynamically determines the adaptation category and rule logic information of the message. Implement the separation of the front-end and the back-end, and realize the dynamic changes in real-time message processing and logical calculation.

Benefits of technology

It improves the efficiency and flexibility of service message processing, realizes dynamic changes in real-time logical computing requirements, reduces the risk of service interruption, and enhances the flexibility and response capabilities of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115665284B_ABST
    Figure CN115665284B_ABST
Patent Text Reader

Abstract

The present disclosure relates to the technical field of real-time processing of big data, and specifically discloses a message processing method, device and computer equipment based on a distributed configuration center, the method comprising: receiving configuration data transmitted by a client, configuring message parsing configuration information, message adaptation configuration information and message rule matching configuration information according to the configuration data; obtaining a business message, parsing and obtaining the first target field of the business message based on the message parsing configuration information; determining the second target field in the first target field according to the message adaptation configuration information, and determining the adaptation category of the business message according to the second target field; determining the rule logic information corresponding to the adaptation category based on the message rule matching configuration information; and performing logical calculation processing on the business message according to the rule logic information. The present disclosure realizes real-time message processing and dynamic changes in the logical calculation requirements of business messages.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of big data real-time processing, and in particular to a method and device for real-time message processing of FLINK jobs based on a distributed configuration center. Background Art

[0002] With the development of communication data exchange technology, large quantities of business data often rely on message processing systems for data exchange. In the era of information explosion, business message data is being generated at a huge volume every second. Users need fast responses to bring convenient experience. At the same time, timely decision-making for business scenarios is also crucial. Therefore, it is imperative to use big data technology to extract and analyze key information online and maximize the use of data value.

[0003] In the related technologies, the low-latency and high-response message perception process in the big data environment is implemented based on distributed real-time computing technology, and the functions of each module are realized by developing stream processing logic. When the rule logic changes, the stream operation needs to be modified, and the effectiveness process involves back-end intervention and service interruption. At the same time, this method solidifies the processing logic, and the modules are closely connected. If there are new or modified requirements, it will involve the impact of each module, making it difficult to add or modify flexibly. Summary of the invention

[0004] Based on this, it is necessary to provide a message processing method, device, computer equipment, storage medium and computer program product based on a distributed configuration center to address the above technical problems.

[0005] In a first aspect, the present disclosure provides a message processing method based on a distributed configuration center. The method comprises:

[0006] Receive configuration data transmitted by the client, and configure message parsing configuration information, message adaptation configuration information, and message rule matching configuration information according to the configuration data;

[0007] Acquire a service message, and parse and obtain a first target field of the service message based on the message parsing configuration information;

[0008] Determine a second target field in the first target field according to the message adaptation configuration information, and determine an adaptation category of the service message according to the second target field; there is a mapping relationship between the second target field and the adaptation category;

[0009] Determine the rule logic information corresponding to the adaptation category based on the message rule matching configuration information;

[0010] Perform logical calculation processing on the service message according to the rule logic information.

[0011] In one of the embodiments, the receiving configuration data transmitted by the client, and configuring the message parsing configuration information, the message adaptation configuration information, and the message rule matching configuration information according to the configuration data includes:

[0012] Storing the received configuration data in the configuration directory;

[0013] The configuration directory is monitored in real time, and in response to an update event of the configuration directory, the message parsing configuration information, the message adaptation configuration information and the message rule matching configuration information are determined according to the updated configuration directory.

[0014] In one embodiment, the acquiring of the service message, and obtaining the first target field of the service message based on the message parsing configuration information includes:

[0015] Read the configuration directory and generate a parsing mapping table according to the message parsing configuration information;

[0016] Determine the protocol type of the service message, and determine the message parsing configuration information corresponding to the protocol type of the service message;

[0017] A first target field in the service message is extracted based on the parsing mapping table, and the first target field is mapped to output details.

[0018] In one embodiment, the acquiring of the service message, and obtaining the first target field of the service message based on the message parsing configuration information further comprises:

[0019] Determining transaction information and source information of the business message;

[0020] Screening based on the transaction information and source information according to the message parsing configuration information, and storing the business messages that meet the screening rules in the global database;

[0021] The business messages that meet the screening rules are stored in a distributed memory database, and the business messages in the distributed memory database are matched with request messages and response messages, and the matched business messages and the business messages that have not been matched within a preset time are cleared.

[0022] In one of the embodiments, determining the second target field in the first target field according to the message adaptation configuration information, and determining the adaptation category of the service message according to the second target field includes:

[0023] Read the configuration directory and generate an adaptation mapping table according to the message adaptation configuration information;

[0024] A second target field in the first target field is captured based on the adaptation mapping table, and the service message is distributed to the corresponding adaptation category according to the second target field.

[0025] In one of the embodiments, determining the rule logic information corresponding to the adaptation category based on the message rule matching configuration information includes:

[0026] Read the configuration directory and generate a rule matching mapping table according to the message rule matching configuration information;

[0027] Rule logic information associated with the adaptation category is determined based on the rule matching mapping table.

[0028] In a second aspect, the present disclosure further provides a message processing device based on a distributed configuration center. The device comprises:

[0029] A configuration module, used to receive configuration data transmitted by a client, and configure the message parsing configuration information, message adaptation configuration information and message rule matching configuration information according to the configuration data;

[0030] A message acquisition module, used to acquire a service message, and parse and obtain a first target field of the service message based on the message parsing configuration information;

[0031] An adaptation category module, used to determine a second target field in the first target field according to the message adaptation configuration information, and determine an adaptation category of the service message according to the second target field; there is a mapping relationship between the second target field and the adaptation category;

[0032] A rule determination module, used to determine the rule logic information corresponding to the adaptation category based on the message rule matching configuration information;

[0033] A processing module is used to perform logical calculation processing on the service message according to the rule logic information.

[0034] In one embodiment, the configuration module includes:

[0035] A configuration directory unit, used to store the received configuration data into a configuration directory;

[0036] The monitoring unit is used to monitor the configuration directory in real time, and in response to the configuration directory update event, determine the message parsing configuration information, message adaptation configuration information and message rule matching configuration information according to the updated configuration directory.

[0037] In one embodiment, the message acquisition module includes:

[0038] A parsing mapping table unit is used to read the configuration directory and generate a parsing mapping table according to the message parsing configuration information.

[0039] A protocol type unit, used to determine the protocol type of the service message, and determine the message parsing configuration information corresponding to the protocol type of the service message;

[0040] The first target mapping unit is used to extract the first target field in the service message based on the parsing mapping table, and map the first target field to output details.

[0041] In one embodiment, the message acquisition module further includes:

[0042] A message header unit, used to determine the transaction information and source information of the business message;

[0043] A screening unit, configured to screen the business messages that meet the screening rules based on the transaction information and the source information according to the message parsing configuration information, and store the business messages in a global database;

[0044] The memory cleaning unit stores the business messages that meet the screening rules into a distributed memory database, matches the request messages and response messages of the business messages in the distributed memory database, and cleans up the matched business messages and the unmatched business messages that exceed the preset time.

[0045] In one of the embodiments, the adaptation category module includes:

[0046] An adaptation mapping table unit, used to read the configuration directory and generate an adaptation mapping table according to the message adaptation configuration information;

[0047] The second target mapping unit is used to capture the second target field in the first target field based on the adaptation mapping table, and distribute the service message to the corresponding adaptation category according to the second target field.

[0048] In one embodiment, the rule determination module includes:

[0049] A rule matching mapping table unit, used to read the configuration directory and generate a rule matching mapping table according to the message rule matching configuration information;

[0050] The third target mapping unit is configured to determine rule logic information associated with the adaptation category based on the rule matching mapping table.

[0051] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the message processing method based on the distributed configuration center are implemented.

[0052] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned message processing method based on a distributed configuration center when executed by a processor.

[0053] In a fifth aspect, the present disclosure further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the message processing method based on the distributed configuration center are implemented.

[0054] The above-mentioned message processing method, apparatus, computer device, storage medium and computer program product based on the distributed configuration center have at least the following beneficial effects:

[0055] The present invention obtains the configuration information of the front end and acts on the processing logic of the business message of the back end. Specifically, the business message is pre-processed according to the message parsing configuration information to extract the first target field of the business message, and then the second target field in the first target field is determined according to the message adaptation configuration information, and the adaptation category (business scenario) of the business message is determined according to the second target field, and the rule logic information corresponding to the adaptation category is further determined according to the message rule matching configuration information, and the business message is logically calculated and processed according to the rule logic information under the business message adaptation business scenario, thereby realizing the separation of the front end and the back end. The front end is used to integrate the configuration information, and the back end performs real-time logical processing on the business message according to the configuration information, thereby realizing real-time message processing and dynamic change of the logical calculation requirements of the business message, and improving the efficiency and flexibility of business message processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 An application environment diagram of a message processing method based on a distributed configuration center in one embodiment;

[0058] Figure 2 It is a flowchart of a message processing method based on a distributed configuration center in one embodiment;

[0059] Figure 3 It is a flowchart of a message processing method based on a distributed configuration center in another embodiment;

[0060] Figure 4 It is a flowchart of a message processing method based on a distributed configuration center in another embodiment;

[0061] Figure 5 It is a flowchart of a message processing method based on a distributed configuration center in another embodiment;

[0062] Figure 6 A schematic diagram of a message preprocessing process in an embodiment;

[0063] Figure 7 It is a flowchart of a message processing method based on a distributed configuration center in another embodiment;

[0064] Figure 8 A schematic diagram of a message adaptation category process in an embodiment;

[0065] Fig. 9 It is a flowchart of a message processing method based on a distributed configuration center in another embodiment;

[0066] Fig.10 A schematic diagram of a message rule matching process in an embodiment;

[0067] Fig.11 A schematic diagram of a message processing flow in one embodiment;

[0068] Fig.12 A schematic diagram of a message configuration process in an embodiment;

[0069] Fig.13 It is a structural block diagram of a message processing device based on a distributed configuration center in one embodiment;

[0070] Fig.14 It is a structural block diagram of a message processing device based on a distributed configuration center in another embodiment;

[0071] Fig.15 It is a structural block diagram of a message processing device based on a distributed configuration center in another embodiment;

[0072] Fig.16 It is a structural block diagram of a message processing device based on a distributed configuration center in another embodiment;

[0073] Fig.17 It is a structural block diagram of a message processing device based on a distributed configuration center in another embodiment;

[0074] Fig.18 It is a structural block diagram of a message processing device based on a distributed configuration center in another embodiment;

[0075] Fig.19 FIG. 4 is a block diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein in the specification of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure.

[0078] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described here can be implemented in an order other than those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0079] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, in this specification, the term "and / or" includes any and all combinations of the relevant listed items.

[0080] The message processing method based on the distributed configuration center provided in the embodiment of the present disclosure can be applied to Figure 1 In the application environment shown. Among them, the message processing system 102 based on the distributed configuration center can be run on the server, and the server can be implemented by an independent server or a server cluster composed of multiple servers. The message processing system 102 is connected to the online transaction system 106 through the message middleware (such as the Kafka message queue 104), and the online transaction system 106 is used to send business messages to the message processing system 102. After the message processing system 102 processes the business message, it is sent to the application scenario through the Kafka data output interface 108, for example, it can be an application system (such as a real-time marketing system), an application program (such as a monitoring and early warning program, a real-time risk control program) or an application terminal (such as a real-time large screen display). The data asset system 101 can store the data that the message processing system 102 and the Kafka data output interface 108 need to store. The data asset system 101 can be integrated on the server 100, or it can be placed on the cloud or other network servers. The data asset system 101 can include an ElasticSearch module 103 and a ClickHouse (Click Stream, Data WareHouse, column storage database) module 105. Elasticsearch generally refers to a distributed search and analysis engine at the core of the Elastic Stack, which can be used for distributed full-text retrieval. ClickHouse module 105 can be used to retain detailed data of warnings for troubleshooting and subsequent query analysis scenarios.

[0081] Among them, the message processing system 102 includes a distributed configuration center 110 and a logic processing center 120. The distributed configuration center 110 includes a front-end portal 111, an Ignite storage module 112, an Oracle database 113 and a Zookeeper management component 114. The front-end portal 111 is used to connect to the client. The Ignite storage module 112 can be a memory-centric distributed database, cache system and computing platform, the purpose of which is to support data-intensive program development. The Ignite storage module 112 can support millions of transactions per second and PB-level memory data management. The Oracle database 113 can be a relational database management system for organizing data using a relational model. The Zookeeper management component 114 is used to provide distributed application coordination services, which can encapsulate complex and error-prone key services, and provide users with simple and easy-to-use interfaces and systems with high performance and stable functions.

[0082] The logic processing center 120 includes a distributed computing engine 121 and a Yarn resource manager 122. The distributed computing engine 121 is used to perform logical calculations on business messages. The distributed computing engine 121 can be implemented based on the Flink framework. The Yarn resource manager 122 can provide unified resource management and scheduling for upper-layer applications. Its introduction has brought great benefits to the cluster in terms of utilization, unified resource management, and data sharing.

[0083] In some embodiments of the present disclosure, Figure 2 As shown, a message processing method based on a distributed configuration center is provided, and the method is applied to Figure 1 Taking the message processing system 102 in the example as an example, the following steps are included:

[0084] Step S10: receiving configuration data transmitted by the client, and configuring the message parsing configuration information, the message adaptation configuration information and the message rule matching configuration information according to the configuration data.

[0085] Specifically, through the client network interface (such as Figure 1 The front-end portal 111 in the embodiment receives the configuration information transmitted by the client, that is, obtains the relevant configuration information of the user for the service message processing through the client. The configuration information may include message parsing configuration information, message adaptation configuration information and message rule matching configuration information.

[0086] Among them, the message parsing configuration information may refer to the configuration information for preprocessing the message, such as different protocol types, system sources, transaction codes, perception field paths and mapping information. Preprocessing may include processes such as implementing message structured parsing, transaction filtering, message body parsing, transaction adaptation and transaction matching. The message adaptation configuration information may refer to the configuration information of different processing logics corresponding to various business scenarios, for example. In the transaction scenarios of the financial industry, based on the diverse business scenario requirements, there are multiple processing logics corresponding to the different processing logics. There are also differences in the information that different processing logics need to extract from the message. The message adaptation configuration information can be used to divert the business message according to the business scenario and realize the rule matching docking with subsequent different scenarios. The message rule matching configuration information may refer to the configuration information of rule logic information (such as the screening and early warning output of the implementation rules) for the business needs of different business scenarios. The rule logic information may refer to the information for logical calculation of the business message, such as the large-amount monitoring business scenario, where different regulatory agencies have different amount judgments; for the risk control business scenario, it involves the corresponding search and matching of the customer number and account number.

[0087] Step S20: Acquire a service message, and parse and obtain a first target field of the service message based on the message parsing configuration information.

[0088] Specifically, by docking with the business system, the business message generated by the business system is obtained. The business message obtained in real time is parsed in real time. According to the message parsing configuration information, the protocol type of the business message can be determined according to the header, and then the corresponding operation is performed according to the message header protocol according to the text parsing configuration information to achieve the message header parsing of the business message. The message body of the business message is continued to be parsed according to the message parsing configuration information. Combined with the protocol of the message header, the message body parsing can extract the message body content according to different protocols, and finally extract the first target field from the business message. The first target field can be a field in the business message used to characterize the specific business information of the business message.

[0089] Step S30: determining a second target field in the first target field according to the message adaptation configuration information, and determining an adaptation category of the service message according to the second target field; a mapping relationship exists between the second target field and the adaptation category.

[0090] Specifically, the first target field in step S20 is obtained, for example, communication can be carried out through a Kafka queue. According to the message adaptation configuration information, the second target field is captured in the first target field, and the second target field can be a field in the business message used to characterize the business scenario information of the business message. Then, the adaptation category corresponding to the second target field is determined according to the message adaptation configuration information. For example, the business scenario can correspond to the adaptation category, and the business messages of different business scenarios are sent to the adaptation category corresponding to the business scenario. The adaptation category can be implemented through the Kafka topic queue, and each business scenario has a separate Kafka Topic.

[0091] Step S40: Determine rule logic information corresponding to the adaptation category based on the message rule matching configuration information.

[0092] Specifically, the adaptation category in step S30 is obtained, and the rule logic information corresponding to each adaptation category can be determined according to the message rule matching configuration information, that is, the rule logic information corresponding to the business scenario adapted by the business message can be determined.

[0093] Step S50: performing logic calculation processing on the service message according to the rule logic information.

[0094] Specifically, according to the determined rule logic information, the business message is logically calculated and processed. For example, the rule logic information includes "transaction amount transamt.mutiply(exchangeRate)>=1000, transamt is the transaction amount field, exchangeRate is the set exchange rate indicator". The meaning of this rule is to issue an early warning for the message with a transaction amount greater than 1000 in the business message.

[0095] In the above-mentioned message processing method based on the distributed configuration center, by obtaining the configuration information of the front end, the processing logic of the business message of the back end is acted on, and the business message is pre-processed according to the message parsing configuration information to extract the first target field of the business message, and then the second target field in the first target field is determined according to the message adaptation configuration information, and the adaptation category (business scenario) of the business message is determined according to the second target field, and the rule logic information corresponding to the adaptation category is further determined according to the message rule matching configuration information, and the business message is logically calculated and processed according to the rule logic information under the business message adaptation business scenario, thereby realizing the separation of the front end and the back end. The front end is used to integrate the configuration information, and the back end performs real-time logical processing on the business message according to the configuration information, thereby realizing real-time message processing and dynamic changes in the logical calculation requirements of the business message, and improving the efficiency and flexibility of business message processing.

[0096] In some embodiments of the present disclosure, Figure 3As shown, step S10 includes:

[0097] Step S12: Store the received configuration data into the configuration directory.

[0098] Specifically, the configuration data received from the client is stored in the form of a configuration directory to facilitate dynamic management of the configuration information. For example, the registration and service discovery functions of the distributed service management component Zookeeper can be used to write the front-end configuration information to the node directory (i.e., the configuration directory). In some embodiments, the front-end portal supports the client to add, query, invalidate, query historical records, and other operations on configuration information. Newly added configuration information records will be written to the configuration directory, and invalid configuration information records will delete the corresponding records in the configuration directory. Historical configuration information can be stored in a relational database (e.g. Figure 1 Oracle database 113).

[0099] Step S14: monitoring the configuration directory in real time, and in response to the configuration directory update event, determining the message parsing configuration information, the message adaptation configuration information and the message rule matching configuration information according to the updated configuration directory.

[0100] Specifically, the configuration directory is monitored in real time, and after an update event occurs in the configuration directory, in response to the update event of the configuration directory, the message parsing configuration information, the message adaptation configuration information, and the message rule matching configuration information are determined according to the updated configuration directory, that is, the updated message parsing configuration information, the message adaptation configuration information, and the message rule matching configuration information are used to perform logical processing on the business message. In some embodiments, the aforementioned steps are implemented by a Flink job flow, and the Flink job flow can dynamically obtain configuration information by subscribing to the configuration directory. When the Flink job flow starts, the initial configuration information can be preloaded, and a watcher can be registered at the same time. When the configuration directory is updated, the watcher triggers the action of notifying the Flink job that subscribes to the configuration directory, thereby realizing real-time synchronization of the front-end and back-end configurations.

[0101] This embodiment strengthens the real-time management of configuration information by establishing a configuration directory, and monitors the configuration directory in real time to promptly determine the message parsing configuration information, message adaptation configuration information and message rule matching configuration information according to the update of the configuration directory, thereby realizing real-time synchronization of front-end and back-end configurations.

[0102] In some embodiments of the present disclosure, Figure 4 As shown, step S20 includes:

[0103] Step S22: Read the configuration directory, and generate a parsing mapping table according to the message parsing configuration information.

[0104] Specifically, by reading the configuration directory, the message parsing configuration information in the configuration directory is obtained, and a parsing mapping table (such as a Parser configuration Mapping table, a Mapping table generally refers to a data structure that can be used to store mapping relationships) is generated according to the message parsing configuration information.

[0105] Step S24: Determine the protocol type of the service message, and determine the message parsing configuration information corresponding to the protocol type of the service message.

[0106] Specifically, the protocol type of the business message can be determined by receiving the business message of the message middleware as input and parsing the message header of the input business message. The message parsing configuration information is called based on the determined protocol type, for example, the Parser configuration is called. Parser generally refers to the process of converting a text (string) of a certain format into a certain data structure. By calling the Parser configuration corresponding to the protocol to complete the message header parsing, the message header information of the business message, such as the transaction code, can be obtained.

[0107] Step S26: extracting the first target field in the business message based on the parsing mapping table, and mapping the first target field to output details.

[0108] Specifically, based on the mapping relationship in the parsing mapping table, the first target field in the service message is extracted according to the protocol of the service message, and the first target field is mapped to the output details to achieve the output of the first target field.

[0109] This embodiment generates a parsing mapping table according to the message parsing configuration information, establishes a mapping relationship, improves the parsing efficiency of the service message, strengthens data management, and facilitates the reuse of mapping data.

[0110] In some embodiments of the present disclosure, Figure 5 As shown, step S20 also includes:

[0111] Step S252: Determine the transaction information and source information of the business message.

[0112] Specifically, the transaction information (eg, transaction code) and source information may be determined based on the message header information obtained in the aforementioned step S24. For example, the source information may be business system information of the message source determined based on the message header information.

[0113] Step S254: Screening is performed based on the transaction information and source information according to the message parsing configuration information, and the business messages that meet the screening rules are stored in the global database.

[0114] Specifically, according to the message parsing configuration information, the business messages are screened based on the transaction information and the source information. The message parsing configuration information includes the screening rules. The business messages that meet the screening rules are stored in the global database (e.g. Figure 1 Elastic Search module 103 in the system to facilitate historical data query.

[0115] Step S256: Store the business messages that meet the screening rules in the distributed memory database, match the request messages and response messages in the distributed memory database, and clean up the matched business messages and the unmatched business messages that exceed the preset time.

[0116] Specifically, the service messages that meet the screening rules are stored in a distributed memory database (e.g. Figure 1 The ignite storage module 112 in the distributed memory database is used to match the request message and the response message of the business message in the distributed memory database, that is, the request message and the response message in the distributed memory database are matched according to the matching rule, and the request response message is matched. The matching rule is included in the message parsing configuration information. The matching business messages and the business messages that have not been matched beyond the preset time are cleared.

[0117] This embodiment can filter out some business messages that do not need to be processed through message filtering, thereby reducing the processing burden, and by matching request messages and response messages in the distributed memory database, the distributed memory database can temporarily store message data for a recent period of time, realize asynchronous pairing of request messages and response messages, and promptly clean up business messages that have completed matching and business messages that have not completed matching within a preset time, thereby reducing memory consumption.

[0118] In some embodiments, Figure 6 The message preprocessing process shown also includes deserializing the input business message before step S22, and performing output serialization of the first target field after step S26. Serialization and deserialization may refer to the preservation and reconstruction of the state of the target object. Deserializing the input business message may refer to converting the transmission data state (e.g., byte sequence) of the input business message into the language description information (e.g., source code) of the business message. The output serialization of the first target field may refer to converting the language description information (e.g., source code) of the first target field into the transmission data state (e.g., byte sequence).

[0119] This embodiment improves the stability and efficiency of data transmission by deserializing the input service message and serializing the output of the first target field.

[0120] In some embodiments of the present disclosure, Figure 7 As shown, step S30 includes:

[0121] Step S32: Read the configuration directory, and generate an adaptation mapping table according to the message adaptation configuration information.

[0122] Specifically, by reading the configuration directory, the message adaptation configuration information in the configuration directory is obtained, and an adaptation mapping table (eg Adpter configuration Mapping table, Mapping table generally refers to a data structure that can be used to store mapping relationships) is generated according to the message adaptation configuration information.

[0123] Step S34: capturing the second target field in the first target field based on the adaptation mapping table, and distributing the service message to the corresponding adaptation category according to the second target field.

[0124] Specifically, based on the mapping relationship in the adaptation mapping table, the message adaptation configuration information is called. For example, the Adpter configuration is called. Adpter (adapter) generally refers to the interface of a class being transformed into an interface expected by the client, so that two classes that originally could not work together due to interface mismatch can work together. The second target field in the first target field is mapped by calling the Adpter configuration. Based on the mapping relationship in the adaptation mapping table, the service message is distributed to the corresponding adaptation category (e.g., Kafka Topic) according to the second target field. In some embodiments, such as Figure 8 As shown, before step S34, the input first target field is also deserialized, and after step S34, the output serialization of the adaptation category is performed, so that the adaptation category can be output to the corresponding business scenario of the business system.

[0125] This embodiment generates an adaptation mapping table according to the message adaptation configuration information, establishes a mapping relationship, improves the efficiency of adapting the service message to the service scenario, strengthens data management, and facilitates the reuse of mapping data.

[0126] In some embodiments of the present disclosure, Fig. 9 As shown, step S40 includes:

[0127] Step S42: Read the configuration directory, and generate a rule matching mapping table according to the message rule matching configuration information.

[0128] Specifically, the configuration directory is read to obtain the rule matching configuration information in the configuration directory, and a rule matching mapping table is generated according to the rule matching configuration.

[0129] Step S44: Determine rule logic information associated with the adaptation category based on the rule matching mapping table.

[0130] Specifically, each adaptation category may be associated with corresponding rule logic information, and the rule logic information may be used to instruct corresponding logic calculation processing to be performed on the service message. For example, the service message may be screened and early warning judged through the rule logic information.

[0131] This embodiment generates a rule matching mapping table according to the message rule matching configuration information, establishes a mapping relationship, improves the efficiency of logical calculation processing corresponding to the service message, and can strengthen message screening and warning issuance.

[0132] In some embodiments, Fig.10 As shown, the message rule matching configuration information may include rule configuration information, indicator configuration information, indicator query configuration information, indicator data source configuration information, warning output configuration information, etc. Taking the adaptation category outputted in step S30 as input, by loading the message rule matching configuration information in the configuration directory, a rule configuration mapping table, an indicator configuration mapping table, an indicator query configuration mapping table, an indicator data source configuration mapping table, and a warning output configuration mapping table are generated. The above mapping tables. Among them, the indicator may refer to a parameter in a logic rule, and the rule configuration may be used through the indicator configuration. For example, if a logic rule is "transaction amount transamt.mutiply(exchangeRate)>=1000, transamt is the transaction amount field, and exchangeRate is the set exchange rate indicator", then its indicator configuration may be the configuration of the transaction amount. In some business scenarios such as foreign currency exchange to local currency, amount setting, customer number account matching, etc., the indicator may be an amount indicator, an account matching query indicator, and a currency exchange rate conversion indicator.

[0133] Among them, the indicator configuration information also includes the pointing information of the indicator query configuration mapping table and the indicator data source configuration mapping table. The query statement and the indicator data source address can be loaded through the indicator query configuration mapping table, and then the external data table can be connected for query. In this embodiment, the external data table can refer to a data table other than the current business message data. The external data table can be stored in a distributed memory database, such as Figure 1 In some embodiments, in a business scenario where a Flink job needs to perform a search for an account number based on a customer number, the rule configuration mapping table may first point to the indicator configuration information (e.g., indicator configuration ID), with the customer number as a parameter, and the corresponding indicator configuration information is queried through the indicator configuration mapping table, and then the customer number is used as a query input, and the data source address in the query indicator data source configuration mapping table is queried, and the queried information is synchronized to the Flink job to complete the query output of the account number.

[0134] Among them, the warning output configuration information is used to configure the warning output field mapping, set the access field name according to the field access requirements, and realize the access of the warning field. Furthermore, when the message data in the adaptation category is obtained, the message data is first deserialized. Determine the indicators according to the indicator configuration mapping table, the indicator query configuration mapping table, and the indicator data source configuration mapping table, and determine the rule logic information according to the rule configuration mapping table combined with the indicators, and perform logical calculations on the message data (such as rule judgment and screening) according to the logical rule logic information. Map the fields in the report data according to the warning output configuration mapping table to obtain the warning sending field. The warning sending field is serialized and output as input for subsequent scenarios. The warning sending field can also be stored in Figure 1 ClickHouse module 105 in .

[0135] This embodiment can be used in conjunction with rule configuration through indicator configuration, which greatly improves the reusability of the rules and realizes the function of issuing warning fields.

[0136] In some embodiments, this method realizes the preprocessing, scenario classification and rule matching process of business messages, combined with Fig.11 As shown in the message processing flow diagram, the distributed configuration center 110 can receive the configuration information of the client (message parsing configuration information, message adaptation configuration information and message rule matching configuration information), and establish a configuration directory through the Zookeeper management component 114. The pre-processing Flink job, the scenario adaptation Flink job, and the business scenario Flink job that needs to perform logical processing on the business message (such as the Flink job of the large amount of funds change scenario, the Flink job of the sales scenario, the Flink job of the real-time risk control scenario, etc.) in the logic processing center 120 perform corresponding processing according to the real-time configuration information by monitoring the configuration directory in the Zookeeper management component 114 in real time. Flink jobs can communicate with each other through Kafka.

[0137] Furthermore, combined with Fig.12The business message configuration process diagram shown in the figure explains the message perception processing steps. After the message processing system 102 receives the business message, the distributed configuration center 110 first determines the message parsing configuration information. The pre-processing Flink job can perceive the message parsing configuration information in real time, and filter and filter according to the configuration content, and extract the first target field of the business message. Secondly, the distributed configuration center 110 determines the message adaptation configuration information. The scenario adaptation Flink job determines the second target field in the first target field according to the message adaptation configuration information, and determines the adaptation category of the business message according to the second target field, and puts it into the corresponding business scenario Topic. Subsequently, the distributed configuration center 110 determines the message rule matching configuration information. If indicators are needed, determine the indicator configuration information, indicator query configuration information, indicator data source configuration information, etc. The business scenario Flink job is executed according to the configuration content of the message rule matching configuration information to realize the logical calculation processing of the business message. Finally, the distributed configuration center 110 configures the final output field in the Action (the Action class is a bridge between user requests and business logic, and each Action acts as a business agent for the customer), and the business scenario Flink job outputs the processing results of the business message.

[0138] This embodiment can process messages in real time, and realize synchronous dynamic changes of back-end job logic according to the front-end configuration information, and the modification is flexible. For example, for the same business requirement, the same message preprocessing process and message warning sending field are available, but the rule logic is different. In this way, the rules can be changed quickly, and other configuration information can be reused. Or, for different business requirements, the message preprocessing process and message warning sending field are different, but the rule logic is the same. The same rule logic can be used without adding anything in the rule configuration, and the upper layer can conveniently and efficiently operate the business requirement changes without going deep into the underlying code, thereby improving the continuity and flexibility of the configuration.

[0139] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0140] Based on the same inventive concept, the embodiment of the present disclosure also provides a message processing device based on a distributed configuration center for implementing the message processing method based on a distributed configuration center involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more embodiments of the message processing device based on a distributed configuration center provided below can refer to the limitations of the message processing method based on a distributed configuration center above, and will not be repeated here.

[0141] The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiments of this specification and a device in combination with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiments of the present disclosure is as described in the following embodiments. Since the implementation scheme and method for solving the problem of the device are similar, the implementation of the specific device in the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware for predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0142] In some embodiments of the present disclosure, Fig.13 As shown, a message processing device based on a distributed configuration center is provided, and the device may be the server running the message processing system 102, or a module, component, device, unit, etc. integrated in the server. The device Z00 may include:

[0143] Configuration module Z10, used to receive configuration data transmitted by the client, and configure the message parsing configuration information, message adaptation configuration information and message rule matching configuration information according to the configuration data;

[0144] A message acquisition module Z20, used for acquiring a service message, and obtaining a first target field of the service message based on the message parsing configuration information;

[0145] An adaptation category module Z30, configured to determine a second target field in the first target field according to the message adaptation configuration information, and determine an adaptation category of the service message according to the second target field; a mapping relationship exists between the second target field and the adaptation category;

[0146] A rule determination module Z40, configured to determine rule logic information corresponding to the adaptation category based on the message rule matching configuration information;

[0147] The processing module Z50 is used to perform logical calculation processing on the service message according to the rule logic information.

[0148] In some embodiments of the present disclosure, Fig.14 As shown, the configuration module Z10 includes:

[0149] A configuration directory unit Z12, used to store the received configuration data into a configuration directory;

[0150] The monitoring unit Z14 is used to monitor the configuration directory in real time, and in response to the configuration directory update event, determine the message parsing configuration information, message adaptation configuration information and message rule matching configuration information according to the updated configuration directory.

[0151] In some embodiments of the present disclosure, Fig.15 As shown, the message acquisition module Z20 includes:

[0152] The parsing mapping table unit Z22 is used to read the configuration directory and generate a parsing mapping table according to the message parsing configuration information.

[0153] The protocol type unit Z24 is used to determine the protocol type of the service message and determine the message parsing configuration information corresponding to the protocol type of the service message;

[0154] The first target mapping unit Z26 is used to extract the first target field in the service message based on the parsing mapping table, and map the first target field to output details.

[0155] In some embodiments of the present disclosure, Fig.16 As shown, the message acquisition module Z20 also includes:

[0156] A message header unit Z21, used to determine the transaction information and source information of the business message;

[0157] A screening unit Z23, configured to screen the business messages that meet the screening rules based on the transaction information and the source information according to the message parsing configuration information, and store the business messages that meet the screening rules into a global database;

[0158] The memory cleaning unit Z25 stores the business messages that meet the screening rules into the distributed memory database, matches the request messages and response messages of the business messages in the distributed memory database, and cleans up the matched business messages and the unmatched business messages that exceed the preset time.

[0159] In some embodiments of the present disclosure, Fig.17 As shown, the adaptation category module Z30 includes:

[0160] An adaptation mapping table unit Z32 is used to read the configuration directory and generate an adaptation mapping table according to the message adaptation configuration information;

[0161] The second target mapping unit Z34 is used to capture the second target field in the first target field based on the adaptation mapping table, and distribute the service message to the corresponding adaptation category according to the second target field.

[0162] In some embodiments of the present disclosure, Fig.18 As shown, the rule determination module Z40 includes:

[0163] A rule matching mapping table unit Z42 is used to read the configuration directory and generate a rule matching mapping table according to the message rule matching configuration information;

[0164] The third target mapping unit Z44 is used to determine the rule logic information associated with the adaptation category based on the rule matching mapping table.

[0165] Each module in the above-mentioned message processing device based on the distributed configuration center can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules. It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0166] Based on the above-mentioned description of the embodiment of the message processing method based on the distributed configuration center, in another embodiment provided by the present disclosure, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Fig.19 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a message processing method based on a distributed configuration center is implemented.

[0167] Those skilled in the art will understand that the structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] Based on the aforementioned description of the embodiment of the message processing method based on the distributed configuration center, in another embodiment provided in the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0169] Based on the aforementioned description of the embodiment of the message processing method based on the distributed configuration center, in another embodiment provided in the present disclosure, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0172] In the description of this specification, the description with reference to the terms "some embodiments", "other embodiments", "ideal embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example.

[0173] It is understandable that the various embodiments of the above method in this specification are described in a progressive manner, and the same / similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related parts, refer to the description of other method embodiments.

[0174] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features of the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the patent of the present disclosure shall be subject to the attached claims.

Claims

1. A FLINK job real-time message processing method based on a distributed configuration center, characterized in that: Applied to a message processing system, the message processing system includes a distributed configuration center and a logic processing center, and the method includes: Receiving configuration data based on the distributed configuration center and storing it in a configuration directory of a management component of the distributed configuration center; The Flink job based on the logic processing center monitors the configuration directory in real time, responds to the configuration directory update event, and determines the message parsing configuration information, message adaptation configuration information, and message rule matching configuration information according to the updated configuration directory; Acquire a service message, and parse and obtain a first target field of the service message based on the message parsing configuration information; Determine a second target field in the first target field according to the message adaptation configuration information, and determine an adaptation category of the service message according to the second target field; there is a mapping relationship between the second target field and the adaptation category; Determine the rule logic information corresponding to the adaptation category based on the message rule matching configuration information; Perform logical calculation processing on the service message according to the rule logic information.

2. The method according to claim 1, characterized in that The acquiring of the service message, and obtaining the first target field of the service message based on the message parsing configuration information, comprises: Read the configuration directory and generate a parsing mapping table according to the message parsing configuration information; Determine the protocol type of the service message, and determine the message parsing configuration information corresponding to the protocol type of the service message; A first target field in the service message is extracted based on the parsing mapping table, and the first target field is mapped to output details.

3. The method according to claim 2, characterized in that The acquiring of the service message, and obtaining the first target field of the service message based on the message parsing configuration information also includes: Determining transaction information and source information of the business message; Screening based on the transaction information and source information according to the message parsing configuration information, and storing the business messages that meet the screening rules in the global database; The business messages that meet the screening rules are stored in a distributed memory database, and the business messages in the distributed memory database are matched with request messages and response messages, and the matched business messages and the business messages that have not been matched within a preset time are cleared.

4. The method according to claim 1, characterized in that: The determining the second target field in the first target field according to the message adaptation configuration information, and determining the adaptation category of the service message according to the second target field includes: Read the configuration directory and generate an adaptation mapping table according to the message adaptation configuration information; A second target field in the first target field is captured based on the adaptation mapping table, and the service message is distributed to the corresponding adaptation category according to the second target field.

5. The method according to claim 1, characterized in that The determining of the rule logic information corresponding to the adaptation category based on the message rule matching configuration information includes: Read the configuration directory and generate a rule matching mapping table according to the message rule matching configuration information; Rule logic information associated with the adaptation category is determined based on the rule matching mapping table.

6. A FLINK job real-time message processing device based on a distributed configuration center, characterized in that: Applied to a message processing system, the message processing system includes a distributed configuration center and a logic processing center, and the device includes: A configuration module, configured to receive configuration data based on the distributed configuration center and store the configuration data in a configuration directory of a management component of the distributed configuration center; The Flink job based on the logic processing center monitors the configuration directory in real time, responds to the configuration directory update event, and determines the message parsing configuration information, message adaptation configuration information, and message rule matching configuration information according to the updated configuration directory; A message acquisition module, used to acquire a service message, and parse and obtain a first target field of the service message based on the message parsing configuration information; An adaptation category module, used to determine a second target field in the first target field according to the message adaptation configuration information, and determine an adaptation category of the service message according to the second target field; there is a mapping relationship between the second target field and the adaptation category; A rule determination module, used to determine the rule logic information corresponding to the adaptation category based on the message rule matching configuration information; A processing module is used to perform logical calculation processing on the service message according to the rule logic information.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.