A transaction subscription center data flow conversion method and device based on Pulsar Function, an electronic device, and a medium

Abstract

The application discloses a transaction subscription center data flow conversion method and device based on Pulsar Function, electronic equipment and medium, comprising: acquiring each business message generated by an upstream business system, and writing each business message into different Topic subjects of a Pulsar message queue in real time according to different types of the business messages; using a Function component to pull a subscription relationship registered or changed by a downstream client, and routing data in each Topic subject that meets the subscription relationship to a corresponding downstream Topic subject after processing, so that the downstream client actively consumes data in the corresponding downstream Topic subject of the Pulsar message queue according to the subscription relationship. The application can greatly improve the expansibility, stability and performance of the transaction subscription center service.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a data transfer method, apparatus, electronic device, and medium for a transaction subscription center based on Pulsar Function. Background Technology

[0002] With the rapid development of the mobile internet, the scale of data is growing exponentially, reaching petabyte (PB) levels or even larger daily. Business systems require efficient and reliable data processing methods. Similarly, the securities industry faces similar challenges during its comprehensive internetization and informatization process. For example, the entire chain of securities issuance, pricing, sales, trading, and services, covering traditional securities businesses such as brokerage, wealth management product sales, securities issuance and underwriting, derivatives trading, proprietary trading, and margin trading, all require solutions to the problem of large-scale data flow between various business systems.

[0003] The service hours of securities industry systems coincide with securities trading hours, requiring extremely high data real-time performance. Requests are also highly concentrated, currently handling tens of thousands of data points per second, and this volume continues to grow rapidly. To improve business processing capabilities, a transaction subscription center based on message middleware technology (subscription microservices) is typically used to facilitate data flow between services, such as... Figure 1 As shown, upstream business messages, such as Business 1 and Business 2, are written to the message queue in real time. The subscription center microservice extracts the corresponding consumption data from the message queue according to the subscription relationships registered by downstream subscription businesses, such as Business A, Business B, and Business C. The subscription relationship includes information such as data filtering rules and subscription business service addresses. The extracted data is processed according to the data filtering rules and then pushed to the corresponding downstream subscription business via the TCP protocol.

[0004] However, this existing data flow method is highly dependent on the reliability and performance of the subscription center microservice, and it suffers from several problems: First, there is the issue of system scalability. The subscription center microservice is highly coupled with downstream subscription services, making it a hotspot service. Upgrades and feature expansions can introduce instability risks, and upgrades to the subscription center microservice typically take weeks. Second, there is the issue of reliability. The subscription center microservice uses a push model to deliver messages to downstream subscription services, making exceptions very difficult to handle and uncontrollable. For example, downstream subscription service processing failures, push timeouts, and reconnections usually require bypass methods for remediation. Upgrading downstream services requires manual data recovery, and in some cases, recovery is impossible, increasing system complexity and uncertainty. Third, there is the issue of performance. As the number of downstream subscribers increases, message push processing encounters performance bottlenecks. The number of subscription center microservices is usually limited by the number of message queue partitions, preventing true horizontal scaling. Furthermore, slow message processing in some downstream services can slow down the entire subscription message forwarding process. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data flow method, apparatus, electronic device and medium for transaction subscription centers based on Pulsar Function, which can significantly improve the stability, scalability and performance of transaction subscription center services.

[0006] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0007] In a first aspect, the present invention provides a data flow method for a transaction subscription center based on Pulsar Function, the method comprising:

[0008] It acquires various business messages generated by upstream business systems and writes them into different topic bodies of the Pulsar message queue in real time according to the different types of each business message;

[0009] The Function component is used to retrieve subscription relationships registered or changed by downstream clients. Data in each Topic body that matches the subscription relationship is processed and routed to its corresponding downstream Topic body, so that the downstream client can actively consume data of its corresponding downstream Topic body in the Pulsar message queue according to the subscription relationship.

[0010] In conjunction with the first aspect, preferably, the registration or modification of the subscription relationship is implemented based on a DB database and a distributed Redis cache.

[0011] In conjunction with the first aspect, preferably, the subscription relationship includes the original Topic name, data filtering rules, target Topic name, and route ID.

[0012] In conjunction with the first aspect, preferably, the step of using the Function component to retrieve subscription relationships registered or changed by downstream clients, and then processing the data in each Topic body that conforms to the subscription relationship and routing it to its corresponding downstream Topic body includes the following steps:

[0013] Based on the original Topic name registered by the downstream client, obtain the set of subscription rules from the distributed Redis cache through fuzzy matching;

[0014] Based on the aforementioned set of subscription rules, monitor for changes in specific rule route IDs in real time.

[0015] Subscribe via broadcast mode and update the target Topic name based on the changed route ID;

[0016] Based on the updated target Topic name, extract data from the Topic body to which it belongs in the Pulsar message queue;

[0017] After processing the extracted data according to the data filtering rules, it is routed to its corresponding downstream Topic based on the changed route ID.

[0018] In conjunction with the first aspect, preferably, the Function component is deployed on a K8S container platform.

[0019] In conjunction with the first aspect, preferably, the downstream client actively consumes data from its corresponding downstream Topic body in the Pulsar message queue according to the subscription relationship, including:

[0020] The message queue in Pulsar is retrieved using a long polling method, and the consumption rate is dynamically adjusted.

[0021] Downstream clients establish connections with the Pulsar message queue server via the TCP protocol.

[0022] In conjunction with the first aspect, preferably, the step of using long polling to retrieve messages and dynamically adjusting the consumption rate includes:

[0023] The downstream client sends a pull request to the Pulsar message queue server, and the pull request includes the number of messages to be pulled in this batch.

[0024] The Pulsar message queue server queries whether there are new messages in the downstream Topic body corresponding to the pull request;

[0025] If there is a new message, it will be pushed to the downstream client via the TCP protocol, so that the downstream client can process the business logic according to the received new message and continuously send pull requests to the Pulsar message queue server during the processing of this batch of messages. The number of messages pulled is dynamically adjusted according to the current processing time.

[0026] When the downstream client successfully processes a new message, the Pulsar message queue server marks the new message as successfully processed according to the success instruction, and updates and persists the consumption offset so that when the downstream client resumes consumption due to a crash, it can find the position of the last consumption and continue consumption.

[0027] When the downstream client fails to process a new message, the Pulsar message queue server marks the new message as a re-push according to the failure instruction and re-consumes it according to the re-push policy. If the number of re-consumptions exceeds the configured number of retries, the new message is delivered to the dead letter queue.

[0028] If there are no new messages, the Pulsar message queue server will hold the current fetch request to avoid invalid requests.

[0029] Secondly, the present invention provides a data transfer device for a transaction subscription center based on Pulsar Function, the device comprising:

[0030] The acquisition module is used to acquire various business messages generated by the upstream business system and write them into different Topic bodies of the Pulsar message queue in real time according to the different types of each business message.

[0031] The routing module is used to use the Function component to pull the subscription relationships registered or changed by downstream clients, and to process the data in each Topic body that conforms to the subscription relationship and then route it to its corresponding downstream Topic body, so that the downstream client can actively consume the data of its corresponding downstream Topic body in the Pulsar message queue according to the subscription relationship.

[0032] Thirdly, the present invention provides an electronic device, including a processor and a storage medium;

[0033] The storage medium is used to store instructions;

[0034] The processor is configured to operate according to the instructions to execute the steps of the Pulsar Function-based transaction subscription center data flow method as described in any of the first aspects.

[0035] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the transaction subscription center data flow method based on the Pulsar Function as described in any of the first aspects.

[0036] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0037] This invention uses a Function component to process data from upstream Topic entities and route it to downstream Topic entities in the Pulsar message queue for temporary storage. Downstream clients actively consume data from downstream Topic entities based on their subscription relationships, completely decoupling the subscription relationships and message forwarding of the transaction subscription center. This avoids the strong coupling problem between the subscription center microservice and downstream subscription business in existing technologies, improves the stability of the entire transaction subscription center service, reduces the probability of errors, and effectively prevents the loss of subscription data due to downstream client business upgrades. At the same time, leveraging the high-performance Pub / Sub processing characteristics of the Pulsar message queue itself, the overall service performance is improved by about 10 times, from 10,000 / s to 100,000 / s. Furthermore, this invention has no performance bottlenecks, can be horizontally scaled to support greater concurrency, and can adapt to the long-term goals of subsequent business development. Attached Figure Description

[0038] Figure 1 This is a logical diagram illustrating data flow based on subscription microservices in existing technologies.

[0039] Figure 2 A logical diagram illustrating the data flow of a transaction subscription center based on Pulsar Function, provided for an embodiment of the present invention;

[0040] Figure 3 This is a logical diagram illustrating message routing when the subscription relationship changes, provided as an embodiment of the present invention. Detailed Implementation

[0041] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0042] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0043] Example 1:

[0044] This invention provides a data flow method for a transaction subscription center based on Pulsar Functions, which specifically includes the following steps:

[0045] Step 1: Obtain the business messages generated by the upstream business system and write them into the different topic bodies of the Pulsar message queue in real time according to the different types of each business message;

[0046] Step 2: Use the Function component to pull the subscription relationships registered or changed by downstream clients, and process the data in each Topic body that matches the subscription relationship before routing it to its corresponding downstream Topic body. This allows downstream clients to actively consume data from their corresponding downstream Topic bodies in the Pulsar message queue based on the subscription relationship.

[0047] It should be further explained that the subscription relationship is the subscription configuration that establishes a relationship between upstream business messages and downstream client businesses. Registration and modification of subscription relationships are low-frequency operations. This embodiment utilizes a DB database and a distributed Redis cache to implement subscription relationship registration and retrieval, ensuring data reliability and real-time performance. Furthermore, leveraging the Pub / Sub capabilities of the Pulsar message queue itself, downstream clients can directly consume data relevant to their business through Topics, achieving complete decoupling of consumption and forwarding between downstream business services and the subscription center's microservices. Based on the high consistency, high reliability, and high performance characteristics of the Pulsar message queue, performance and reliability issues in existing technologies are resolved. Downstream clients can start the appropriate number of service processes according to actual needs. For abnormal situations such as consumption failures and disconnections, the persistent management of consumption offsets and failure retry strategies of the Pulsar message queue ensure that messages are always processed by the downstream business at least once.

[0048] Specifically, such as Figure 1The diagram illustrates the data flow process of the Pulsar Function transaction subscription center provided by this invention. Users initiate the registration and modification of subscription relationships through the console. Subscription relationship data is stored in the DB database and the distributed Redis cache. Business messages are generated by upstream business systems, such as Business 1 and Business 2, and are written in real time to different topics in the Pulsar message queue, such as Topic 1 and Topic 2. Utilizing the data processing and data routing capabilities of Pulsar Function, upstream data processing is routed to the corresponding downstream topic bodies, such as Topic A, Topic B, and Topic C, based on the subscription relationship. The Pulsar component pulls subscription relationship data from the DB database and the distributed Redis cache. Downstream client businesses, such as Business A, Business B, and Business C, actively consume data from the corresponding downstream topic bodies, such as Topic A, Topic B, and Topic C, in the Pulsar message queue according to their own business rules (subscription relationships).

[0049] As one embodiment of the present invention, in practice, subscription relationships are added, deleted, and modified by business personnel through a front-end control page. The relationship configuration is persisted to a DB database, and high-frequency query capabilities are provided through a distributed Redis cache. The subscription relationship includes the original Topic name, data filtering rules, target Topic name, and route ID. The code logic is as follows:

[0050]

[0051] The key format stored in Redis is inputTopic + "_" + id, and the value is the JSON string mentioned above.

[0052] As an embodiment of the present invention, refer to Figure 3 As shown, step 2 provided in this embodiment specifically includes:

[0053] Step a: Obtain the subscription rule set from the distributed Redis cache using fuzzy matching based on the original Topic name registered by the downstream client.

[0054] Step b: Based on the aforementioned set of subscription rules, monitor for changes in specific rule routing IDs in real time;

[0055] Step c: Subscribe via broadcast mode and update the target Topic name based on the changed route ID;

[0056] Step d: Extract data from the Pulsar message queue to the subject body of the target topic based on the updated target topic name;

[0057] Step e: After processing the extracted data according to the data filtering rules, route it to its corresponding downstream Topic based on the changed route ID.

[0058] Specifically, 1) the code logic for message routing is as follows:

[0059] (1) Function task initialization;

[0060] (2) Get the rule set from Redis according to the command get inputTopic+"_*".

[0061] (3) The Function task subscribes to rule_change_topic and listens for rule changes.

[0062] (4) If there are any changes, the Function task receives a notification of the specific rule ID change ((nputTopic + "_" + id and action). If the action is modification or addition, the update is pulled from Redis using the key. If it is deletion, the rule is deleted.

[0063] 2) Routing logic:

[0064] (1) Get the message content inputTopic.

[0065] (2) Determine whether the input meets the rule requirements based on the rule expression.

[0066] (3) If satisfied, the message is forwarded to outputTopic.

[0067] (4) If not satisfied, skip.

[0068] As an embodiment of the present invention, because the deployment, high availability, and performance of the message routing service are very important, the Function task needs to have horizontal scaling and service monitoring capabilities. Therefore, the Function component provided in this embodiment of the present invention is deployed on the K8S container platform. It can ensure the parallelism of the message routing service, start the corresponding number of Pods in a declarative manner according to the current needs, and the Pods are monitored and scheduled by the K8S container. When a Pod crashes, it can be quickly restarted to ensure that the number of currently running Pods meets the requirements. It can also realize the rapid scaling of the message routing service. When the original data fluctuates and grows, the message routing service processing may encounter bottlenecks. By monitoring the ratio of the original data sending rate to the message routing processing rate, if it exceeds the set threshold, the process of scaling up the message routing service Pod is triggered to achieve dynamic elastic scaling.

[0069] Furthermore, in this embodiment, the downstream client's message subscription is based on a shared subscription model, creating consumers. The number of consumers or business services is dynamically adjusted according to the processing time of business logic, message size, and latency requirements. Consumers collaborate to process messages, achieving horizontal scaling. In addition, the downstream client establishes a connection with the Pulsar message queue server via TCP protocol, using long polling to pull messages and dynamically adjust the consumption rate. The specific process is as follows:

[0070] A: The downstream client sends a pull request to the Pulsar message queue server, and the pull request includes the number of messages to be pulled in this batch;

[0071] B: The Pulsar message queue server queries whether there are new messages in the downstream Topic body corresponding to the pull request;

[0072] C: If there is a new message, the new message is pushed to the downstream client via the TCP protocol; the downstream client processes the business logic according to the received new message, and continuously sends pull requests to the Pulsar message queue server during the processing of this batch of messages, and the number of messages pulled is dynamically adjusted according to the current processing time.

[0073] D: When the downstream client successfully processes a new message, the Pulsar message queue server marks the new message as successfully processed according to the success instruction, and updates and persists the consumption offset so that when the downstream client resumes consumption due to a crash, it can find the position of the last consumption and continue consumption.

[0074] E: When the downstream client fails to process a new message, the Pulsar message queue server marks the new message as a re-push according to the failure instruction and re-consumes it according to the re-push strategy. If the number of re-consumptions exceeds the configured number of retries, the new message is delivered to the dead letter queue.

[0075] F: If there are no new messages, the Pulsar message queue server will hold this pull request to avoid invalid requests.

[0076] In summary, the Pulsar Function-based data transfer method for transaction subscription centers provided in this invention can significantly improve the scalability, stability, and performance of transaction subscription center services. It can ensure high consistency and high reliability of transaction subscription center services, guarantee that data is not lost or disordered, and has multi-replica disaster recovery capabilities. At the same time, it takes into account high performance and high availability, has the ability to support massive topics, and the number of consumers does not depend on the number of topic partitions. It can meet the needs of high-traffic and complex business scenarios and is suitable for widespread application.

[0077] Example 2:

[0078] like Figure 2 As shown, this embodiment of the invention provides a data transfer device for a transaction subscription center based on Pulsar Function, which can be used to implement the method described in Embodiment 1. The device includes:

[0079] The acquisition module is used to acquire various business messages generated by the upstream business system and write them into different Topic bodies of the Pulsar message queue in real time according to the different types of each business message.

[0080] The routing module is used to use the Function component to pull the subscription relationships registered or changed by downstream clients, and to process the data in each Topic body that conforms to the subscription relationship and then route it to its corresponding downstream Topic body, so that the downstream client can actively consume the data of its corresponding downstream Topic body in the Pulsar message queue according to the subscription relationship.

[0081] The Pulsar Function-based transaction subscription center data transfer device provided in this embodiment of the invention is based on the same technical concept as the Pulsar Function-based transaction subscription center data transfer method provided in Embodiment 1, and can produce the beneficial effects described in Embodiment 1. For the contents not described in detail in this embodiment, please refer to Embodiment 1.

[0082] Example 3:

[0083] This invention provides an electronic device, including a processor and a storage medium;

[0084] The storage medium is used to store instructions;

[0085] The processor is configured to operate according to instructions to perform steps according to any of the methods in Embodiment 1.

[0086] Example 4:

[0087] This invention provides a computer-readable storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of any of the methods in Embodiment 1.

[0088] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data flow method for a transaction subscription center based on Pulsar Function, characterized in that, The method includes: It acquires various business messages generated by upstream business systems and writes them into different topic bodies of the Pulsar message queue in real time according to the different types of each business message; The Function component is used to retrieve subscription relationships registered or changed by downstream clients. Data in each Topic body that matches the subscription relationship is processed and routed to its corresponding downstream Topic body, so that the downstream client can actively consume data of its corresponding downstream Topic body in the Pulsar message queue according to the subscription relationship. The registration or modification of the subscription relationship is implemented based on the DB database and distributed Redis cache; The subscription relationship includes the original Topic name, data filtering rules, target Topic name, and route ID; The process of using the Function component to retrieve subscription relationships registered or changed by downstream clients, and then processing the data in each Topic body that matches the subscription relationship and routing it to its corresponding downstream Topic body includes the following steps: Based on the original Topic name registered by the downstream client, obtain the set of subscription rules from the distributed Redis cache through fuzzy matching; Based on the aforementioned set of subscription rules, monitor for changes in specific rule route IDs in real time. Subscribe via broadcast mode and update the target Topic name based on the changed route ID; Based on the updated target Topic name, extract data from the Topic body to which it belongs in the Pulsar message queue; After processing the extracted data according to the data filtering rules, it is routed to its corresponding downstream Topic based on the changed route ID.

2. The data flow method for a transaction subscription center based on Pulsar Function according to claim 1, characterized in that, The Function component is deployed on the K8S container platform.

3. The data flow method for a transaction subscription center based on Pulsar Function according to claim 1, characterized in that, The downstream client actively consumes data from its corresponding downstream Topic in the Pulsar message queue according to the subscription relationship, including: pulling messages from the Pulsar message queue using a long polling method and dynamically adjusting the consumption rate; wherein, the downstream client establishes a connection with the Pulsar message queue server through the TCP protocol.

4. The data flow method for a transaction subscription center based on Pulsar Function according to claim 3, characterized in that, The steps of using long polling to retrieve messages and dynamically adjusting the consumption rate include: The downstream client sends a pull request to the Pulsar message queue server, and the pull request includes the number of messages to be pulled in this batch. The Pulsar message queue server queries whether there are new messages in the downstream Topic body corresponding to the pull request; If there is a new message, it will be pushed to the downstream client via the TCP protocol, so that the downstream client can process the business logic according to the received new message and continuously send pull requests to the Pulsar message queue server during the processing of this batch of messages. The number of messages pulled is dynamically adjusted according to the current processing time. When the downstream client successfully processes a new message, the Pulsar message queue server marks the new message as successfully processed according to the success instruction, and updates and persists the consumption offset so that when the downstream client resumes consumption due to a crash, it can find the position of the last consumption and continue consumption. When the downstream client fails to process a new message, the Pulsar message queue server marks the new message as a re-push according to the failure instruction and re-consumes it according to the re-push policy. If the number of re-consumptions exceeds the configured number of retries, the new message is delivered to the dead letter queue. If there are no new messages, the Pulsar message queue server will hold the current fetch request.

5. A data transfer device for a transaction subscription center based on Pulsar Function, characterized in that, The device includes: The acquisition module is used to acquire various business messages generated by the upstream business system and write them into different Topic bodies of the Pulsar message queue in real time according to the different types of each business message. The routing module is used to use the Function component to pull the subscription relationships registered or changed by downstream clients, and to process the data in each Topic body that conforms to the subscription relationship and then route it to its corresponding downstream Topic body, so that the downstream client can actively consume the data of its corresponding downstream Topic body in the Pulsar message queue according to the subscription relationship.

6. An electronic device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the Pulsar Function-based transaction subscription center data flow method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the transaction subscription center data flow method based on any one of claims 1 to 4.

Images