A market delay method and system based on streaming system

By deploying NTP services in kafka cluster and market delay service system, using producer threads, consumer threads and delay threads to work together, the problems of excessive memory consumption and complex processing logic caused by large market data in the existing technology are solved. It implements a general delay market processing solution, supports delay in any streaming message format, and maintains the consistency of downstream consumer logic.

CN114661490BActive Publication Date: 2024-11-29FEIMA FINANCIAL TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210167705.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-23
Publication Date
2024-11-29
Estimated Expiration
2042-02-23

AI Technical Summary

Technical Problem

When processing market data, it is difficult for the existing technology to effectively solve the problems of excessive memory consumption and complex processing logic caused by the large amount of data in a single market, especially when it is necessary to generate delayed markets.

Method used

By deploying the network time synchronization service NTP in the kafka cluster and the quotation delay service system, the producer thread, consumer thread and delay thread work together to realize the delay processing of real-time quotation messages, ensuring that message delay is completed based on time rather than content related.

Benefits of technology

Implement a general delay market solution that is independent of message content, supports delay in any streaming message format, read delay intervals through configuration files, meet various time delay needs, and downstream consumers are logically consistent when handling real-time market and delayed market conditions. , no code tweaking is required.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114661490B_ABST
    Figure CN114661490B_ABST
Patent Text Reader

Abstract

The present invention provides a market delay method and system based on a streaming system, and the method comprises the following steps: S1, deploying a network time synchronization service NTP in a Kafka cluster and a market delay service system; S2, a producer thread writes data to the Kafka cluster; S3, a consumer thread connects to the Kafka cluster and sets real-time market topic information to be consumed; S4, a decoding service system decodes messages of the Kafka cluster to generate real-time market message data, and the real-time market message data is read by the consumer thread through the Kafka cluster; S5, a delay thread obtains a time t1 corresponding to a current message, obtains a local current time t2, and when it is determined that the value of t2-t1 is greater than a specified delay interval, writes the current message into a producer message queue, and notifies the producer thread that it is readable; S6, a producer thread reads a message from the producer message queue, and sends it to a delay topic corresponding to the Kafka cluster. After the data is successfully written, the message is submitted to the real-time topic of the Kafka cluster for use by downstream consumers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of market data processing, and in particular to a market delay method and system based on a streaming system. Background Art

[0002] The market decoding system receives original real-time market information from third parties or exchanges. After internal decoding, the merged market information is serialized according to the pre-defined pb message format and published to the Kafka streaming system for use by downstream systems. pb, or protobuf, is Google's open source mechanism for automatic data serialization and deserialization. Kafka is a high-throughput distributed publish-subscribe messaging system.

[0003] Exchanges usually have certain requirements for client-side market display. Taking Hong Kong stocks as an example, they support Level 2 streaming quotes, BMP instant quotes and delayed quotes. Level 2 streaming quotes and BMP instant quotes both display real-time quotes. The difference is that Level 2 streaming quotes will display the latest quotes in real time by pushing, while BMP must manually refresh the latest quotes; delayed quotes display quotes 15 minutes ago relative to the current time. This data is not directly provided by the exchange, so it is necessary to generate delayed quotes based on real-time quotes according to the specified delay time interval.

[0004] For downstream market subscription services, one possible implementation method is for each downstream service to consume real-time data from Kafka and cache it locally, and start processing when the message generation time reaches a specified delay interval relative to the current time. This implementation has the following disadvantages: (1) The amount of data for a single market is uncontrollable. If the amount of market data is large, all services will delay their own data, which will use too much memory space and increase resource consumption. In addition, as downstream consumer businesses increase, overall resource consumption will increase. (2) Downstream services have different processing logic for real-time and delayed market conditions, and cannot reuse the same processing logic, which complicates program development. Therefore, a simpler and more effective market delay solution is needed. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a market delay method and system based on a streaming system to solve the above-mentioned technical problems.

[0006] The technical method adopted by the present invention to solve its technical problem is: a market delay method based on a streaming system, and the improvement thereof is that it comprises the following steps: S1, deploying a network time synchronization service NTP in a kafka cluster and a market delay service system, wherein the market delay service system comprises a consumer thread, a producer thread and a delay thread; S2, the producer thread writes data to the kafka cluster, and synchronizes the writing time with the time of the market delay service system; S3, starting the consumer thread, connecting to the kafka cluster, setting the real-time market topic information to be consumed, and at the same time, closing the automatic submission attribute after the message is read; S4, the decoding service system decodes the message of the kafka cluster, generates real-time market message data, and the real-time market message data is read by the consumer thread through the kafka cluster; S5, the delay thread obtains the current message pair The corresponding time t1 is obtained, and the local current time t2 is obtained at the same time, and it is determined whether the value of t2-t1 is greater than the specified delay interval. If it is judged to be yes, the current message is written into the producer message queue and the producer thread is notified that it is readable. If it is judged to be no, the delay thread sleeps for a certain period of time, and then re-obtains the local current time t2', and determines whether the value of t2'-t1 is greater than the specified delay interval. This cycle is repeated until it is judged to be greater than the specified delay interval, then the current message is written into the producer message queue and the producer thread is notified that it is readable; then the next message is obtained and the cycle is processed; S6, the producer thread reads the message from the producer message queue and sends it to the delay topic corresponding to the kafka cluster. After the data is successfully written, the message is submitted to the real-time topic of the kafka cluster to confirm that the message has been processed and provided to downstream consumers.

[0007] In the above method, in step S4, after the consumer thread reads the Kafka cluster, the market delay service system internally opens a corresponding number of message partition queues according to the number of partitions of the real-time market topic, and the consumer thread writes the real-time market message data from each partition into the corresponding message partition queue. Each message partition queue is associated with a delay thread, and the corresponding delay thread is notified that it can be read.

[0008] In the above method, after the delay thread corresponding to the notification is readable, it also includes the steps of: the delay thread reads the corresponding message partition queue, and when the message partition queue is empty, the corresponding delay thread is blocked by the conditional variable until the consumer thread writes real-time market message data to the message partition queue; when the message partition queue is not empty, jump to step S5.

[0009] In the above method, in step S6, when the producer thread reads a message from the producer message queue, if the producer message queue is empty, it is blocked by the conditional variable; when the producer message queue is not empty, the producer thread reads the message and sends it to the delay topic corresponding to the kafka cluster.

[0010] The present invention also provides a market delay system based on a streaming system, including a kafka cluster, a decoding service system and a market delay service system, wherein the network time synchronization service NTP is deployed in both the kafka cluster and the market delay system, and the market delay service system includes a consumer thread, a producer thread and a delay thread;

[0011] The producer thread is connected to the Kafka cluster and is used to write data to the Kafka cluster;

[0012] The consumer thread is connected to the Kafka cluster to set the real-time market topic information that needs to be consumed. At the same time, the automatic submission property after the message is read is turned off;

[0013] The decoding service system is connected to the Kafka cluster to decode the messages of the Kafka cluster and generate real-time market message data;

[0014] The consumer thread is connected to the Kafka cluster to consume the real-time market message data decoded and generated in the Kafka cluster;

[0015] The delay thread is used to obtain the time t1 corresponding to the current message, and at the same time obtain the local current time t2, and judge whether the value of t2-t1 is greater than the specified delay interval. If it is judged to be yes, the current message is written into the producer message queue, and the producer thread is notified that it is readable. If it is judged not to be, the delay thread sleeps for a certain period of time, and then re-obtains the local current time t2', and judges whether the value of t2'-t1 is greater than the specified delay interval. This cycle is repeated until it is judged to be greater than the specified delay interval, then the current message is written into the producer message queue; then the next message is obtained and the cycle is processed;

[0016] The producer thread is also used to read messages from the producer message queue and send them to the delayed topic corresponding to the Kafka cluster. After the data is successfully written, the message is submitted to the real-time topic of the Kafka cluster to confirm that the message has been processed and provided to downstream consumers.

[0017] The beneficial effects of the present invention are: through the message time characteristics of the streaming system, a universal delayed market solution that is independent of the message content is realized, supporting delays in any streaming message format; the delay interval is read through the configuration file to meet various time delay needs; and the universal data delay method enables downstream consumers to maintain logical consistency when processing real-time market conditions and delayed market conditions, without the need for code adjustment, and only requires configuration of the market topic corresponding to the consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Attached Figure 1 The present invention is a logic diagram of a market delay method based on a streaming system.

[0019] Attached Figure 2 The present invention is a flow chart of a method for delaying market information based on a streaming system. DETAILED DESCRIPTION

[0020] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0021] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0022] Reference Figure 1 , a market delay method based on a streaming system, comprising the following steps:

[0023] S1. Deploy the network time synchronization service NTP in the Kafka cluster and the market delay service system. Since the message delay interval is based on time, the time consistency needs to be ensured between the multiple machines running the Kafka cluster and the machines deploying the market delay service. Therefore, it is necessary to deploy the network time synchronization service NTP. The time of all machines is consistent with the time of the NTP server. NTP is a network time protocol. The market delay service system includes consumer threads, producer threads and delay threads.

[0024] S2, the producer thread writes data to the Kafka cluster, and the producer thread also uses the unified network time synchronization service NTP;

[0025] When the producer thread writes data to the Kafka cluster, each message will be associated with a timestamp, which can be set to the producer server time or the time when it is successfully written to the Kafka cluster, depending on the time format. In this embodiment, the producer thread uses the unified network time synchronization service NTP, so there is no special requirement for the time format. Otherwise, it is necessary to set the message generated by the producer to use the time of writing to the Kafka cluster to ensure the time synchronization between the market delay service system and the message writing time.

[0026] S3. Start a single consumer thread, connect to the Kafka cluster, set the real-time market topic information to be consumed, and turn off the automatic submission attribute after the message is read.

[0027] S4. The decoding service system decodes the message of the Kafka cluster to generate real-time market message data, which is read by the consumer thread through the Kafka cluster.

[0028] Since a single Kafka topic usually has multiple partitions, and the time progress between partitions may not be completely consistent (that is, a single partition is written sequentially, and the message time is strictly increasing, but the time between multi-partition messages is not guaranteed to be strictly increasing), the market delay service system opens up a corresponding number of message partition queues internally according to the number of partitions of the real-time market topic. The consumer thread writes the real-time market message data from each partition into the corresponding message partition queue. Each message partition queue is associated with a delay thread, and the corresponding delay thread is notified that it can be read. For example, the consumer thread writes the real-time market message data from the nth partition into the nth message partition queue, and notifies the corresponding nth delay thread to read.

[0029] The delay thread reads the corresponding message partition queue. When the message partition queue is empty, the corresponding delay thread is blocked by the conditional variable until the consumer thread writes real-time market message data to the message partition queue; when the message partition queue is not empty, it jumps to the next step S5.

[0030] S5. The delay thread obtains the time t1 corresponding to the current message, and obtains the local current time t2 at the same time, and determines whether the value of t2-t1 is greater than the specified delay interval. If it is judged to be yes, the current message is written into the producer message queue, and the producer thread is notified that it is readable. If it is judged to be no, the delay thread sleeps for a certain period of time (for example, 100ms), and then re-obtains the local current time t2', and determines whether the value of t2'-t1 is greater than the specified delay interval. This cycle is repeated until it is judged to be greater than the specified delay interval, then the current message is written into the producer message queue, and the producer thread is notified that it is readable; then the next message is obtained and the cycle is processed.

[0031] S6. The producer thread reads the message that meets the delay condition from the producer message queue. When the producer message queue is empty, it is blocked by the condition variable. When the producer message queue is not empty, the producer thread reads the message and sends it to the delay topic corresponding to the Kafka cluster (if the real-time topic is T, then the delay topic name is T_DELAY). After the data is successfully written, the message is submitted to the real-time topic of the Kafka cluster to confirm that the message has been processed and provided to downstream consumers.

[0032] Since the market delay service system may restart abnormally, the producer thread must wait for the message to be successfully written before it can confirm submission. Otherwise, if the consumption data is automatically submitted, there may be a large number of messages in the queue that are successfully consumed but not written to the delay topic. Once the market delay service system restarts, it will start consuming from the latest offset of the real-time market topic, and a large amount of data will be lost, resulting in the problem of missing market information.

[0033] Through the above method, the market delay service system can delay writing messages from the real-time market topic to the specified delay topic according to the specified delay interval under any conditions, and provide them to other downstream consumers for use, without requiring all downstream consumers to implement delay logic. In addition, since the message delay is based on the message time associated with the Kafka streaming system and has nothing to do with the message content, any data written to the Kafka streaming system can generate delayed market information in this way, which has strong versatility.

[0034] Taking advantage of the characteristics of the Kafka streaming message system, real-time market topic data is consumed by a single consumer thread, cached locally, and written to the delayed topic of the streaming message system after the specified delay time is reached; all downstream services can generate new real-time and delayed data by using the same logic to consume the market data of the real-time topic and delayed topic respectively, without local delay.

[0035] The present invention also provides a market delay system based on a streaming system, including a kafka cluster, a decoding service system and a market delay service system, wherein the network time synchronization service NTP is deployed in both the kafka cluster and the market delay system, and the market delay service system includes a consumer thread, a producer thread and a delay thread;

[0036] The producer thread is connected to the Kafka cluster and is used to write data to the Kafka cluster;

[0037] The consumer thread is connected to the Kafka cluster to set the real-time market topic information that needs to be consumed. At the same time, the automatic submission property after the message is read is turned off;

[0038] The decoding service system is connected to the Kafka cluster to decode the messages of the Kafka cluster and generate real-time market message data;

[0039] The consumer thread is connected to the Kafka cluster to consume the real-time market message data decoded and generated in the Kafka cluster;

[0040] The delay thread is used to obtain the time t1 corresponding to the current message, and at the same time obtain the local current time t2, and judge whether the value of t2-t1 is greater than the specified delay interval. If it is judged to be yes, the current message is written into the producer message queue, and the producer thread is notified that it is readable. If it is judged not to be, the delay thread sleeps for a certain period of time, and then re-obtains the local current time t2', and judges whether the value of t2'-t1 is greater than the specified delay interval. This cycle is repeated until it is judged to be greater than the specified delay interval, then the current message is written into the producer message queue, and the producer thread is notified that it is readable; then the next message is obtained and the cycle is processed;

[0041] The producer thread is also used to read messages from the producer message queue and send them to the delayed topic corresponding to the Kafka cluster. After the data is successfully written, the message is submitted to the real-time topic of the Kafka cluster to confirm that the message has been processed and provided to downstream consumers.

[0042] The present invention uses the message time characteristics of the streaming system to implement a universal delayed market solution that is independent of the message content and supports delays in any streaming message format; the delay interval is read through the configuration file to meet various time delay requirements; and the universal data delay method enables downstream consumers to maintain logical consistency when processing real-time market conditions and delayed market conditions, without the need for code adjustment, and only requires configuration of the market topic corresponding to the consumption.

[0043] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A market delay method based on a streaming system, characterized by: The steps include: S1. Deploy the network time synchronization service NTP in the Kafka cluster and the market delay service system. The market delay service system includes a consumer thread, a producer thread, and a delay thread. S2. The producer thread writes data to the Kafka cluster and synchronizes the writing time with the time of the market delay service system; S3. Start the consumption thread, connect to the Kafka cluster, set the real-time market topic information to be consumed, and turn off the automatic submission attribute after the message is read; S4. The decoding service system decodes the message of the Kafka cluster to generate real-time market message data, which is read by the consumer thread through the Kafka cluster. S5. The delay thread obtains the time t1 corresponding to the current message, and obtains the local current time t2 at the same time, and determines whether the value of t2-t1 is greater than the specified delay interval. If it is judged to be yes, the current message is written into the producer message queue, and the producer thread is notified that it is readable. If it is judged to be no, the delay thread sleeps for a certain period of time, and then re-obtains the local current time t2', and determines whether the value of t2'-t1 is greater than the specified delay interval. This cycle is repeated until it is judged to be greater than the specified delay interval, then the current message is written into the producer message queue, and the producer thread is notified that it is readable; then the next message is obtained, and the cycle is processed; S6. The producer thread reads the message from the producer message queue and sends it to the delayed topic corresponding to the Kafka cluster. After the data is successfully written, the message is submitted to the real-time topic of the Kafka cluster to confirm that the message has been processed and provided to downstream consumers.

2. A method for delaying market information based on a streaming system as claimed in claim 1, characterized in that: In step S4, after the consumer thread reads the Kafka cluster, the market delay service system internally opens a corresponding number of message partition queues according to the number of partitions of the real-time market topic. The consumer thread writes the real-time market message data from each partition into the corresponding message partition queue. Each message partition queue is associated with a delay thread, and the corresponding delay thread is notified that it is readable.

3. A method for delaying market information based on a streaming system as claimed in claim 2, characterized in that: After the delay thread corresponding to the notification is readable, the process also includes the following steps: the delay thread reads the corresponding message partition queue; when the message partition queue is empty, the corresponding delay thread is blocked by a conditional variable until the consumer thread writes real-time market message data to the message partition queue; when the message partition queue is not empty, the process jumps to step S5.

4. A method for delaying market information based on a streaming system as claimed in claim 1, characterized in that: In step S6, when the producer thread reads a message from the producer message queue, if the producer message queue is empty, it is blocked by the conditional variable. If the producer message queue is not empty, the producer thread reads the message and sends it to the delay topic corresponding to the kafka cluster.

5. A market delay system based on a streaming system, characterized by: It includes Kafka cluster, decoding service system and market delay service system. The network time synchronization service NTP is deployed in both Kafka cluster and market delay system. The market delay service system includes consumer thread, producer thread and delay thread. The producer thread is connected to the Kafka cluster and is used to write data to the Kafka cluster; The consumer thread is connected to the Kafka cluster to set the real-time market topic information that needs to be consumed. At the same time, the automatic submission property after the message is read is turned off; The decoding service system is connected to the Kafka cluster to decode the messages of the Kafka cluster and generate real-time market message data; The consumer thread is connected to the Kafka cluster to consume the real-time market message data decoded and generated in the Kafka cluster; The delay thread is used to obtain the time t1 corresponding to the current message, and at the same time obtain the local current time t2, and judge whether the value of t2-t1 is greater than the specified delay interval. If it is judged to be yes, the current message is written into the producer message queue, and the producer thread is notified that it is readable. If it is judged not to be, the delay thread sleeps for a certain period of time, and then re-obtains the local current time t2', and judges whether the value of t2'-t1 is greater than the specified delay interval. This cycle is repeated until it is judged to be greater than the specified delay interval, then the current message is written into the producer message queue; then the next message is obtained and the cycle is processed; The producer thread is also used to read messages from the producer message queue and send them to the delayed topic corresponding to the Kafka cluster. After the data is successfully written, the message is submitted to the real-time topic of the Kafka cluster to confirm that the message has been processed and provided to downstream consumers.

Citation Information

Patent Citations

  • Message caching system and message caching method based on Kafka

    CN109451072A

  • Lightweight KAFKA management and timed task queue system based on DB

    CN113961329A