A method to realize production and consumption of real-time data flow across data sources

A technology that crosses data sources and data streams, applied in the information field, can solve problems such as resource waste and consumer consumption, and achieve the effect of reducing the number of concurrent connections and the risk of data leakage

CN111913821BActive Publication Date: 2021-07-16武汉众邦银行股份有限公司
7 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2021-07-16

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
Patent Text Reader

Abstract

The invention relates to the field of information technology, and provides a method for realizing production and consumption of real-time data flow across data sources. The purpose is to solve the existing problem that usually a consumer does not need all the data in the same message queue, but consumes part of the data in multiple message queues, because the data in the message queue can only be consumed sequentially, which will cause the consumer to consume Unnecessary data is generated, on the one hand, it causes a waste of resources, and on the other hand, it brings about data security issues. The main plan includes Step 1: Unify the data standards of the data producer and add data category identifiers; Step 2: Develop a message queue filter board, filter and classify the data in the message queue, and realize the data identification in the message column, Obtain the data source address, database name, and table name corresponding to each piece of data; Step 3: Develop the data distribution mechanism function, collect the data consumption needs of consumers, and perform data configuration distribution according to the needs.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the field of information technology, and provides a method for realizing production and consumption of real-time data flow across data sources. Background technique

[0002] In order to better understand this proposal, it is necessary to understand the basic technologies:

[0003] Message queuing is a communication method between processes. It has a remarkable feature: asynchronous processing, which allows users to put a message into the queue, but not process it immediately, and consume it in the queue when needed, and then to deal with them. Therefore, in order to ensure data consistency, the message queue adopts the FIFO first-in-first-out feature to ensure the order of messages.

[0004] In daily use, usually a consumer does not need all the data in the same message queue, but consumes part of the data in multiple message queues, because the data in the message queue can only be consumed sequentially, which will cause the...

Examples

Embodiment Construction

[0031] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, and are not intended to limit the present invention, that is, the described embodiments are only some of the embodiments of the present invention, but not all of the embodiments. The components of the embodiments of the invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations.

[0032] Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the embodiments of the...