Unlock instant, AI-driven research and patent intelligence for your innovation.

A Correlation Completion Method for Multi-channel Real-time Streaming Data

A real-time stream and data technology, applied in the direction of electronic digital data processing, structured data retrieval, database management system, etc., can solve problems such as imperfect support, affecting the timely and accurate identification of risks by the system, and affecting data quality, so as to reduce user Loss of assets, improvement of data governance efficiency, and accurate identification of risky transactions

Active Publication Date: 2020-02-07
ZHEJIANG BANGSUN TECH CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] In traditional entity relationship analysis based on batch processing scenarios such as databases, data marts, and data warehouses, it is often necessary to perform association completion for multiple entities, and to meet flexible and diverse business analysis requirements based on the obtained complete information. The method performs very poorly in the case of large data volume and is inefficient
For example, in anti-fraud scenarios, the need for data integrity and timeliness is very urgent, which affects whether the system identifies risks in a timely and accurate manner. The existing open source stream processing framework does not fully support real-time correlation completion, and there is a time window. Misalignment, causing data loss and affecting data quality

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Correlation Completion Method for Multi-channel Real-time Streaming Data
  • A Correlation Completion Method for Multi-channel Real-time Streaming Data
  • A Correlation Completion Method for Multi-channel Real-time Streaming Data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0021] Such as figure 1 As shown, this embodiment provides an association completion method for multi-channel real-time streaming data. In the real-time collection and streaming processing of multi-channel data, there are often associated relationships between different data streams, and it is necessary to Real-time completion of multi-channel streaming data to obtain complete data in real time; the method specifically includes the following steps:

[0022] (1) Using the database, file, and message bus of the computer system as the data source to form multiple data streams, the data source can be each table of the database, each file, and the message bus (such as kafka, jms).

[0023] To make data streams can be associated and completed, it is necessary to determine that there is a correlation between the data streams, that is, there is a value of a field or a value of multiple fields concatenated in a certain data stream and a value of a field or multiple fields in other data...

Embodiment 2

[0039] Step 1: Collect data from the computer system to form multiple data streams A and B. Stream A is a secondary data stream, and its data object contains two fields of uid and status; stream B is the main data stream, and its data object contains orderId, userId, amt, createTime, and chnl fields;

[0040] Step 2: Create a Tout bypass output task, and temporarily store the auxiliary data stream A to the KV storage, where uid is used as the cache primary key (Key), and status is used as the cache data field (Value);

[0041] Step 3: Create a Tin bypass input task, use the userId field of the data object in the main data stream B as the cache primary key (Key) to obtain the temporarily stored auxiliary data from the KV storage, and fill the value of the status field in the auxiliary data into the main In the data object of data stream B;

[0042] Step 4: The complete main data stream B' obtained after association completion includes the complete fields orderId, userId, amt, ...

Embodiment 3

[0044] Step 0: Database A and database B exist in a certain computer system, such as Figure 4 As shown, there is a transaction status table in database A, and a transaction information table in database B. Now it is necessary to associate these two database tables, and add the value of the status field in the transaction status table to the transaction information table, and finally form a The complete structure of the merged table for regulatory system analysis;

[0045] Step 1: Use the data collection tool to collect data in real time from the transaction status table of database A to form data flow A;

[0046] Step 2: After the transaction status table data is collected, create a Tout bypass output task to temporarily store the collected data to external storage for processing, and use the serial number field in the transaction status table as the primary key so that the data in the cache can be queried ;

[0047] Step 3: Perform the same real-time collection operation on ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses an association completion method for multi-channel real-time streaming data, and the method comprises the steps: temporarily storing to-be-associated completed auxiliary data to an external KV for storage in a data real-time flowing process, enabling an association primary key to be Key, and enabling the to-be-completed data to be Value; And in the main data flow process, obtaining auxiliary data from the KV storage through the Key which is matched in an associated mode for completion. The method is suitable for solving the problem that in the prior art, high-quality and high-efficiency real-time data association cannot be guaranteed at the same time, the method is an efficient data association completion method, the data acquisition efficiency and quality can be improved, and the service requirement is better met.

Description

technical field [0001] The invention is applicable to the fields of financial anti-fraud, real-time marketing, real-time credit granting, real-time network robot prevention and control, and real-time operation and maintenance scenarios, and in particular relates to a correlation complement method for multi-channel real-time streaming data. Background technique [0002] In traditional entity relationship analysis based on batch processing scenarios such as databases, data marts, and data warehouses, it is often necessary to perform association completion for multiple entities, and to meet flexible and diverse business analysis requirements based on the obtained complete information. The performance of the method is very poor in the case of large amount of data, and the efficiency is low. For example, in anti-fraud scenarios, the need for data integrity and timeliness is very urgent, which affects whether the system identifies risks in a timely and accurate manner. The existin...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25G06F16/215
Inventor 李白王新根杨志强王新宇黄滔鲁萍
Owner ZHEJIANG BANGSUN TECH CO LTD