Check patentability & draft patents in minutes with Patsnap Eureka AI!

Data processing system, method and device, electronic equipment and storage medium

A data processing system and data technology, applied in the computer field, can solve the problems of low collection frequency, difficulty in batch data processing and reporting methods to meet regulatory requirements, and low data timeliness

Pending Publication Date: 2021-06-11
CHINA CONSTRUCTION BANK
View PDF0 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0009] The above-mentioned existing data collection and processing methods need to unload data from the production database of the transaction line, which will have an impact on the current transactions of the transaction line. The frequency is low, and the timeliness of obtaining data is low
As regulatory agencies increasingly require timeliness of submission, batch data processing and submission methods have been difficult to meet regulatory requirements

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing system, method and device, electronic equipment and storage medium
  • Data processing system, method and device, electronic equipment and storage medium
  • Data processing system, method and device, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0035] figure 1 It is a schematic structural diagram of a data processing system provided by Embodiment 1 of the present invention. The system has high timeliness and is applicable to the application scenarios of real-time collection, real-time processing and real-time reporting of data generated in real time. For example, the bank's real-time transaction data needs to be reported to the bank supervisor for supervision within a few seconds after the transaction data is generated.

[0036] like figure 1 As shown, the data processing system provided in this embodiment includes: a first data collection component OGG 110 , a data bus component Kafka 120 , a streaming computing engine Apache Flink 130 , a first database 140 and a data reporting terminal 150 .

[0037] Wherein, the first data collection component OGG 110 is communicatively connected with the data bus component Kafka 120 for collecting the first data in real time and sending the collected first data to the data bus ...

Embodiment 2

[0054] figure 2 It is a schematic structural diagram of a data processing system provided by Embodiment 2 of the present invention. On the basis of the above embodiments, this embodiment further expands the functions of the data processing system, specifically adding a change data capture CDC component, the change data capture CDC component communicates with the data bus component Kafka for The second data is collected in real time, and the collected second data is sent to the data bus component Kafka, so that the data processing system can collect, process and report more business data. Wherein, the same or similar content as in the foregoing embodiments will not be described in detail in this embodiment, and relevant parts in the foregoing embodiments may be referred to.

[0055] like figure 2 As shown, the data processing system includes:

[0056] The first data acquisition component OGG 210 , the data bus component Kafka 220 , the streaming computing engine ApacheFlin...

Embodiment 3

[0069] image 3 It is a schematic structural diagram of a data processing system provided by Embodiment 3 of the present invention. On the basis of the above embodiments, this embodiment further expands the functions of the data processing system, specifically adding a network file transfer NFT component, a batch computing area, a data synchronization component and a second database. Wherein, the network file transfer NFT component is communicatively connected to the batch computing area, and is used to collect the third data in batches according to a set period, and send the collected third data to the batch computing area; The batch computing area is respectively connected to the data bus component Kafka and the second database through the data synchronization component, and is used to perform calculations based on the third data, the first data, and the second data. calculation, and the calculation results are respectively synchronized to the second database and the first ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a data processing system, method and device, electronic equipment and a storage medium. The system comprises a first data acquisition component OGG, a data bus component Kafka, a streaming computation engine Apache Flink, a first database and a data submission end. Transaction data are collected into Kafka in real time through an OGG or CDC technology, dimension information is collected into Hbase through a day end batch mode, data integration is carried out through a FLink operation mode, and final data message submission is completed through an MQ message mode. According to the technical scheme provided by the embodiment of the invention, the purpose of completing data acquisition, processing and submission within a few seconds after the data is generated is realized, and the real-time transaction is not influenced.

Description

technical field [0001] The embodiments of the present invention relate to the field of computer technology, and in particular, to a data processing system, method, device, electronic equipment, and storage medium. Background technique [0002] Since the monitoring frequency of the regulatory agency is based on the original daily, monthly, quarterly, and annual reports, the method of real-time submission has been added. Therefore, in order to meet the timeliness of reporting, financial institutions must adopt new data collection and processing methods. [0003] The traditional data collection and processing method is to realize the loading and processing of batch data files through a batch job scheduling platform like Control-M. Specifically, copy data from the database, then load the data and trigger the downstream data processing script job, Finally, trigger unloading to data transmission, the detailed steps are as follows: [0004] Step 1: Before data processing, first p...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F16/25G06F16/2455
CPCG06F16/24568G06F16/254G06F16/27
Inventor 刘强张东凯曾垂鑫
Owner CHINA CONSTRUCTION BANK
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com