Data real-time processing method and system

A technology for real-time data processing and data processing, which is applied in the field of information processing and can solve problems such as large data processing delays

Inactive Publication Date: 2016-10-05
CHINA MOBILE GRP FUJIAN CO LTD
View PDF4 Cites 12 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] In view of this, the embodiment of the present invention expects to provide a real-time data processing method and system to at least partially solve the problem of large data processing delay in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data real-time processing method and system
  • Data real-time processing method and system
  • Data real-time processing method and system

Examples

Experimental program
Comparison scheme
Effect test

example 1

[0138] Such as image 3 As shown, this example provides a specific method for obtaining the data change information, as follows:

[0139] Step 101: The database log of external business acceptance changes. This means that the database log of the above-mentioned source system has changed.

[0140] Step 102: capture by the data replication module. The data replication module here is the above-mentioned replication sub-module. The capture here is to obtain data change information, specifically, use the data replication product GoldenGate to capture the Redo Log of the database, obtain the changed business acceptance data, and capture the change data of the source database, track queues, data pump distribution, route compression encryption and In the process of delivering the target, the log information of the change of the business acceptance database is captured in real time with minimal system expenditure.

[0141] Step 103: Pump out the trail file, this step is automatical...

example 2

[0146] Such as Figure 4 As shown, this example provides a specific method for processing the first data, which may include:

[0147] Step 1: The log parsing module in the real-time data acquisition module incrementally generates data files in csv format by parsing and processing the change logs of the source system database. The log parsing module here is detailed in the parsing sub-module above.

[0148] Step 2: The data cleaning module queries data cleaning, summarizing, and loading rule bases to obtain data cleaning rules.

[0149] Step 3: Data cleaning, summarizing, loading rule base, and returning the cleaning rules to be queried in the data cleaning module. The cleaning rule here is a part of the first preset processing strategy.

[0150] Step 4: According to the data cleaning rules, check the consistency of the csv business acceptance data incrementally generated by the log parsing module, process invalid and missing values, and clean the data. Stream processing tec...

example 3

[0158] Such as Figure 5 As shown, this example provides a system management method that may include:

[0159] Step s1: Receive a data service request from a data service requester. Specifically, if the real-time monitoring interface is used to monitor and initiate a data service request, multiple data service requesters can send multiple data request services concurrently.

[0160] Step s2: Use a load balancer cluster to receive high-concurrency data service requests, perform service balancing on high-concurrency data request services, and hosts in the cluster all back up each other, realize high-concurrency data service request processing, and speed up service request response speed . The mutual backup mode of the hosts in the cluster can be two-by-two, or one backup for multiple.

[0161]Step s3: Distributing the service request after the route adaptation calculates the mod (the said determination of the mod is the calculation of the modulus). Specifically, after balanc...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data real-time processing method and system. The method is applied to the data real-time processing system. The method comprises the steps of obtaining source system data change information; performing first data processing on the data change information in real time according to a first preset processing policy to form processed first data; loading the first data in a memory database; and performing second data processing on the first data in the memory database in real time according to a second preset processing policy based on a data processing demand to obtain a data processing result.

Description

technical field [0001] The invention relates to information processing technology in the field of information processing, in particular to a method and system for real-time data processing. Background technique [0002] With the development of social informatization, the value of data (specifically, business operation data) is increasing day by day, and the timeliness requirements for operational information analysis are also increasing. It is required to achieve real-time analysis and real-time decision-making through real-time monitoring of business acceptance data. . [0003] Existing data analysis generally includes two types: [0004] The first method: performing statistical analysis directly on the original business system may occupy a large amount of resources of the business system, which in turn will cause a great impact on the normal operation of the business system, and it is difficult to guarantee the timeliness of analysis. [0005] The second type: build a da...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 阮前黄启雄陈欻李建明
Owner CHINA MOBILE GRP FUJIAN CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap