Data processing method and device

A data processing and data sub-technology, applied in the field of big data, can solve problems such as the inability to guarantee data authenticity, and achieve the effect of ensuring data consistency, improving user experience, and ensuring authenticity

Active Publication Date: 2018-03-02
新智云数据服务有限公司
View PDF5 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The embodiment of the present invention provides a data processing method and device to solve the problem that the existing big data platform cannot guarantee the authenticity of data when processing data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device
  • Data processing method and device
  • Data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] In order to solve the problem that the existing big data platform cannot guarantee the authenticity of data during data processing, in the embodiment of the present invention, a data processing method is redesigned. The method is to extract and load from the specified source system through a specified tool The initial data set is sent to the designated big data platform, and the loaded initial data set is divided into several initial data subsets, and it is judged in batches whether the currently selected initial data subset needs to modify the initial data bar, and for the initial data subset to be modified Perform the incremental merge operation to obtain the corresponding target data subsets, and directly determine the initial data subsets that do not need to be modified as the corresponding target data subsets, and then use the preset business logic rules to Establish an association relationship with several target data items contained in the set.

[0052] The followi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the technical field of big data, in particular relates to a data processing method and device, and the problem that data authenticity cannot be guaranteed during data processing by a big data platform in the prior art is solved. The data processing method is that an initial data set is extracted from a designated source system by a designated tool and loaded, and then theloaded initial data set is divided into a plurality of initial data subsets; whether initial data bars are required for modification is determined for the current initial data subsets by batches; increment combination operation is executed to to-be-modified initial data subsets, so a corresponding target data subset is acquired; an incidence relation is built for a plurality of target data bars contained in each acquired target data subset according to a preset business logic rule; therefore, a designated big data platform can be linked with the designated source system via the designated tool; besides, even the data are loaded to the designated big data platform and then modified, data consistency can still be guaranteed via the increment combination operation; therefore, data authenticity can be guaranteed and user experience can be improved.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data processing method and device. Background technique [0002] The rapid development of big data and the Internet has brought about explosive growth of massive data, as well as various data source systems that provide data. With the increase in data volume, data warehouses based on traditional data structures are becoming more and more overwhelmed. Big data platforms The emergence of the big data platform has solved the above problems very well. At present, the big data platforms with a wide range of applications include Hadoop platform, Storm platform, Spark platform and so on. [0003] However, not all data source systems can be connected to different big data platforms. For example, under the existing technology, the connection between the SAP source system and the Hadoop platform cannot be realized, that is, the data of the SAP source system cannot be extracted t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/2358G06F16/2365G06F16/254
Inventor 李红伟
Owner 新智云数据服务有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap