Unlock instant, AI-driven research and patent intelligence for your innovation.

A data processing method and device

A processing method and data technology, applied in the database field, can solve the problem of inability to add, modify or delete data, achieve the effect of good incremental extraction performance and save processing time

Active Publication Date: 2020-07-24
GUANGZHOU SHIYUAN ELECTRONICS CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The shortcomings of timestamp-based incremental extraction are as follows: some data tables in the database are directly designed as non-timestamp fields, so it is impossible to know when the data was added, modified or deleted

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data processing method and device
  • A data processing method and device
  • A data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0020] figure 1 It is a schematic flow chart of a data processing method provided by Embodiment 1 of the present invention. This processing method is applicable to the situation of incrementally extracting updated data across data warehouses. The method can be executed by a data processing device, wherein the device can be implemented by software and / or hardware implementation.

[0021] It should be noted that the processing method of the embodiment of the present invention can implement incremental extraction across data warehouses for source data tables that do not have update identification fields, that is, extract update data from source data tables that do not have update data identification fields in source data warehouses. The data is stored in the target data table of the target data warehouse. It can be understood that there is a one-to-one correspondence between the source data tables in the source data warehouse and the target data tables in the target data wareho...

Embodiment 2

[0034] figure 2 It is a schematic flowchart of a data processing method provided by Embodiment 2 of the present invention. Embodiment 2 of the present invention is optimized based on the above-mentioned embodiment. In this embodiment, the attribute information in the current data table will be searched for The associated increment data table is specifically optimized as follows: according to the attribute information in the current data table, determine the associated key in the current data table; determine the increment associated with the current data table according to the associated key data sheet.

[0035] Further, according to the current incremental data in the incremental data table, the target incremental data of the current data table is determined, specifically optimized as: obtaining the current incremental data of the associated key in the incremental data table The corresponding incremental data value in the data is stored; and the target incremental data in t...

Embodiment 3

[0061] image 3 It is a schematic flowchart of a preferred embodiment of a data processing method provided by Embodiment 3 of the present invention. The application background of the embodiment of the present invention is: as the target data warehouse, the data warehouse under the big data platform cannot contact the production system, and can only provide historical data from the source data warehouse connected to the production system. In addition, the source data warehouse includes Basic customer information table, website customer table, offline customer table, VIP customer table, etc., among them, only the basic customer information table has the update identification field, and the rest of the data tables do not have the update identification field, and the website customer table , offline customer table, and VIP customer table are all associated with the basic customer information table based on their associated keys.

[0062] Based on the above statement, based on the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a data processing method and device. The method comprises the following steps: acquiring attribute information of the current data table; searching an associated incremental data table according to the attribute information in the current data table, wherein the incremental data table contains the current incremental data acquired in the current processing period; and determining the target incremental data of the current data table according to the current incremental data in the incremental data table. The incremental extraction across the databases can be simply and efficiently performed on the update data in the database table without an update identification field by use of the processing method; compared with the existing data incremental extraction mode, the processing method disclosed by the invention has better incremental extraction property, the processing time of the data processing can be saved, thereby realizing the diversity of the data incremental extraction.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of databases, and in particular, to a data processing method and device. Background technique [0002] Data extraction mainly refers to the process of extracting data in the database. Common data extraction methods include full extraction and incremental extraction. Full extraction refers to extracting the data in the data source from the database intact; and Incremental extraction refers to the extraction from the database of data that has been added, modified, and / or deleted since the last extraction. In the extract-transform-load (Extract-Transform-Load, ETL) process of the database, the incremental extraction method is more widely used than the full extraction. [0003] Currently, common incremental extraction methods can be based on triggers, timestamps, full table comparisons, and / or log comparisons. The above-mentioned methods have their own advantages and disadvantages in ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25
CPCG06F16/254
Inventor 徐骄
Owner GUANGZHOU SHIYUAN ELECTRONICS CO LTD