Data processing method, device and system

A data processing and data technology, applied in the field of data processing, can solve the problems of storage space waste, data redundancy, etc., and achieve the effect of avoiding waste, reducing the moving distance of the magnetic head, and improving the performance of query statistics

Inactive Publication Date: 2013-09-11
上海淼云文化传播有限公司
View PDF0 Cites 26 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to provide a data processing method, device and system to solve the technical p

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, device and system
  • Data processing method, device and system
  • Data processing method, device and system

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0083] reference image 3 , Which shows a flowchart of Embodiment 2 of a data processing method provided by the present invention. Method Embodiment 2 of the present invention is based on Method Embodiment 1, and further includes the following steps:

[0084] Step 301: Receive a data query request, where the data query request includes query conditions.

[0085] Among them, after receiving the data query request input by the user, the query conditions included in the data query request are parsed.

[0086] Step 302: Perform hash algorithm calculation on the query condition to obtain the query key value.

[0087] Among them, it should be noted that when the hash algorithm calculation is performed on the query conditions, the hash algorithm described in the first embodiment of the method of the present invention needs to be calculated.

[0088] Step 303: In the key value of the data set, search whether there is a key value that matches the query key value, if yes, go to step 304, otherwis...

Example Embodiment

[0146] reference Picture 9 , Which shows a schematic structural diagram of Embodiment 4 of a data processing apparatus provided by the present invention. Based on Embodiment 2 of the apparatus of the present invention, the apparatus may further include:

[0147] The data cache unit 609 is configured to store the data returned by the data search unit 602 into a preset cache data set.

[0148] The data search unit 602 stores the returned data as cache data in the cache data set after finishing a successful data query. When the receiving unit 605 receives the data query request again, the data caching unit 609 first determines whether the cached data set contains data corresponding to the data query request, and if so, the data is described If the data requested by the query has been queried, the data search unit 602 can directly query the data in the cached data set. If not, it means that the data requested by the data query has not been queried before. The data searching unit 602 ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data processing method, device and system. The data processing method comprises the steps that to-be-stored data is subjected to hash algorithm calculation to obtain a key value of the to-be-stored data; and whether a key value matched with the key value is contained is searched in a preset data set, if yes, the to-be-stored data is rejected, otherwise, the to-be-stored data is stored in a column type storage method, and the key value of the to-be-stored data is stored in the data set. According to the data processing method, device and system provided by the embodiment of the invention, the hash calculation is performed before storage of the to-be-stored data in the column type storage method, duplication eliminating processing is performed on the to-be-stored data based on the obtained key value, so that data redundancy during mass data processing is avoided, and storage space waste is avoided.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a data processing method, device and system. Background technique [0002] Massive data refers to fields such as enterprise-level software, the Internet, and cloud computing, which need to process large-capacity data. At present, the mass data processing scheme is generally adopted as a slice separation scheme. [0003] The above slicing separation scheme means that when storing data, the data to be stored is stored in slices according to the slicing rules, that is, the data to be stored is stored in different data slices according to the data attributes, and the horizontal expansion of data storage is realized. to efficiently process massive amounts of data. [0004] Although the slice separation scheme can store data into different data slices, the data in the data slices is not deduplicated during the storage process, resulting in data redundancy. Moreover, as the amount of d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 李晨马向晖
Owner 上海淼云文化传播有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap