Redis-based mass data classified storage method and system

A mass data and storage system technology, applied in the computer field, can solve problems such as increasing response timeout rate, increasing query time, and difficulty in meeting the continuous growth of user data volume, and achieve the effect of reducing memory fragmentation and memory occupation

Active Publication Date: 2021-08-24
上海艾麒信息科技股份有限公司
View PDF12 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Store user data directly in the database: This method has almost no limit to the growth of user data, and the cost of the hard disk is cheaper than memory, but the efficiency and response time of query data are far inferior to the memory database redis. For real-time advertising trading systems In terms of advertising, it will increase the response timeout rate of advertising bidding, and with the increase of user data, the query time will increase, wh

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Redis-based mass data classified storage method and system
  • Redis-based mass data classified storage method and system

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0052] Example 1

[0053]According to the present invention, a mass-based mass data classification storage method is provided, including:

[0054] Step S1: Classify the data and define a data category ID for each category data;

[0055] Step S2: For each of the data categories, based on the data size of the corresponding actual service, the number of HASH buckets is calculated; according to the characteristics of the REDIS, the Hash type data is the most space Ziplist data structure storage mode requires Hash Bucket ( That is, the data of the Field stored in a Hash Key is less than 512, so the number N of the HASH bucket is calculated as: n = total data amount / 512, and hindered up.

[0056] Step S3: Use the amount of data identifier ID, data category ID, and data bucket as an incoming factor, performing Hash KEY and FIELD;

[0057] Step S4: The data identifies the data identifies the data as a Hash Value;

[0058] Step S5: Store the Hash Key, Field, and Hash Value into the Redi...

Example Embodiment

[0089] Example 2

[0090] Example 2 is a preferred example of Example 1

[0091] The present invention stores the mass-quantity of data to the REDIS to save memory, and improve access speed, and can classify independently.

[0092] The present invention provides a method of redis based massive data classification storage, including:

[0093] Step 1: Define the user data category ID according to the business category;

[0094] Step 2: Define the amount of user data bucket according to the data level corresponding to the data category ID;

[0095] Step 3: The user data identifier of the string type, the data category ID, the data category ID, the data category ID, using the HASH algorithm A calculate the integer type hashing result A;

[0096] Step 4: Take the intensive type of hash result A as a Hash Key of the REDIS's HASH data type;

[0097] Step 5: A user data identifying the string type as a unique factor, calculates the integer type has been obtained by the HASH algorithm B, ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a Redis-based mass data classified storage method and system, and the method comprises the steps: S1, classifying data, and defining a data category ID for each category of data; S2, for each data category, calculating the number N of hash buckets according to the data scale of the corresponding actual service; S3, taking the data identifier ID, the data category ID and the number N of the data buckets as input parameter factors, and performing hash calculation to obtain a hash key and a field; S4, using the data content corresponding to the data identification ID as a hashvalue; and S5, storing the hashkey, the field and the hashvalue into the redis. According to the method and the device, the user data identifier is converted into a digital form and is stored in redis in the form of hash type data, so that the purpose of reducing memory occupation is achieved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a Redis-based massive data classification storage method and system. Background technique [0002] At present, when processing huge user data, there are generally the following common processing methods: [0003] Store user data directly in the database: This method has almost no limit to the growth of user data, and the cost of the hard disk is cheaper than memory, but the efficiency and response time of query data are far inferior to the memory database redis. For real-time advertising trading systems In terms of advertising, it will increase the response timeout rate of advertising bidding, and with the increase of user data, the query time will increase, which will further increase the response timeout rate; [0004] Store user data directly in redis in the form of key-value key-value pairs. This method will take up 8 times more memory space than the method used in...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/22G06F11/10
CPCG06F16/2255G06F16/2219G06F11/1004
Inventor 周单健盛猛林立
Owner 上海艾麒信息科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap