Data migration deployment method based on access heat

A technology of visit popularity and number of visits, applied in the field of data processing, which can solve the problems of dividing data areas by a single data field and not considering the actual access behavior of users to data sets, etc.

Active Publication Date: 2019-07-12
SOUTH CHINA UNIV OF TECH
View PDF11 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In the field of big data migration, the traditional data migration algorithm does not take into account the actual access behavior of the user to the data set, but only divides the data area according to the data field, and then divides the data, migrates and deploys the data to each node of the distributed platform

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data migration deployment method based on access heat
  • Data migration deployment method based on access heat
  • Data migration deployment method based on access heat

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0077] Such as figure 1 and figure 2 As shown, a data migration and deployment method based on access heat mainly includes a distributed platform-oriented big data migration deployment control system, statistical analysis of access heat based on data set access logs, and splitting the data according to the column with the highest number of accesses. Update the working steps of data deployment, centralize the log-based access heat load balancing data segmentation algorithm and an access details table for storing access information within a cycle.

[0078] Concrete steps of the present invention are as follows:

[0079] The S1 user specifies the dataset DataSet that needs to be migrated and deployed in the data migration deployment control system.

[0080] The S2 data migration deployment control system obtains the access log data set DataSetAccessLog of the data set DataSet in the distributed platform.

[0081] The S3 user specifies the segment number SegmentNum of the Data...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data migration deployment method based on access heat, which comprises the following steps: for a column type data set migrated and deployed on a distributed platform, predicting access quantity distribution in a next time period by using a prediction algorithm according to user access log information during operation of the column type data set; calculating an access frequency sequence of each field according to the predicted access quantity distribution; re-dividing the distribution of the data values of the column of fields with the highest predicted access frequency into data sub-regions, so that the access frequency of the fields is uniformly distributed into new data sub-regions; and the system splitting the data according to the re-divided data area of thecolumn with the highest access frequency and updating the data deployment on the distributed platform. According to the method, the data migration deployment oriented to the distributed platform is realized by combining the actual access behavior of the user to the data set, so that the access heat of the highest access heat column of the original data set is balanced to each data node by load, and the optimal comprehensive access performance of the data set on the distributed platform is realized.

Description

technical field [0001] The present invention relates to the field of data processing, in particular to a data migration and deployment method based on access heat. Background technique [0002] With the popularization and application of computer and information technology, the scale of data information is increasing rapidly, and most enterprises still store all kinds of data generated by their various businesses in relational databases. With the rapid growth of data scale, for traditional relational databases, the storage bottleneck problem caused by massive data and the low performance of data analysis and processing are particularly prominent, which has become an urgent problem for enterprises to solve. In the field of cloud computing and big data, the distributed platform architecture in its field is of outstanding significance and generates practical application value. Migrating massive data to a distributed platform and using the resource sharing and collaborative comp...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/21G06F16/28G06F9/50
CPCG06F16/214G06F16/284G06F9/5083
Inventor 杨灿刘宇
Owner SOUTH CHINA UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap