Cold and hot data analysis method and system
A technology of hot and cold data and analysis methods, applied in the field of big data, can solve problems such as omissions and deletions by mistake, waste of manpower and material resources, and cumbersome work, so as to avoid misoperation, improve work efficiency, and save system resources.
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
specific Embodiment 1
[0061] figure 1 It is a schematic diagram of the cold and hot data analysis principle of the embodiment of the present invention; figure 2 It is a preferred schematic flow diagram of the cold and hot data analysis method of the embodiment of the present invention; combined with reference figure 1 , 2 As shown, this embodiment discloses a cold and hot data analysis method, including:
[0062] The file data information acquisition step S1 is used to obtain the data information of the file in the HDFS by analyzing the metadata node NameNode and the audit log AuditLog of a Hadoop distributed file system HDFS, and the data information includes but is not limited to data storage information and data usage information;
[0063] The hot and cold data analysis step S2 is used to analyze the data information of the files in HDFS by using the neural network algorithm model LSTM, and divide the files of HDFS into cold data and hot data by using the data information as a reference.
[...
specific Embodiment 2
[0087] Only the differences between this embodiment and the specific embodiment 1 are described below, and the similarities are not repeated here. Figure 4-5 They are another preferred principle schematic diagram and flow diagram of the hot and cold data analysis method of the embodiment of the present invention, combined with reference Figure 4-5 As shown, the difference between the cold and hot data analysis method of this embodiment and the specific embodiment 1 is:
[0088] Considering that only the neural network algorithm model LSTM is used to divide the hot and cold data, there may be a problem that some files have not been accessed for a long time from the current time point but are mistakenly marked as 1. Therefore, the hot and cold data analysis step S2 also includes:
[0089] The LRU hot and cold data division step S204 is used to analyze the data storage information and data usage information to obtain the last modification time and / or access time of the file in ...
PUM
Abstract
Description
Claims
Application Information
- R&D Engineer
- R&D Manager
- IP Professional
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com