Data heterogeneous storage method and device, data reading method and device and electronic equipment
By obtaining access logs and dynamically migrating data to the appropriate storage medium according to the access frequency, the problems of high storage costs and low access efficiency in traditional data storage methods are solved, and flexible allocation of resources and efficient data access are achieved.
Patent Information
- Application Number
- CN202411168385.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-29
AI Technical Summary
When facing data volumes at PB or EB level, traditional data storage methods have high storage costs and low access efficiency, especially improper processing of hot and cold data, resulting in insufficient resource utilization and difficulty in dealing with the dual factors of data access speed and cost. challenge.
By obtaining the access log, determine the path with access frequency higher than the preset threshold, store it in the corresponding high-frequency or low-frequency storage medium, and migrate it through the metadata management center to ensure that the data is stored on the matching medium, and dynamically provision it Storage resources.
It reduces storage costs, improves data access efficiency, and can intelligently allocate storage resources based on actual access frequency, reduces data processing volume, and improves query efficiency.
Smart Images

Figure CN119045740A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data, and in particular to a data heterogeneous storage method, a data reading method, a device and an electronic device. Background Art
[0002] With the rapid development of information technology, big data has become a key force in promoting social progress and digital transformation of enterprises. With the widespread application of technologies such as the Internet of Things, cloud computing, and artificial intelligence, the speed and scale of data generation have exploded. How to efficiently manage and utilize these massive amounts of data has become a major challenge facing enterprises and scientific research institutions. When faced with PB (Petabyte) or even EB (Exabyte)-level data volumes, traditional data storage methods often expose problems such as high storage costs, low access efficiency, and insufficient resource utilization. In particular, improper handling of hot data that needs to be accessed frequently and cold data that is rarely accessed has exacerbated this contradiction. Therefore, exploring an intelligent storage solution that can both ensure data access speed and effectively control storage costs has become a hot topic in the industry. Summary of the invention
[0003] In view of this, an embodiment of the present invention provides a data heterogeneous storage method, a data reading method, a device and an electronic device to improve data access efficiency.
[0004] According to one aspect of the present invention, a method for heterogeneous data storage is provided, the method comprising:
[0005] Obtaining an access log, wherein the access log records the access path of each of the big data engines and the access time corresponding to the access path;
[0006] Based on the access log, determine a first access path whose access frequency is higher than a preset frequency threshold, store the first access path in a preset storage medium, and store description information of the first access path in the preset storage medium in a first metadata management center;
[0007] Comparing the first metadata management center and the preset second metadata management center, wherein the second metadata management center is used to store the original access path of each storage medium;
[0008] For a high-frequency access path that exists in the first metadata management center but does not exist in the second metadata management center, migrate the original access path corresponding to the high-frequency access path to a high-frequency storage medium corresponding to the high-frequency access path, wherein the high-frequency storage medium is pre-set based on the access frequency of the high-frequency access path;
[0009] For a low-frequency access path that exists in the second metadata management center but not in the first metadata management center, the original access path corresponding to the low-frequency access path is migrated to a low-frequency storage medium corresponding to the low-frequency access path, wherein the low-frequency storage medium is pre-set based on the access frequency of the low-frequency access path.
[0010] In a possible embodiment, determining, based on the access log, a first access path whose access frequency is higher than a preset frequency threshold, and storing the first access path in a preset storage medium includes:
[0011] Extracting each access information contained in the access log, each access information including an access path, an access time and a path identifier;
[0012] The path identifiers in each access information are stored in a bidirectional linked list in the order of access time, and each access information is stored in a hash table, wherein the path identifier with the latest access time is stored at the head of the bidirectional linked list, and the path identifier with the earliest access time is stored at the tail of the bidirectional linked list;
[0013] Based on each first path identifier in the bidirectional linked list, obtaining the access frequency corresponding to each first path identifier from the hash table;
[0014] The first access paths corresponding to the first path identifiers are stored in the storage medium corresponding to the access frequencies, wherein the first access paths with access frequencies higher than a first frequency threshold are stored in the first storage medium, and the first access paths with access frequencies lower than the first frequency threshold but higher than a second frequency threshold are stored in the second storage medium.
[0015] In a possible embodiment, the method further includes:
[0016] When the doubly linked list overflows, the path identifier at the end of the list is deleted so that the doubly linked list does not overflow.
[0017] In a possible embodiment, migrating the original access path corresponding to the high-frequency access path to the high-frequency storage medium corresponding to the high-frequency access path includes:
[0018] Disabling the write operation permission of the original access path corresponding to the high-frequency access path;
[0019] Migrating data in an original access path corresponding to the high-frequency access path to the high-frequency access path using a preset synchronization tool, wherein the high-frequency access path is in the high-frequency storage medium;
[0020] Check whether the original access path corresponding to the high-frequency access path is consistent with the data in the high-frequency access path. If the data is consistent, update the metadata information of each data in the high-frequency access path, wherein the metadata information includes the access path information.
[0021] According to another aspect of the present invention, a data reading method is provided, wherein the storage path of each data is stored in advance by any of the above-mentioned data heterogeneous storage methods, and the data reading method comprises:
[0022] receiving a read request, wherein the access request includes a target data identifier;
[0023] Based on the metadata information corresponding to the target data identifier, determine the latest storage path as the target storage path;
[0024] Access the target storage path to obtain the target data.
[0025] In a possible embodiment, each storage medium stores data in a multi-computer room storage manner, and the method further includes:
[0026] Determining a target computer room for processing the read request based on an identifier of a client sending the read request;
[0027] In the target computer room, the target data is acquired from a target storage path in a target storage medium based on metadata information identified by the target data.
[0028] According to another aspect of the present invention, a data heterogeneous storage device is provided, the device comprising:
[0029] An acquisition module, used to acquire an access log, wherein the access log records the access path of each of the big data engines and the access time corresponding to the access path;
[0030] A storage module, configured to determine, based on the access log, a first access path whose access frequency is higher than a preset frequency threshold, store the first access path in a preset storage medium, and store description information of the first access path in the preset storage medium in a first metadata management center;
[0031] A comparison module, used for comparing the first metadata management center with a preset second metadata management center, wherein the second metadata management center is used for storing the original access path of each storage medium;
[0032] A migration module, for migrating, for a high-frequency access path that exists in the first metadata management center but does not exist in the second metadata management center, an original access path corresponding to the high-frequency access path to a high-frequency storage medium corresponding to the high-frequency access path, wherein the high-frequency storage medium is pre-set based on the access frequency of the high-frequency access path; and for a low-frequency access path that exists in the second metadata management center but does not exist in the first metadata management center, migrating, for the low-frequency access path, an original access path corresponding to the low-frequency access path to a low-frequency storage medium corresponding to the low-frequency access path, wherein the low-frequency storage medium is pre-set based on the access frequency of the low-frequency access path.
[0033] According to another aspect of the present invention, a data reading device is provided, wherein the storage path of each data is stored in advance by any of the above-mentioned data heterogeneous storage methods, and the data reading device comprises:
[0034] A receiving module, used for receiving a read request, wherein the access request includes a target data identifier;
[0035] A determination module, configured to determine the latest storage path as the target storage path based on metadata information corresponding to the target data identifier;
[0036] An access module is used to access the target storage path to obtain the target data.
[0037] According to another aspect of the present invention, there is provided an electronic device, comprising:
[0038] Processor; and
[0039] Memory for storing programs,
[0040] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes any of the above-mentioned data heterogeneous storage methods and / or data reading methods.
[0041] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned data heterogeneous storage methods and / or data reading methods.
[0042] One or more technical solutions provided in an embodiment of the present invention store a first access path whose access frequency is higher than a preset frequency threshold in a preset storage medium, and store description information of the first access path in the preset storage medium in a first metadata management center, and store description information of the original access path in each storage medium in a second metadata management center. By comparing the access paths in the first metadata management center and the second metadata management center, the access paths that need to be migrated can be determined. Specifically, since the description information of the first metadata management center stores the access path in the storage medium corresponding to its access frequency, if the access path exists in the first metadata management center but not in the second metadata management center, it means that the current storage medium of the access path does not match its access frequency, and therefore, these access paths need to be migrated. If the access path exists in the second metadata management center but not in the first metadata management center, it means that the access frequency of the access path is low, and therefore, these access paths also need to be migrated.
[0043] In this way, by comparing to determine whether the access path needs to be migrated, the access path can be switched to the storage medium corresponding to its access frequency in a timely manner, that is, the access path is stored in a suitable storage medium, which reduces storage costs while better responding to data reading requests and improving data query efficiency. The path that needs to be migrated can be determined only through simple comparison, and the amount of data processing is relatively small. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0045] Figure 1 A schematic diagram of a process flow of a data heterogeneous storage method provided by an embodiment of the present invention;
[0046] Figure 2 A schematic diagram of a process for recording access logs in an embodiment of the present invention;
[0047] Figure 3 A schematic diagram of another process of the data heterogeneous storage method provided by an embodiment of the present invention;
[0048] Figure 4 A schematic diagram of a process of migrating a storage path in an embodiment of the present invention;
[0049] Figure 5 A schematic diagram of another process of the data heterogeneous storage method provided by an embodiment of the present invention;
[0050] Figure 6 A schematic diagram of a flow chart of a data reading method provided by an embodiment of the present invention;
[0051] Figure 7 A schematic diagram of another flow chart of a data reading method provided by an embodiment of the present invention;
[0052] Figure 8 A schematic diagram of another flow chart of a data reading method provided by an embodiment of the present invention;
[0053] Fig. 9 A schematic diagram of the structure of a data heterogeneous storage device provided by an embodiment of the present invention;
[0054] Fig.10 A schematic diagram of the structure of a data reading device provided by an embodiment of the present invention;
[0055] Fig.11 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0056] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.
[0057] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.
[0058] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0059] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0060] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.
[0061] In related technologies, the hot and cold tiered storage of big data engine data is usually performed in the following ways:
[0062] (1) Manual hot and cold data stratification based on rules
[0063] In this tiering strategy, enterprises will manually allocate data to different storage media based on pre-set rules or directly determined by IT managers. For example, frequently queried report data will be placed on high-speed SSDs (Solid State Disks or Solid State Drives), while historical archives will be migrated to tapes or low-cost HDDs (Hard Disk Drives). Although this approach allows a certain degree of flexibility and can customize storage strategies based on specific business needs, it is highly dependent on manual decision-making, lacks automation and intelligence, and has difficulty coping with the dynamic changes of large-scale data sets. In addition, the manual tiering process is time-consuming and error-prone, and is also limited to a single data center or storage system. It lacks support for off-site storage and cannot achieve cross-regional data protection and optimize access efficiency.
[0064] (2) Tiered storage strategy based on fixed ratio
[0065] Under this strategy, the storage system is configured with a fixed ratio of hot, warm, and cold storage areas when it is initialized, for example, 30% hot storage (such as SSD), 40% warm storage (such as SAS (Serial Attached SCSI, Serial Attached SCSI (Small Computer System Interface)) hard disk) and 30% cold storage (such as NL-SAS (near-line SAS, a combination of SAS interface and SATA (Serial ATA, Serial ATA (Advanced Technology Attachment)) disk) or tape). Data is allocated according to preset rules. For example, newly written data enters the hot zone by default, and gradually migrates to the warm zone and cold zone as time goes by or the access frequency decreases. However, this method ignores the dynamic changes in the actual data access mode, and the storage resource allocation is not flexible enough. It cannot be intelligently adjusted according to the real-time access frequency, and it cannot achieve efficient storage and access across regions.
[0066] (3) Static hot and cold separation based on database internal partitions
[0067] In this method, the entire table in the database is partitioned according to the attributes of the data (such as timestamp) during the database design phase. For example, recent data is placed in a high-performance partition and historical data is placed in a low-performance partition. Although this method can improve query efficiency to a certain extent and reduce the pressure of large table queries, it is a static strategy that does not support dynamic adjustment of the storage medium where the partition is located based on the frequency of data access, nor can it cross to a heterogeneous storage environment in different locations. Partition adjustment usually requires manual intervention, and the operation is complex when the amount of data is large, which can easily lead to data inconsistency or service interruption risks.
[0068] When implementing hot and cold tiered management of big data storage, existing technologies generally have core defects such as insufficient automation and intelligence, single supported storage media, inefficient storage resource utilization, limited flexibility, and lack of data security strategies. These shortcomings limit the overall performance and cost-effectiveness of data storage systems.
[0069] (1) Existing tiered storage strategies are limited to single storage media and homogeneous environments
[0070] In the current field of big data storage, traditional tiered storage strategies have exposed obvious limitations, which are mainly reflected in their neglect of storage media diversity and their lack of application in heterogeneous storage environments. Specifically, these strategies are often based on a single type of storage medium. Whether it is a high-speed solid-state drive or a low-cost mechanical hard drive, they cannot dynamically and intelligently allocate between multiple storage media according to the access frequency and importance of different data. This limitation leads to a huge waste of resources: on the one hand, frequently accessed hot data may be bound to inefficient storage devices, increasing the delay in data acquisition and affecting the agility of business operations; on the other hand, cold data that has not been accessed for a long time occupies expensive high-speed storage resources, which increases maintenance costs.
[0071] On the other hand, as enterprises have increasingly higher requirements for data security and business continuity, heterogeneous storage architectures based on multiple data centers, such as the "three locations and one center" model, are gradually becoming industry standards. However, the current tiered storage technology is inadequate in this regard. Their design and implementation do not fully consider how to achieve efficient tiering and migration of data between different geographical locations and different storage platforms. This limitation not only limits the optimization of data access speed, but also greatly reduces data recovery capabilities and business continuity in the face of regional disasters, increasing the risk of data loss and business interruption.
[0072] (2) Insufficient level of automation and intelligence
[0073] Existing technologies mostly rely on preset rules or manual configuration to distinguish between hot and cold data, and it is difficult to respond to changes in data access patterns in real time. This means that even if the data access popularity changes, the storage strategy cannot be adjusted in time, resulting in hot data being incorrectly stored on low-speed media, reducing access efficiency, while cold data may occupy expensive high-speed storage resources, increasing unnecessary costs.
[0074] (3) Inefficient use of storage resources
[0075] Although the method of dividing the storage area by a fixed ratio or static partitioning within the database is simple and easy to implement, it ignores the dynamic nature of data access and the volatility of storage demand. This rigid resource allocation strategy cannot be flexibly adjusted according to actual needs, which easily leads to waste or shortage of storage resources and fails to achieve the optimal balance between storage cost and performance.
[0076] In order to reduce data costs and improve data reading efficiency, an embodiment of the present invention provides a data heterogeneous storage method, a data reading method, an apparatus, an electronic device and a storage medium. The data heterogeneous storage method provided by the embodiment of the present invention can be applied to any electronic device with data storage function, which can be a computer, a server or a mobile terminal, etc.
[0077] like Figure 1 As shown, Figure 1 A schematic diagram of a flow chart of a data heterogeneous storage method provided in an embodiment of the present invention may include the following steps:
[0078] S101, obtaining an access log, wherein the access log records the access paths of various data engines and the access time corresponding to the access paths;
[0079] S102: Based on the access log, determine a first access path whose access frequency is higher than a preset frequency threshold, store the first access path in a preset storage medium, and store description information of the first access path in the preset storage medium in a first metadata management center;
[0080] S103, comparing the first metadata management center with a preset second metadata management center, wherein the second metadata management center is used to store original access paths of each storage medium;
[0081] S104, for a high-frequency access path that exists in the first metadata management center but does not exist in the second metadata management center, migrate an original access path corresponding to the high-frequency access path to a high-frequency storage medium corresponding to the high-frequency access path, wherein the high-frequency storage medium is pre-set based on the access frequency of the high-frequency access path;
[0082] S105. For a low-frequency access path that exists in the second metadata management center but does not exist in the first metadata management center, migrate the original access path corresponding to the low-frequency access path to a low-frequency storage medium corresponding to the low-frequency access path, wherein the low-frequency storage medium is pre-set based on the access frequency of the low-frequency access path.
[0083] By applying the embodiment of the present invention, a first access path having an access frequency higher than a preset frequency threshold is stored in a preset storage medium, and description information of the first access path in the preset storage medium is stored in a first metadata management center, and description information of the original access path in each storage medium is stored in a second metadata management center. By comparing the access paths in the first metadata management center and the second metadata management center, the access paths that need to be migrated can be determined. Specifically, since the description information of the first metadata management center stores the access paths in the storage medium corresponding to their access frequencies, if the access path exists in the first metadata management center but not in the second metadata management center, it means that the current storage medium of the access path does not match its access frequency, and therefore, these access paths need to be migrated. If the access path exists in the second metadata management center but not in the first metadata management center, it means that the access frequency of the access path is low, and therefore, these access paths also need to be migrated.
[0084] In this way, by comparing to determine whether the access path needs to be migrated, the access path can be switched to the storage medium corresponding to its access frequency in a timely manner, that is, the access path is stored in a suitable storage medium. By flexibly scheduling each storage medium, it is possible to dynamically and intelligently allocate between multiple storage media according to the access frequency and importance of different data, thereby reducing storage costs and improving storage efficiency. At the same time, it can better respond to data reading requests and improve data query efficiency. Moreover, the path that needs to be migrated can be determined only through simple comparison, and the amount of data processing is relatively small.
[0085] The above S101-S105 are exemplarily described below:
[0086] The above access log records the data writing status of each big data engine, among which the major data engines can include Hive, Spark, Presto, etc. In the field of big data technology, it is usually necessary to use big data engines to perform calculations based on business data and obtain results. Big data engines can store these results persistently.
[0087] In a possible embodiment, each data engine can store data in HDFS (Hadoop Distributed File System). HDFS is a distributed file system designed to be suitable for general-purpose hardware and can provide high-throughput data access. HDFS can store data through storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive).
[0088] As a possible implementation, the data generated by the big data engine can be stored in HDFS-HDD (HDFS cluster with HDD disk as storage medium) by default. After storing the data, HDFS-HDD can write the writing status of the data into the access log of HDFS-HDD. The writing status can include the storage path, storage time, business identification, writing type, etc. of the written data. The writing type can be new addition, modification, etc. The above access log can record all the data reading and writing status in HDFS-HDD. For example, for write data requests, the writing status of the data can be recorded. For read data requests, the reading status of the data can be recorded. The reading status can include the access path, reading time, business identification, and user identification of the read data, etc. In a possible embodiment, different access logs can be set for different data operation types. For example, write operation access logs and read operation access logs can be set.
[0089] like Figure 2 As shown, Figure 2 Schematic diagram of a process of recording access logs in an embodiment of the present invention. Specifically, after the big data engine performs data calculation to obtain the calculation result, it sends a write request to HDFS-HDD. After HDFS-HDD stores the data, it records the data writing status to the access log (i.e. Figure 2 in the audit log).
[0090] In a possible embodiment, access logs may be set for different storage media, and the access logs may record the data reading and writing status in the corresponding storage media. The above storage media may include HDFS-HDD, HDFS-SSD ((HDFS cluster with SSD disk as storage medium)), Alluxio and BOS object storage, etc. Among them, Alluxio is an open source, memory-centric virtual distributed storage system. It is designed as an intermediate layer between the underlying storage system (such as HDFS, Amazon S3, Azure Blob Storage, etc.) and the upper-level computing framework (such as Apache Spark, MapReduce, Presto, etc.). The core value of Alluxio lies in its ability to provide a unified data access interface so that data can be quickly loaded into memory, thereby speeding up data access, which is particularly suitable for big data and machine learning applications that require high-speed data access and processing. BOS object storage is an object-based massive storage service that can provide low-cost data storage capabilities.
[0091] When using the access log to obtain the access frequency of each storage path, the access logs of different storage media can be aggregated, that is, the storage path reading situation is obtained from each access log. As a possible implementation, since each write situation or read situation in the access log is written in text form, it is possible to preset the data information that needs to be read, and match each access log based on the data information to obtain the data information recorded in the access log. Exemplarily, a regular matching formula can be written for the data information that needs to be read, and the data information can be read from each access log through the regular matching formula. The above data information can be flexibly set according to actual needs. As a possible implementation, the above data information may include data access path, access time, and operation type, etc., wherein the operation type may be write, read or delete, etc.
[0092] After extracting the above data information, the data information can be stored. In a possible embodiment, the above data information can be stored through a hash table and a bidirectional linked list. Among them, the bidirectional linked list is mainly used to store data access records. Each file data record may include access path, access time, path identifier, operation type, data size, and operation status, etc., wherein the above data identifier can be a data name or data ID, etc., which can uniquely identify the data. The operation type can be read, write or delete, etc., and the operation status can be success or failure. As a possible implementation method, each data access record can be stored in a bidirectional linked list according to the access time. Specifically, the data access record with the most recent access time is stored at the head of the linked list, and the data access record with the longest access time is stored at the end of the linked list. In this way, the most recently accessed file can be quickly found according to the storage position of each data access record in the linked list.
[0093] The hash table is mainly used to quickly locate file records. It usually stores data in the form of Key-Value pairs. In the embodiment of the present invention, the key in the hash table can be a data identifier, and the value can be a linked list node reference pointer, data access information summary, etc. The linked list node reference pointer refers to a pointer to the corresponding data access record in the doubly linked list. The data access information summary can be the number of accesses, the latest access time, etc. By using the hash table to store data access records, all access records of specific data can be quickly retrieved without traversing the entire linked list, which greatly improves the query efficiency.
[0094] In a possible embodiment, the access frequency of each access path can be extracted based on the above-mentioned hash table and the bidirectional linked list. Specifically, the data with the earliest access time can be obtained from the bidirectional linked list based on the path identifier of each data, and the number of access times of each path can be directly obtained based on the path identifier and the path access information summary stored in the hash table.
[0095] As a possible implementation, the frequency range can be pre-divided, each frequency range corresponds to a data type, and different data types are stored in different storage media. The above data types may include hot data, high-frequency access warm data, low-frequency access warm data, and cold data, where hot data is the data type with the highest access frequency, and cold data is the data with the lowest access frequency. Exemplarily, data with more than 30 daily accesses in the past week can be set as hot data, data with 10 to 30 daily accesses in the past week can be set as high-access warm data, data that has not been accessed in the past 6 months can be set as cold data, and the rest of the data is low-frequency access warm data. The storage media of each data type can be pre-set according to the read and write efficiency of each storage medium. Exemplarily, hot data can be stored in Alluxio, high-frequency access warm data can be stored in HDFS-SSD, low-frequency access warm data can be stored in HDFS-HDD, and cold data can be stored in BOS.
[0096] After each access path is stored in a storage medium corresponding to the access frequency, the metadata information of each access path in the corresponding storage medium may be uniformly stored in the first metadata management center. The above metadata information refers to data description information, which may include the storage path of the data, the corresponding business identifier and other information. Specifically, the above storage path may include a storage medium identifier and an access path identifier, such as alluxio: / / / baihua / user / zhuanli.
[0097] The above process of determining the data type and storage medium of different data can be implemented by a preset LRU (Least Recently Used) algorithm. The basic idea of the LRU algorithm is: when the cache is full, remove the data item that has been used least recently. This ensures that the most active data is stored in the cache. In a possible embodiment, when a new data access record is received, the LRU algorithm can first check whether the data targeted by the new data access record is already in a bidirectional linked list. If it is, it is removed from the current position of the data in the access linked list and placed at the head of the linked list, indicating the most recent access. If it is not, and the storage in the bidirectional linked list is full, the oldest accessed file (i.e., the file that has not been used for the longest time) is first removed from the tail of the linked list, and then the newly accessed file is inserted into the head of the linked list. After a period of statistics, the file at the head of the LRU linked list is the hot data with the highest access frequency or the most recently accessed.
[0098] As a possible implementation, the LRU algorithm can perform hot and cold data evaluation on each data according to a preset time period to determine the storage medium of the data. The above time period can be set according to the actual application scenario, such as every hour, every day, etc.
[0099] Since the access frequency of data may change over time, the current storage medium of the data may not be able to meet business needs, so it is necessary to perform a hot and cold assessment of the data and migrate the data that needs to be migrated. In a possible embodiment, it can be determined whether there is data that needs to be migrated based on the metadata information stored in the first metadata management center and the metadata information of the access path stored in each storage medium. The metadata information of the above-mentioned storage media and the access path stored in the storage medium can be stored in the second metadata management center. The metadata information can also include a storage medium identifier and an access path identifier, such as hdfs: / / / baihua / user / zhuanli.
[0100] In a possible embodiment, the first metadata management center and the second metadata management center can be compared to filter out inconsistent data in the first metadata management center and the second metadata management center. Among them, the data that exists in the first metadata management center but not in the second metadata management center, because the metadata information stored in the first metadata management center is determined based on the real-time access frequency of the data, and the metadata information stored in the second metadata management center is the original storage information of the data, therefore, if the data exists in the first metadata management center but not in the second metadata management center, it means that the original storage location of the data does not match the access frequency of the data, so the data can be stored in the corresponding storage medium according to the access frequency of the data. Exemplarily, path B originally belongs to low-frequency access warm data, which is stored in HDFS-HDD, but the number of daily accesses in the past week is greater than 30 times, which meets the standard of hot data, then the data will be reallocated to the Alluxio storage medium at this time.
[0101] If the data exists in the second metadata management center but not in the first metadata management center, it means that the path may not have been accessed for a long time, so the storage medium of the data can be changed. Specifically, the data can be migrated from the current data to a storage medium that meets the access frequency of the data. For example, path A is considered hot data and is currently stored in Alluxio. However, when the storage policy evaluation is triggered this time, it is found that path A does not meet the hot data standard. In this case, the storage medium will be adjusted based on the recent access situation of path A.
[0102] like Figure 3 As shown, Figure 3 A schematic diagram of a process for determining whether a storage medium of data needs to be changed in an embodiment of the present invention. Specifically, the storage medium of data can be changed based on a write operation access log (ie Figure 3 Audit logs (write) in the Figure 3The access information of each access path is extracted from the audit log (read) in the hash table and the bidirectional linked list. The access information of each access path is stored in the hash table and the bidirectional linked list. The LRU algorithm is used to determine the access paths with access frequencies such as the number of daily visits in the past week being greater than 30, the number of daily visits in the past week being between 10-30, and no one accessing in the past six months, and the metadata information of the access paths with the number of daily visits in the past week being greater than 30, the number of daily visits in the past week being between 10-30, and no one accessing in the past six months is stored in the first metadata management center (i.e., Figure 3 hot and cold standby storage metadata management center).
[0103] At the same time, each storage medium has a metadata management center (collectively referred to as set 2) for storing metadata information of access paths in the storage medium. Set 2 specifically includes Alluxio metadata management center, HDFS-SSD metadata management center, object storage metadata management center, and HDFS-HDD metadata management center. The storage policy evaluator can then be used to determine the path set that exists in set 2 but not in set 1, and the path set that exists in set 1 but not in set 2. The path set contains path information, access times, and the current storage location of the path. The above path set is the path set that needs to change the storage location.
[0104] When it is confirmed that the storage medium of the data needs to be changed, information such as the source path of the data to be changed, the number of source path accesses, the source path storage medium, and the path target storage medium can be extracted. The above target storage medium is determined based on the access frequency of the data.
[0105] For the path set that exists in set 2 but not in set 1, it can be separated again, that is, its storage medium is determined based on its access frequency. For the path set that exists in set 1 but not in set 2, the data synchronization module can be used to store these paths in each storage medium according to the access frequency.
[0106] In a possible embodiment, Figure 4 As shown, before data migration, the write operation permission of the source path is first prohibited to prevent data inconsistency problems caused by data updates. During data migration, data can be synchronized to the target path through synchronization tools such as Hadoop distcp, and the above synchronization tools can be flexibly selected according to the actual application scenario. When the target path synchronization is completed, the data consistency check can be started to determine whether the data in the source path and the target path are consistent. For example, the file size and number of files of the source data and the target data can be checked.
[0107] In a possible embodiment, after determining that the data is consistent, such as Figure 3 As shown, it can be determined whether the target path belongs to the location storage path in the table or table partition (i.e., the original path when the table or table partition is created in HDFS). If so, Hivemetastore is called to update the target path storage information so that when the big data engine accesses the path again, it is directly routed to the changed storage path for access. In a possible embodiment, after the data is migrated, a timer can be started for both the source path and the target path. After 24 hours, the write-forbidden constraint of the path will be lifted, and then the source data will be deleted through deletion tools such as Hadooprm.
[0108] Figure 5 A schematic diagram of a data heterogeneous storage method provided by an embodiment of the present invention, specifically, the data accessed by the big data engine is stored separately for hot and cold data, data with high access frequency is stored in high-speed access media, data with low access frequency is stored in low-speed access media, and data that is almost not accessed but still needs to be retained is stored in cold standby access media, so as to achieve a balance between access speed and storage cost. For data in different storage media, data will be intelligently migrated according to the actual access situation of the engine.
[0109] According to another aspect of the present invention, a data reading method is provided, wherein the storage path of each data is stored in advance by any of the above-mentioned data heterogeneous storage methods, such as Figure 6 As shown, the data reading method includes:
[0110] S601, receiving a read request, wherein the access request includes a target data identifier;
[0111] S602: Determine the latest storage path as the target storage path based on the metadata information corresponding to the target data identifier;
[0112] S603: Access the target storage path to obtain the target data.
[0113] In a possible embodiment, each storage medium stores data in a multi-computer room storage manner, and the method further includes:
[0114] Determining a target computer room for processing the read request based on an identifier of a client sending the read request;
[0115] In the target computer room, the target data is acquired from a target storage path in a target storage medium based on metadata information identified by the target data.
[0116] In an embodiment of the present invention, various storage media can be deployed in multiple computer rooms and / or remotely to meet the high security requirements of data disaster recovery. In a possible embodiment, when a read request sent by a big data engine is received, the read request can be routed to the nearest data copy according to the client information that sends the read request, such as routing the read request to a computer room that is closer to the client IP address, or routing the read request to a computer room preset for the client. At the same time, the read request can be stored in the access log.
[0117] like Figure 7 As shown, Figure 7 A flow chart of processing data read requests in an embodiment of the present invention: In a possible embodiment, after receiving an RPC read request sent by a client, a big data engine such as Hive, spark, and presto can send the read request to a computer room diverter, which can route the read request to a corresponding computer room based on client information, and an access diverter in the computer room can obtain the target data from the corresponding storage medium based on the storage medium where the target data contained in the read request is located and return it. After the processing is completed, the read access can be recorded in the access log, specifically, the access data identifier, access time and other information can be recorded.
[0118] In the computer room, the access type can be determined based on the read request, and the access type can include table access operations and path access operations. Exemplarily, a read request in the form of SQL can be determined as a table access operation, and a read request in other forms can be determined as a path access operation.
[0119] For table access operations, the original path for creating the table / table partition and the new path after storage tiering can be stored in the metadata of the hivemetastore layer. For example, when creating table A, the location is specified as hdfs: / / / baihua / user / zhuanli. Through the hot and cold evaluation of data access, the path is judged to be a hot path and then transferred to alluxio: / / / baihua / user / zhuanli. Then two fields are stored in the hivemetastore layer: the original path (hdfs: / / / baihua / user / zhuanli) and the new path (alluxio: / / / baihua / user / zhuanli). When accessing the table, the new path address of the hivemetastore layer is first queried. If the path is determined to be of Alluxio type, AlluxioFS is initialized; if it is of HDFS type, including HDFS-SSD and HDFS-HDD, HdfsFS is initialized; if the path is of BOS type, BosFS is initialized. According to the initialized storage medium type, the big data engine accesses the underlying data of the corresponding storage medium.
[0120] For the path access operation, a corresponding file system may be created based on the storage medium of the target data targeted by the read request, and the target data may be accessed through the file system.
[0121] like Figure 8 As shown, Figure 8 The figure is a flow chart of obtaining target data in a computer room in an embodiment of the present invention. Specifically, the access splitter determines the access type based on the type of the read request. If it is an access table type, the Hivemetastore layer determines the table storage path location and obtains the target data from the latest path and returns it. If it is an access path type, a file system FileSystem is created according to the storage medium where the target data contained in the read request is located, and the target data in the corresponding storage medium is accessed and returned through the file system.
[0122] Based on the same inventive concept, an embodiment of the present invention further provides a data heterogeneous storage device, such as Fig. 9 As shown, the device 900 may include:
[0123] An acquisition module 901 is used to acquire an access log, wherein the access log records the access path of each of the big data engines and the access time corresponding to the access path;
[0124] The storage module 902 is used to determine, based on the access log, a first access path whose access frequency is higher than a preset frequency threshold, store the first access path in a preset storage medium, and store description information of the first access path in the preset storage medium in a first metadata management center;
[0125] A comparison module 903, used to compare the first metadata management center with a preset second metadata management center, wherein the second metadata management center is used to store the original access path of each storage medium;
[0126] The migration module 904 is used to migrate the original access path corresponding to the high-frequency access path that exists in the first metadata management center but does not exist in the second metadata management center to the high-frequency storage medium corresponding to the high-frequency access path; and to migrate the original access path corresponding to the low-frequency access path that exists in the second metadata management center but does not exist in the first metadata management center to the low-frequency storage medium corresponding to the low-frequency access path.
[0127] According to another aspect of the present invention, a data reading device is provided, wherein the storage path of each data is stored in advance by any of the above-mentioned data heterogeneous storage methods, such as Fig.10 As shown, the data reading device 1000 includes:
[0128] The receiving module 1001 is used to receive a read request, wherein the access request includes a target data identifier;
[0129] A determination module 1002 is used to determine the latest storage path as the target storage path based on the metadata information corresponding to the target data identifier;
[0130] The access module 1003 is used to access the target storage path to obtain the target data.
[0131] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0132] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.
[0133] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.
[0134] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.
[0135] refer to Fig.11 , a block diagram of an electronic device 1100 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0136] like Fig.11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.
[0137] Multiple components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 can be any type of device that can input information to the electronic device 1100, and the input unit 1106 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 1107 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 can include but is not limited to a disk, an optical disk. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0138] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned data heterogeneous storage method and data reading method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. In some embodiments, the computing unit 1101 may be configured to perform the above-mentioned data heterogeneous storage method and data reading method by any other appropriate means (e.g., by means of firmware).
[0139] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0140] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0141] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0143] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0144] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
Claims
1. A data heterogeneous storage method, characterized in that: The method comprises: Obtaining access logs, wherein the access logs record the access paths of various data engines and the access times corresponding to the access paths; Based on the access log, determine a first access path whose access frequency is higher than a preset frequency threshold, store the first access path in a preset storage medium, and store description information of the first access path in the preset storage medium in a first metadata management center; Comparing the first metadata management center and the preset second metadata management center, wherein the second metadata management center is used to store the original access path in each storage medium; For a high-frequency access path that exists in the first metadata management center but does not exist in the second metadata management center, migrate the original access path corresponding to the high-frequency access path to a high-frequency storage medium corresponding to the high-frequency access path, wherein the high-frequency storage medium is pre-set based on the access frequency of the high-frequency access path; For a low-frequency access path that exists in the second metadata management center but does not exist in the first metadata management center, the original access path corresponding to the low-frequency access path is migrated to a low-frequency storage medium corresponding to the low-frequency access path, wherein the low-frequency storage medium is pre-set based on the access frequency of the low-frequency access path.
2. The method according to claim 1, characterized in that: The determining, based on the access log, a first access path whose access frequency is higher than a preset frequency threshold, and storing the first access path in a preset storage medium includes: Extracting each access information contained in the access log, each access information including an access path, an access time and a path identifier; The path identifiers in each access information are stored in a bidirectional linked list in the order of access time, and each access information is stored in a hash table, wherein the path identifier with the latest access time is stored at the head of the bidirectional linked list, and the path identifier with the earliest access time is stored at the tail of the bidirectional linked list; Based on each first path identifier in the bidirectional linked list, obtaining the access frequency corresponding to each first path identifier from the hash table; The first access paths corresponding to the first path identifiers are stored in the storage medium corresponding to the access frequencies, wherein the first access paths with access frequencies higher than a first frequency threshold are stored in the first storage medium, and the first access paths with access frequencies lower than the first frequency threshold but higher than a second frequency threshold are stored in the second storage medium.
3. The method according to claim 2, characterized in that The method further comprises: When the doubly linked list overflows, the path identifier at the end of the list is deleted so that the doubly linked list does not overflow.
4. The method according to claim 1, characterized in that: The step of migrating the original access path corresponding to the high-frequency access path to the high-frequency storage medium corresponding to the high-frequency access path includes: Disabling the write operation permission of the original access path corresponding to the high-frequency access path; Migrating data in an original access path corresponding to the high-frequency access path to the high-frequency access path using a preset synchronization tool, wherein the high-frequency access path is in the high-frequency storage medium; Check whether the original access path corresponding to the high-frequency access path is consistent with the data in the high-frequency access path. If the data is consistent, update the metadata information of each data in the high-frequency access path, wherein the metadata information includes the access path information.
5. A data reading method, characterized in that: The storage path of each data is stored in advance by any data heterogeneous storage method as described in claims 1 to 4, and the data reading method includes: Receiving a read request, wherein the access request includes a target data identifier; Based on the metadata information corresponding to the target data identifier, determine the latest storage path as the target storage path; Access the target storage path to obtain the target data.
6. The method according to claim 5, characterized in that Each storage medium stores data in a multi-computer room storage manner, and the method further includes: Determining a target computer room for processing the read request based on an identifier of a client sending the read request; In the target computer room, the target data is acquired from a target storage path in a target storage medium based on metadata information identified by the target data.
7. A data heterogeneous storage device, characterized in that: The device comprises: An acquisition module is used to acquire access logs, wherein the access logs record the access paths of various data engines and the access times corresponding to the access paths; A storage module, configured to determine, based on the access log, a first access path whose access frequency is higher than a preset frequency threshold, store the first access path in a preset storage medium, and store description information of the first access path in the preset storage medium in a first metadata management center; A comparison module, used for comparing the first metadata management center with a preset second metadata management center, wherein the second metadata management center is used for storing the original access path of each storage medium; A migration module, for migrating, for a high-frequency access path that exists in the first metadata management center but does not exist in the second metadata management center, an original access path corresponding to the high-frequency access path to a high-frequency storage medium corresponding to the high-frequency access path, wherein the high-frequency storage medium is pre-set based on the access frequency of the high-frequency access path; and for a low-frequency access path that exists in the second metadata management center but does not exist in the first metadata management center, migrating, for the low-frequency access path, an original access path corresponding to the low-frequency access path to a low-frequency storage medium corresponding to the low-frequency access path, wherein the low-frequency storage medium is pre-set based on the access frequency of the low-frequency access path.
8. A data reading device, characterized in that: The storage path of each data is stored in advance by any data heterogeneous storage method as described in claims 1 to 4, and the data reading device includes: A receiving module, used for receiving a read request, wherein the access request includes a target data identifier; A determination module, configured to determine the latest storage path as the target storage path based on metadata information corresponding to the target data identifier; An access module is used to access the target storage path to obtain the target data.
9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-4 or 5-6.
10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-4 or 5-6.
Citation Information
Cited By
Routing method and device for multiple data sources
CN120508536A
Data processing method and device based on heterogeneous storage, equipment and storage medium
CN120909517A
Data processing method and device based on heterogeneous storage, equipment and storage medium
CN120909517B
Online migration management method and device for database hotspot data and medium
CN121092527A
An online migration management method, device and medium for hotspot data of a database
CN121092527B
