Data access method, device, system, storage medium and terminal

Abstract

The present invention discloses a method, device, system, storage medium and terminal for data access. The method comprises: storing first-category data in a first storage system table and second-category data in a second storage system table, wherein the access frequency of the first-category data is greater than that of the second-category data; receiving a data request message, wherein the data request message is used to indicate the storage system table where the data to be processed is located, wherein the storage system table includes the first storage system table and the second storage system table; and returning the first-category data or the second-category data according to the data request message. The present invention solves the technical problem of low access efficiency when accessing hot data and cold data due to the prior art.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method, device, system, storage medium and terminal for data access. Background Art

[0002] In distributed storage technology, taking HBase as an example, HBase is a distributed, column-oriented open source database suitable for storing unstructured data. In the process of commercial use of HBase, as the amount of business data increases, the amount of data stored in HBase also gradually increases.

[0003] Among the stored data, the data that the business cares about most and accesses the longest is called hot data, and the opposite is called cold data, that is, the data has very few accesses and low performance requirements. The characteristic of cold data is large data volume. Among them, the characteristic of hot data is data in a specific range, for example, data in the last 7 days. The business has a high access frequency to this type of data and has high latency requirements.

[0004] With respect to the above-mentioned problem of low access efficiency when accessing hot data and cold data in the prior art, no effective solution has been proposed yet. Summary of the invention

[0005] Embodiments of the present invention provide a method, device, system, storage medium and terminal for data access, so as to at least solve the technical problem of low access efficiency when accessing hot data and cold data in the prior art.

[0006] According to one aspect of an embodiment of the present invention, a method for data access is provided, comprising: storing first-category data in a first storage system table and second-category data in a second storage system table, wherein the first-category data is accessed more frequently than the second-category data; receiving a data request message, wherein the data request message is used to indicate a storage system table where the data to be processed is located, wherein the storage system table includes a first storage system table and a second storage system table; and returning the first-category data or the second-category data according to the data request message.

[0007] Optionally, the data access method is applied to a distributed storage database, the first storage system table is located in a first type of data storage system, and the second storage system table is located in a second type of data storage system.

[0008] Optionally, receiving a data request message includes: parsing a data type of the data to be processed in the data request message, wherein the data type includes: first category data or second category data; determining a storage system table where the data to be processed is located according to the data type; and accessing a first category data storage system or a second category data storage system through a preset calling system.

[0009] Further, optionally, returning the first category of data or the second category of data according to the data request message includes: when the data type of the data to be processed is the first category of data, querying the first storage system table to obtain the data to be processed; when the data type of the data to be processed is the second category of data, querying the second storage system table to obtain the data to be processed.

[0010] Optionally, the method also includes: when the frequency of data access in the first category of data decreases, by synthesizing multiple data files into one data file, identifying the data in the synthesized data file according to the timestamp, storing the data belonging to the first category of data in the first storage system table, and storing the data belonging to the second category of data in the second storage system table.

[0011] Further, optionally, the method further includes: when the process of combining the multiple data files into one data file is not completed, filtering the data according to the timestamp of the data in the first storage system table, and migrating the obtained second type of data to the second storage system table.

[0012] Optionally, the first storage system table and the second storage system table are located in different column families of the same table.

[0013] According to another aspect of an embodiment of the present invention, a data access device is also provided, including: a storage module, used to store first-category data in a first storage system table, and second-category data in a second storage system table, wherein the access frequency of the first-category data is greater than that of the second-category data; a receiving module, used to receive a data request message, wherein the data request message is used to indicate the storage system table where the data to be processed is located, wherein the storage system table includes the first storage system table and the second storage system table; and a response module, used to return the first-category data or the second-category data according to the data request message.

[0014] Optionally, the data access device is applied to a distributed storage database, the first storage system table is located in a first type of data storage system, and the second storage system table is located in a second type of data storage system.

[0015] Optionally, the receiving module includes: a first parsing unit, used to parse the data type of the data to be processed in the data request message, wherein the data type includes: first category data or second category data; a second parsing unit, used to determine the storage system table where the data to be processed is located according to the data type; and an access unit, used to access the first category data storage system or the second category data storage system through a preset calling system.

[0016] Further, optionally, the response module includes: a first response unit, used to query the first storage system table to obtain the data to be processed when the data type of the data to be processed is the first category of data; and a second response unit, used to query the second storage system table to obtain the data to be processed when the data type of the data to be processed is the second category of data.

[0017] Optionally, the device also includes: a first data migration module, which is used to, when the data access frequency in the first category of data decreases, store the data belonging to the first category of data in the first storage system table and store the data belonging to the second category of data in the second storage system table by synthesizing multiple data files into one data file, identifying the data according to the timestamp of the data in the synthesized data file.

[0018] Further, optionally, the device also includes: a second data migration module, which is used to filter the data according to the timestamp of the data in the first storage system table when the process of synthesizing multiple data files into one data file is not completed, and migrate the obtained second type of data to the second storage system table.

[0019] According to another aspect of an embodiment of the present invention, there is also provided a data access system, comprising: a client and a storage system, wherein the client is used to send a data request message to the storage system, wherein the data request message is used to indicate a storage system table where the data to be queried is located, wherein the storage system table comprises a first storage system table and a second storage system table; the storage system comprises a first type of data storage system and a second type of data storage system, the first type of data storage system and the second type of data storage system are respectively connected to a preset calling system, receive the data request message through the preset calling system, and obtain the first type of data from the first storage system table or the second type of data storage system by accessing the first type of data storage system or the second type of data storage system.

[0020] According to another aspect of an embodiment of the present invention, a storage medium is provided. The storage medium includes a stored program, and is characterized in that when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned data access method.

[0021] According to another aspect of an embodiment of the present invention, a terminal is provided, including a storage medium and a processor, wherein the processor is used to run a program stored in the storage medium, and is characterized in that the above-mentioned data access method is executed when the program is running.

[0022] In an embodiment of the present invention, cold data and hot data are stored in two different HBase tables respectively, by storing first-category data in a first storage system table and second-category data in a second storage system table, wherein the access frequency of the first-category data is greater than that of the second-category data; receiving a data request message, wherein the data request message is used to indicate the storage system table where the data to be processed is located, wherein the storage system table includes a first storage system table and a second storage system table; returning the first-category data or the second-category data according to the data request message, thereby achieving the purpose of accurately accessing hot data, thereby realizing the technical effect of improving the access efficiency to hot data and cold data, and further solving the technical problem of low access efficiency when accessing hot data and cold data due to the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 is a schematic diagram of a data access system according to Embodiment 1 of the present invention;

[0025] Figure 2 is a schematic diagram of the structure of a storage system in a data access system according to Embodiment 2 of the present invention;

[0026] Figure 3 It is a hardware structure block diagram of a computer terminal of a data access method according to an embodiment of the present invention;

[0027] Figure 4 is a flow chart of a method for data access according to Embodiment 2 of the present invention;

[0028] Figure 5 It is a schematic diagram of a data access device according to the third embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Technical terms involved in the embodiments of this application:

[0032] Cold data: refers to data that is accessed less frequently and can tolerate greater access delays.

[0033] Cold storage: Storage media used to store cold data, usually with larger access delays and lower costs.

[0034] HFile: The file format for storing data in HBase.

[0035] Example 1

[0036] According to another aspect of the embodiment of the present invention, a data access system is provided. Figure 1 Schematic diagram of a data access system according to Embodiment 1 of the present invention. Figure 1 As shown, it includes: a client 12 and a storage system 14.

[0037] Among them, the client 12 is used to send a data request message to the storage system 14, wherein the data request message is used to indicate the storage system table where the data to be queried is located, wherein the storage system table includes a first storage system table and a second storage system table; the storage system 14 includes a first type of data storage system and a second type of data storage system, the first type of data storage system and the second type of data storage system are respectively connected to a preset calling system, receive the data request message through the preset calling system, and obtain the first type of data from the first storage system table stored by accessing the first type of data storage system or the second type of data storage system, or obtain the second type of data from the second storage system table stored.

[0038] Specifically, Figure 1As shown, taking reading data as an example, the data access system provided in the embodiment of the present application can be applied to the HBase system, wherein the first category of data in the embodiment of the present application is hot data, and the second category of data is cold data, wherein hot data is data with a high access frequency and a low access delay; conversely, cold data is data with a low access frequency and can tolerate a large access delay.

[0039] In the data access system provided in the embodiment of the present application, cold data and hot data are stored in corresponding systems respectively. Figure 1 In the embodiment of the present application, the hot data is stored in the first storage system table in the distributed file system (Hadoop Distributed File System, referred to as HDFS) (i.e., the first type of data storage system in the embodiment of the present application) system as an example, and the cold data is stored in the second storage system table in the object storage (Object Storage Service, referred to as OSS) (i.e., the second type of data storage system in the embodiment of the present application) system as an example for explanation. Based on the characteristics of HBase technology, the access in HBase storage often corresponds to a storage system. In order to separate cold data from hot data, the HDFS system and the OSS system can be accessed separately during the access process. Figure 2 FIG. 1 is a schematic diagram of the structure of a storage system in a data access system according to Embodiment 2 of the present invention. Figure 2 As shown, a data request message sent by the client 12 is received through a distributed file system (Distributed File System), and the HDFS system and the OSS system are respectively called through the Distributed FileSystem according to the data request message. Therefore, when the user sends a data request message to the storage system 14 through the client 12, the data request message will be used to indicate whether hot data or cold data is requested. In the embodiment of the present application, hot data is stored in a first storage system table (the first storage system table is stored in the HDFS system), and cold data is stored in a second storage system table (the second storage system table is stored in the OSS system), so as to avoid the problem of erroneously hitting cold data during access to hot data.

[0040] Among them, two different file systems are used in the data access system provided in the embodiment of the present application, and each of them can be expanded, which is more flexible to use; cold data and hot data are placed in two different tables, and the user must clearly specify whether to query the hot table or the cold table when querying data, and there is no need to analyze the user's semantics in HBase. In addition, in the embodiment of the present application, the cold data is also stored in the HBase table, which can provide real-time reading and writing; the process of importing data from the first storage system table to the second storage system table is completed by synthesizing multiple files into one file (compact process), directly operating the underlying HFile, and the synchronization efficiency is faster than using API reading and writing.

[0041] In the data access system provided in the embodiment of the present application, since hot data will become cold data as the access frequency decreases over time, the cold data is identified through the compact process, that is, the data is identified according to the timestamp of the data in the first storage system table, and the identified cold data is moved from the first storage system table to the second storage system table;

[0042] However, before the compaction is completed, some cold data will appear in the first storage system table. Therefore, the client has processed this situation. When the user queries the first storage system table, the cold data will be automatically filtered out. The data will be filtered according to the timestamp of the data in the first storage system table. When the user queries the second storage system table, the cold data in the first storage system table will also be queried out and merged with the cold data in the second storage system table and returned to the user.

[0043] It should be noted that the data access system provided in the embodiment of the present application is only illustrated by taking the HDFS system and the OSS system as examples, and other systems that can be applied to the HBase technology shall prevail, and there is no specific limitation. In addition, in the embodiment of the present application, cold data and hot data can also be stored in different column families in the same table for implementation.

[0044] Example 2

[0045] According to an embodiment of the present invention, a method embodiment of data access is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] The method embodiment provided in the second embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 3FIG. 1 is a hardware structure diagram of a computer terminal of a data access method according to an embodiment of the present invention. Figure 3 As shown, the computer terminal 30 may include one or more (only one is shown in the figure) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission module 306 for communication functions. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations are shown.

[0047] The memory 304 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the data access method in the embodiment of the present invention. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, the data access method of the above-mentioned application is realized. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 304 may further include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the computer terminal 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The transmission module 306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 30. In one example, the transmission module 306 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] Under the above operating environment, this application provides Figure 4 The data access method is shown. Figure 4 is a flow chart of a method for data access according to a second embodiment of the present invention.

[0050] Step S402, storing the first type of data in a first storage system table, and storing the second type of data in a second storage system table, wherein the access frequency of the first type of data is greater than that of the second type of data;

[0051] In the above step S402 of the present application, the data access method provided in the embodiment of the present application can be applied to the data access system in Example 1, wherein the data access method provided in the embodiment of the present application can be applied to the HBase system, wherein the first category of data in the embodiment of the present application is hot data, and the second category of data is cold data, wherein hot data is data with a high access frequency and a low access delay; conversely, cold data is data with a low access frequency and can tolerate a large access delay.

[0052] Optionally, the data access method provided in the embodiment of the present application is applied to a distributed storage database (HBase), the first storage system table is located in a first type of data storage system, and the second storage system table is located in a second type of data storage system.

[0053] In the data access method provided in the embodiment of the present application, hot data is stored in a distributed file system (Hadoop Distributed File System, referred to as HDFS) (i.e., the first type of data storage system in the embodiment of the present application) system by storing it in a first storage system table, and cold data is stored in an object storage (Object Storage Service, referred to as OSS) (i.e., the second type of data storage system in the embodiment of the present application) system by storing it in a second storage system table.

[0054] In the embodiment of the present application, hot data is stored in the first storage system table (the first storage system table is stored in the HDFS system), and cold data is stored in the second storage system table (the second storage system table is stored in the OSS system), so as to avoid the problem of erroneously hitting cold data during access to hot data.

[0055] Step S404, receiving a data request message, wherein the data request message is used to indicate a storage system table where the data to be processed is located, wherein the storage system table includes a first storage system table and a second storage system table;

[0056] In the above step S404 of the present application, based on step S402, the cold data and the hot data are respectively stored in corresponding systems, and a data request message sent by the client is received, wherein, based on step S402, the data request message sent by the client will indicate that the storage system table where the data to be processed is located is the first storage system table where the hot data is located, or the second storage system table where the cold data is located.

[0057] The data to be processed here can be cold data or hot data, and the client requests cold data or hot data for read operations or write operations.

[0058] Step S406: Return the first category data or the second category data according to the data request message.

[0059] In the above step S406 of the present application, based on the direction in the data request message in step S402, that is, if the data request message requests to obtain hot data, the hot data is returned by querying the first storage system table; if the data request message requests to obtain cold data, the cold data is returned by querying the second storage system table.

[0060] In an embodiment of the present invention, cold data and hot data are stored in two different HBase tables respectively, by storing first-category data in a first storage system table and second-category data in a second storage system table, wherein the access frequency of the first-category data is greater than that of the second-category data; receiving a data request message, wherein the data request message is used to indicate the storage system table where the data to be processed is located, wherein the storage system table includes a first storage system table and a second storage system table; returning the first-category data or the second-category data according to the data request message, thereby achieving the purpose of accurately accessing hot data, thereby realizing the technical effect of improving the access efficiency to hot data and cold data, and further solving the technical problem of low access efficiency when accessing hot data and cold data due to the prior art.

[0061] Optionally, receiving the data request message in step S404 includes:

[0062] Step S4041, parsing the data type of the data to be processed in the data request message, wherein the data type includes: first-category data or second-category data;

[0063] Step S4042, determining the storage system table where the data to be processed is located according to the data type;

[0064] Step S4043, accessing the first type of data storage system or the second type of data storage system through a preset calling system.

[0065] Specifically, in combination with step S4041 to step S4043, due to the pre-separation of hot data and cold data, in the data request message sent by the receiving client, the data request message will indicate whether the accessed data is hot data or cold data. When the accessed data type is hot data, it is determined that the storage system table to be accessed is the first storage system table; conversely, when the accessed data type is cold data, it is determined that the storage system table to be accessed is the second storage system table;

[0066] Furthermore, since the first storage system table is stored in the first type of data storage system and the second storage system table is stored in the second type of data storage system, based on the limitation of HBase technology, if there are two systems under the HBase architecture, they need to be called through the preset calling system, that is, through the distributed file system.

[0067] Further, optionally, returning the first category data or the second category data according to the data request message in step S406 includes:

[0068] Step S4061: when the data type of the data to be processed is the first type of data, query the first storage system table to obtain the data to be processed;

[0069] Step S4062: When the data type of the data to be processed is the second type of data, query the second storage system table to obtain the data to be processed.

[0070] Specifically, based on accessing the first type of data storage system or the second type of data storage system through the preset calling system in step S4041 to step S4043, when it is necessary to access the first type of data storage system, the preset calling system directs the data request message to the first type of data storage system, and the first type of data storage system obtains hot data; conversely, the preset calling system directs the data request message to the second type of data storage system, and the second type of data storage system obtains cold data.

[0071] Optionally, the data access method provided in the embodiment of the present application further includes:

[0072] Step S408, when the access frequency of data in the first category of data decreases, by synthesizing multiple data files into one data file, identifying the data in the synthesized data file according to the timestamp, storing the data belonging to the first category of data in the first storage system table, and storing the data belonging to the second category of data in the second storage system table.

[0073] Specifically, as hot data becomes cold data due to reduced access frequency over time, the cold data is identified through the compaction process, that is, the data is identified based on the timestamp of the data in the first storage system table, and the identified cold data is moved from the first storage system table to the second storage system table.

[0074] Further, optionally, the data access method provided in the embodiment of the present application also includes:

[0075] Step S409: When the process of combining the multiple data files into one data file is not completed, the data is filtered according to the timestamp of the data in the first storage system table, and the obtained second type of data is migrated to the second storage system table.

[0076] Before compaction is completed, some cold data will appear in the first storage system table. Therefore, the client handles this situation. When a user queries the first storage system table, the cold data is automatically filtered out. The data is filtered according to the timestamp of the data in the first storage system table. When a user queries the second storage system table, the cold data in the first storage system table is also queried out and merged with the cold data in the second storage system table and returned to the user.

[0077] Optionally, the first storage system table and the second storage system table are located in different column families of the same table.

[0078] Specifically, in the embodiment of the present application, cold data and hot data may be stored in different column families in the same table for implementation.

[0079] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that the data access method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0081] Example 3

[0082] According to another aspect of an embodiment of the present invention, a data access device is provided. Figure 5 Schematic diagram of a data access device according to Embodiment 3 of the present invention. Figure 5 As shown, the data access device provided in the embodiment of the present application includes:

[0083] A storage module 52 is used to store first-category data in a first storage system table and second-category data in a second storage system table, wherein the first-category data is accessed more frequently than the second-category data; a receiving module 54 is used to receive a data request message, wherein the data request message is used to indicate the storage system table where the data to be processed is located, wherein the storage system table includes the first storage system table and the second storage system table; a response module 56 is used to return the first-category data or the second-category data according to the data request message.

[0084] Optionally, the data access device provided in the embodiment of the present application is applied to a distributed storage database, the first storage system table is located in a first type of data storage system, and the second storage system table is located in a second type of data storage system.

[0085] Optionally, the receiving module 54 includes: a first parsing unit, used to parse the data type of the data to be processed in the data request message, wherein the data type includes: first category data or second category data; a second parsing unit, used to determine the storage system table where the data to be processed is located according to the data type; and an access unit, used to access the first category data storage system or the second category data storage system through a preset calling system.

[0086] Further, optionally, the response module 56 includes: a first response unit, used to query the first storage system table to obtain the data to be processed when the data type of the data to be processed is the first category of data; and a second response unit, used to query the second storage system table to obtain the data to be processed when the data type of the data to be processed is the second category of data.

[0087] Optionally, the data access device provided in the embodiment of the present application also includes: a first data migration module, which is used to, when the data access frequency in the first category of data decreases, by synthesizing multiple data files into one data file, identifying the data according to the timestamp of the data in the synthesized data file, and storing the data belonging to the first category of data in the first storage system table and storing the data belonging to the second category of data in the second storage system table.

[0088] Further, optionally, the data access device provided in the embodiment of the present application also includes: a second data migration module, which is used to filter the data according to the timestamp of the data in the first storage system table when the process of synthesizing multiple data files into one data file is not completed, and migrate the obtained second type of data to the second storage system table.

[0089] Example 4

[0090] According to another aspect of an embodiment of the present invention, a storage medium is provided. The storage medium includes a stored program, and is characterized in that when the program is running, the device where the storage medium is located is controlled to execute the data access method in Example 2.

[0091] Example 5

[0092] According to another aspect of an embodiment of the present invention, a terminal is provided, including a storage medium and a processor, wherein the processor is used to run a program stored in the storage medium, and is characterized in that the method for accessing data in Example 2 is executed when the program is running.

[0093] Specifically, in the embodiment of the present application, the terminal can be an all-in-one machine that can implement the data access method in Example 2, or it can be a server cluster composed of multiple servers, or one or at least one combination of cloud servers applied to the cloud.

[0094] Example 6

[0095] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data access method provided in the first embodiment.

[0096] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0097] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: storing first category data in a first storage system table and second category data in a second storage system table, wherein the first category data is accessed more frequently than the second category data; receiving a data request message, wherein the data request message is used to indicate the storage system table where the data to be processed is located, wherein the storage system table includes the first storage system table and the second storage system table; and returning the first category data or the second category data according to the data request message.

[0098] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: the data access method is applied to a distributed storage database, the first storage system table is located in a first type of data storage system, and the second storage system table is located in a second type of data storage system.

[0099] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: receiving a data request message includes: parsing the data type of the data to be processed in the data request message, wherein the data type includes: first category data or second category data; determining the storage system table where the data to be processed is located according to the data type; accessing the first category data storage system or the second category data storage system through a preset calling system.

[0100] Further, optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: returning the first category of data or the second category of data according to the data request message includes: when the data type of the data to be processed is the first category of data, querying the first storage system table to obtain the data to be processed; when the data type of the data to be processed is the second category of data, querying the second storage system table to obtain the data to be processed.

[0101] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: when the access frequency of data in the first category of data decreases, by synthesizing multiple data files into one data file, identifying the data in the synthesized data file according to the timestamp, storing the data belonging to the first category of data in a first storage system table, and storing the data belonging to the second category of data in a second storage system table.

[0102] Further, optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: when the process of synthesizing multiple data files into one data file is not completed, filtering the data according to the timestamp of the data in the first storage system table, and migrating the obtained second category data to the second storage system table.

[0103] Optionally, the first storage system table and the second storage system table are located in different column families of the same table.

[0104] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0105] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0110] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method of data access, include: storing first-category data in a first storage system table and second-category data in a second storage system table, wherein the first-category data is accessed more frequently than the second-category data; receiving a data request message, wherein the data request message is used to indicate a storage system table where the data to be processed is located, wherein the storage system table includes the first storage system table and the second storage system table; Returning the first category of data or the second category of data according to the data request message; The method further comprises: when the access frequency of data in the first category of data decreases, by combining multiple data files into one data file, identifying according to the timestamp of the data in the combined data file, storing the data belonging to the first category of data into the first storage system table, and storing the data belonging to the second category of data into the second storage system table; The first storage system table and the second storage system table are located in different column families in the same table; the first type of data storage system is HDFS, and the second type of data storage system is OSS system; Before compaction is completed, cold data will be automatically filtered out when users query the first storage system table, where the data is filtered according to the timestamp of the data in the first storage system table; and when users query the second storage system table, the cold data in the first storage system table will also be queried out and merged with the cold data in the second storage system table and returned to the user.

2. The method according to claim 1, in, The data access method is applied to a distributed storage database, wherein the first storage system table is located in a first type of data storage system, and the second storage system table is located in a second type of data storage system.

3. The method according to claim 1, in, Receiving a data request message includes: Parsing the data type of the to-be-processed data in the data request message, wherein the data type includes: the first category of data or the second category of data; Determine the storage system table where the data to be processed is located according to the data type; The first type of data storage system or the second type of data storage system is accessed through a preset calling system.

4. The method according to claim 3, in, Returning the first category of data or the second category of data according to the data request message includes: In a case where the data type of the data to be processed is the first type of data, querying the first storage system table to obtain the data to be processed; When the data type of the data to be processed is the second type of data, the second storage system table is queried to obtain the data to be processed.

5. The method according to claim 1, in, The method further comprises: When the process of combining the multiple data files into one data file is not completed, the data is filtered according to the timestamp of the data in the first storage system table, and the obtained second-category data is migrated to the second storage system table.

6. A device for accessing data, include: A storage module, used to store first-category data in a first storage system table and second-category data in a second storage system table, wherein the first-category data has a greater access frequency than the second-category data; A receiving module, used for receiving a data request message, wherein the data request message is used for indicating a storage system table where the data to be processed is located, wherein the storage system table includes the first storage system table and the second storage system table; A response module, configured to return the first category of data or the second category of data according to the data request message; The device further comprises: a first data migration module, configured to store data belonging to the first category of data in the first storage system table and data belonging to the second category of data in the second storage system table by combining multiple data files into one data file and identifying the data according to the timestamp of the data in the combined data file when the access frequency of the data in the first category of data decreases; The first storage system table and the second storage system table are located in different column families in the same table; the first type of data storage system is HDFS, and the second type of data storage system is OSS system; Before compaction is completed, cold data will be automatically filtered out when users query the first storage system table, where the data is filtered according to the timestamp of the data in the first storage system table; and when users query the second storage system table, the cold data in the first storage system table will also be queried out and merged with the cold data in the second storage system table and returned to the user.

7. The device according to claim 6, in, The data access device is applied to a distributed storage database, the first storage system table is located in a first type of data storage system, and the second storage system table is located in a second type of data storage system.

8. The device according to claim 6, in, The receiving module comprises: A first parsing unit, configured to parse the data type of the to-be-processed data in the data request message, wherein the data type includes: the first category of data or the second category of data; A second parsing unit, configured to determine the storage system table where the to-be-processed data is located according to the data type; An access unit is used to access the first type of data storage system or the second type of data storage system through a preset calling system.

9. The device according to claim 8, in, The response module comprises: A first response unit, configured to query the first storage system table to obtain the data to be processed when the data type of the data to be processed is the first type of data; The second response unit is configured to query the second storage system table to obtain the data to be processed when the data type of the data to be processed is second type data.

10. The device according to claim 6, in, The device also includes: The second data migration module is used to filter the data according to the timestamp of the data in the first storage system table and migrate the obtained second-category data to the second storage system table when the process of combining multiple data files into one data file is not completed.

11. A system for accessing data, include: Client and storage system, where The client is used to send a data request message to the storage system, wherein the data request message is used to indicate a storage system table where the data to be queried is located, wherein the storage system table includes a first storage system table and a second storage system table; The storage system includes a first-type data storage system and a second-type data storage system, wherein the first-type data storage system and the second-type data storage system are respectively connected to a preset calling system, receive the data request message through the preset calling system, and obtain the first-type data from a table stored in the first storage system, or obtain the second-type data from a table stored in the second storage system, by accessing the first-type data storage system or the second-type data storage system; Wherein, when the access frequency of data in the first category of data decreases, by combining multiple data files into one data file, identifying according to the timestamp of the data in the combined data file, storing the data belonging to the first category of data in the first storage system table, and storing the data belonging to the second category of data in the second storage system table; The first storage system table and the second storage system table are located in different column families in the same table; the first type of data storage system is HDFS, and the second type of data storage system is OSS system; Before compaction is completed, cold data will be automatically filtered out when users query the first storage system table, where the data is filtered according to the timestamp of the data in the first storage system table; and when users query the second storage system table, the cold data in the first storage system table will also be queried out and merged with the cold data in the second storage system table and returned to the user.

12. A storage medium comprising a stored program, It is characterized in that When the program is running, the device where the storage medium is located is controlled to execute the data access method described in any one of claims 1 to 5.

13. A terminal comprising a storage medium and a processor, wherein the processor is configured to run a program stored in the storage medium. It is characterized in that When the program is running, the data access method described in any one of claims 1 to 5 is executed.

Images