Data storage method, system and program product

Abstract

The invention discloses a data storage method and system and a program product. The method comprises the following steps that: a big data platform splits to-be-stored data into a plurality of pieces of slice data, stores each piece of slice data into a temporary directory, and allocates at least one target storage node in column type storage nodes for the slice data; receiving a target write-in command, and setting the target storage node to be in a restricted access state in response to the target write-in command; and the big data platform obtains the slice data from the temporary directory, copies the slice data to the access directory of the corresponding target storage node, and maps the slice data under the access directory of the target storage node to the temporary partition corresponding to the target storage node. The big data platform is used for preprocessing to-be-stored data so as to reduce resource consumption of column type storage nodes in the data storage process; and the data storage time is shortened through the unified mapping and writing process of the multiple pieces of slice data, and the data storage efficiency is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application generally relates to the field of data processing, and in particular, to a data storage method, system and program product. BACKGROUND

[0002] In the prior art, enterprises usually effectively store and manage the device information of various kitchen appliances and the data generated in the production and operation process of the kitchen appliances, so as to use the data as an important basis for analyzing market trends and optimizing product services. For example, the storage of the above data can be realized by establishing a relational database. In the storage process, the data warehouse is a key link at the beginning of the data life cycle, and the efficiency and quality of the data warehouse directly affect the effect of subsequent data processing and data analysis.

[0003] With the development of big data, most enterprises currently use columnar storage databases to replace traditional relational databases to store the above data, in order to meet the gradually increasing requirements for the timeliness and response speed of data analysis. However, with the continuous growth of data volume and the increase of business complexity, columnar storage databases still have problems affecting the data warehouse efficiency and the overall performance of the system in actual storage process. SUMMARY

[0004] In view of the above defects or deficiencies in the prior art, it is desirable to provide a data storage method, system and program product. The method uses a big data platform to preprocess the data to be stored, so as to reduce the resource consumption of the columnar storage node in the data storage process. The unified mapping and writing process of the multiple slice data shortens the data warehouse time and greatly improves the data warehouse efficiency.

[0005] In a first aspect, the present application provides a data storage method. The method is applied to a data storage system, and the data storage system includes a big data platform and a columnar storage node. The method includes the following steps: The big data platform divides the data to be stored into multiple slice data, and stores each slice data in a temporary directory. According to the access frequency of the slice data, at least one target storage node in the columnar storage node is allocated to the slice data; A target write command is received, and the target storage node is set to a limited access state in response to the target write command. In the limited access state, only the target write command is allowed to access the target storage node; The big data platform obtains the slice data from the temporary directory, copies the slice data to the access directory of the corresponding target storage node, and maps the slice data in the access directory of the target storage node to the temporary partition corresponding to the target storage node.

[0006] In one possible implementation, the data to be stored is divided into multiple slice data, including: According to the data format requirement of the big data platform, the original data is preprocessed to obtain to-be-stored data; A state marker of the to-be-stored data is obtained, and when the state marker indicates that the to-be-stored data meets the data format requirement, the to-be-stored data is divided into a plurality of slice data.

[0007] In a possible implementation, the slice data under the access directory of the target storage node is mapped into the temporary partition corresponding to the target storage node, including: The number of temporary partitions of the target storage node is verified to be consistent with the number of slice data under the access directory of the target storage node, and if consistent, the slice data is one-to-one mapped to the corresponding temporary partition.

[0008] In a possible implementation, a target write command is received, and the target storage node is set to a limited access state in response to the target write command, including: An attach command is received, and metadata of the target storage node is locked in response to the attach command by executing a detach table command; The metadata lock of the target storage node is acquired in response to the attach command to limit the access of the target storage node to only allow the attach command In a possible implementation, the slice data under the access directory of the target storage node is mapped into the temporary partition corresponding to the target storage node, including: The data source of the first temporary partition is defined as the first slice data in response to the attach command to one-to-one map the first slice data to the corresponding first temporary partition; the first slice data is any slice data under the access directory of the target storage node, and the first temporary partition is any temporary partition of the target storage node.

[0009] In a possible implementation, the method further includes: acquiring a data write amount of each target storage node, and obtaining a total data amount based on the data write amount of each target storage node; All total data amounts are verified to be consistent with the number of to-be-stored data, and if not consistent, a prompt information is generated, and the prompt information is used to indicate a data storage failure.

[0010] In a possible implementation, after the slice data is allocated at least one target storage node in the array storage node according to the access frequency of the slice data, the method further includes: The access state of the target storage node is acquired, and when the access state is an allowed access state, copying the slice data under the temporary directory to the access directory of the target storage node is suspended until the target storage node changes to a limited access state.

[0011] In a second aspect, a data storage system is provided, which includes a big data platform and columnar storage nodes. The big data platform is configured to split to-be-stored data into multiple slice data, store each slice data in a temporary directory, and assign at least one target storage node in the columnar storage nodes to each slice data according to the access frequency of the slice data. The big data platform is further configured to receive a target write command, and set the target storage node to a restricted access state in response to the target write command, wherein the target storage node is only allowed to be accessed by the target write command in the restricted access state. The big data platform is further configured to obtain the slice data from the temporary directory, copy the slice data to an access directory of the corresponding target storage node, and map the slice data in the access directory of the target storage node to a temporary partition of the target storage node, so as to complete data writing.

[0012] In a possible implementation, the target storage node is configured to merge the slice data in all temporary partitions corresponding to the same access directory, and write the merged slice data to a continuous storage space of the target storage node.

[0013] In a third aspect, a computer program product is provided, which includes instructions, and when the instructions are executed, the method of any one of the first aspect is performed.

[0014] In a fourth aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the method of any one of the first aspect is implemented.

[0015] In a fifth aspect, a computer readable storage medium is provided, which stores a computer program, and when the program is executed by a processor, the method of any one of the first aspect is implemented.

[0016] Compared with the prior art of reading and writing the source data in the source database into the target storage database piece by piece, the data storage method, system and program product provided by the embodiment of the application have the following advantages: on the one hand, the data to be stored is preprocessed by the big data platform, and the data to be stored is divided into multiple slice data, so as to reduce the resource consumption of the columnar storage node in the data processing during the data storage process; on the other hand, based on the setting of the temporary directory, the multiple slice data are temporarily stored in the temporary directory, so as to avoid the data disorder when the slice data is directly transmitted from the big data platform to the columnar storage node, thereby enhancing the stability of the data; on the other hand, in the copy transmission stage of the slice data, the write state of the target storage node is modified, so as to avoid that other interference slice data which should not be stored in the target storage node is stored in the target storage node, thereby guaranteeing the consistency of the data; on the other hand, the slice data is stored in the target storage node in a unified mapping write manner, so as to shorten the data storage time and greatly improve the data storage efficiency.

[0017] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0018] Other features, objects, and advantages of the application will become more apparent from the following detailed description when read in conjunction with the accompanying drawings: Figure 1 is a schematic diagram of the architecture of the data storage system 10 provided by the embodiment of the application; Figure 2 is a schematic diagram of the architecture of the data storage system 10 provided by the embodiment of the application; Figure 3 is a schematic diagram of the architecture of the data storage system 10 provided by the embodiment of the application; Figure 4 is a schematic diagram of the architecture of the data storage system 10 provided by the embodiment of the application; DETAILED DESCRIPTION

[0019] The application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings.

[0020] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments. In addition, the term "and / or" in this paper is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B together, and the existence of B alone. The terms "first" and "second" in the specification and claims of the embodiments of the present application are used to distinguish different objects, not to describe the specific order of the objects.

[0021] The following explains the terms related to the present application: (1) Columnar database (or Column-Oriented Database): a database management system that organizes and stores data on disk by column rather than by row, mainly suitable for batch data processing and instant query; Specifically, the columnar database stores data values in a column together, and then stores the data of the next column; (2) On-line analytical processing (OLAP for short): a computer processing technology for analyzing and querying large-scale data sets, mainly through providing a multi-dimensional data model and multi-dimensional query function to facilitate users to analyze and query data from different angles and different levels.

[0022] With the development of big data, most enterprises currently use columnar databases to replace traditional relational databases to store the above data in order to meet the gradually increasing requirements for the timeliness and response speed of data analysis. Specifically, the columnar database can store data in the same column in a centralized manner to reduce unnecessary data scanning when querying data, thereby improving data query performance. Based on this, the columnar storage method is widely used in the fields of big data storage and OLAP analysis.

[0023] However, with the continuous growth of data volume and the increase of business complexity, the columnar database still has many problems affecting the storage efficiency and overall system performance in actual storage process: Firstly, the columnar database has a higher requirement for hardware configuration, especially sensitive to the level of disk performance. Compared with the influence of CPU and memory with different computing power on data write performance in the data storage process, there is a significant difference in data write performance when using mechanical hard disk and solid state disk to store data; Secondly, the prior art generally improves query performance by reducing write performance, so in a high-concurrency write scenario, columnar storage is prone to increase the I / O burden in the storage process, thereby directly affecting query and server performance. Thirdly, a large amount of data needs to be loaded into the memory for operation in the data processing process, so there is a large demand for memory when writing data and executing data queries, especially when executing association queries, a large amount of memory space is needed to cache and process data, which is prone to cause the system to frequently perform disk swapping due to insufficient memory, thereby further reducing system performance, and for servers with low memory configuration, the memory demand of columnar storage when writing and querying may affect the normal execution of other tasks. Fourthly, when writing new data or updating and deleting existing data, a merge operation needs to be performed on the stored data to ensure data consistency and storage efficiency, however, the existing merge algorithm has poor performance, and it may take several hours or even longer to perform file merging after data is imported, and a large amount of I / O, CPU and memory resources are occupied during file merging, affecting overall performance.

[0024] Based on this, the embodiments of the present application provide a data storage method, system and program product, which can utilize a big data platform to pre-process the data to be stored, so as to reduce the resource consumption of columnar storage nodes in the data storage process, and shorten the data import time through unified mapping and writing of multiple slice data, greatly improving the data import efficiency.

[0025] In one possible implementation manner, the data storage method provided by the present application is applied to a data storage system 10, Figure 1 is a schematic diagram of the architecture of the data storage system 10 provided by the embodiments of the present application, as Figure 1 shown, the data storage system 10 includes a big data platform 11 and a columnar storage node 12.

[0026] For example, the big data platform 11 is a resource management and job scheduling platform (Yet Another Resource Negotiator, Yarn for short) of a distributed cluster based on a Hadoop big data framework; the columnar storage node 12 corresponds to the columnar storage database described above.

[0027] Specifically, the big data platform 11 is a big data platform Yarn, and the internal core components thereof run on different servers (nodes) to realize decoupling of resource scheduling and task execution.

[0028] For example, a cluster resource supervisor component (ResourceManager, RM for short) in a big data platform Yarn runs on a master node (Master) to accept job requests from clients and is responsible for the management and scheduling of the allocation of the entire cluster resource; a single-node management component (NodeManager, NM for short) runs on each worker node to manage and control the resources on the corresponding node and execute the tasks allocated by the cluster resource supervisor component; and an application management component (ApplicationMaster, AM for short) is independently started by each application to be responsible for applying for resources from the cluster resource supervisor component RM, communicating with the single-node management component NM to start and stop tasks, and monitoring the execution status of the tasks. In addition, the big data platform Yarn also includes a resource abstraction component (Container, which can be understood as a container), which is specifically started by the big data platform Yarn on the single-node management component NM of each node to run actual computing tasks.

[0029] Based on this, in the above data storage system 10, when a storage task of to-be-stored data is submitted to the big data platform 11, the cluster resource supervisor component RM receives and selects a single-node management component NM on an idle node to start a resource abstraction component Container by the single-node management component NM to run the application management component AM.

[0030] Secondly, the columnar storage node 12 runs on a server to complete the storage and query tasks of data by relying on the computing resources, storage resources, network resources and the like of the underlying server. It should be noted that the servers running the core components of the big data platform 11 and the columnar storage node 12 can be implemented by independent servers or server clusters composed of multiple servers.

[0031] In a specific implementation, according to the task execution process of the above big data platform 11, the big data platform 11 first pre-processes the to-be-stored data, and divides the pre-processed to-be-stored data into multiple slice data to store the multiple slice data in the temporary directory 13; then determines the target storage node 121 of the to-be-stored data in the columnar storage node 12 according to the attributes of the to-be-stored data, and when the target storage node 121 is in a write-prohibited state, copies the slice data corresponding to the to-be-stored data in the temporary target 13 to the target storage node 121, and maps each slice data to each temporary partition in the target storage node 121 one by one.

[0032] Figure 2 is a flowchart of a data storage method provided by an embodiment of the present application, as shown in Figure 2 The method comprises the following steps: In step S201, the big data platform 11 divides the to-be-stored data into multiple slice data, and stores each slice data into the temporary directory 13, and according to the access frequency of the slice data, allocates at least one target storage node 121 in the array storage node 12 for the slice data.

[0033] Compared with the prior art of storing the complete form of to-be-stored data in the source database into the target storage database, the embodiment of the present application divides the to-be-stored data into multiple slice data, so as to greatly shorten the data storage time by parallel copying of multiple slice data.

[0034] In a possible implementation, the big data platform 11 first pre-processes the original data, so that the data format of the to-be-stored data formed by the pre-processing can meet the data transmission requirements of the big data platform 11, and then divides the to-be-stored data into multiple slice data after the to-be-stored data meets the data format requirements.

[0035] For example, according to the expected data size of each slice data, the to-be-stored data is evenly divided based on the data size (i.e. the physical size when actually stored) of the pre-processed to-be-stored data, which can be in units of bytes.

[0036] In an example, the big data platform 11 can realize the division of the to-be-stored data through a Linux command. Specifically, the Linux command is represented as `hdfs dfs -split [-b <blocksize> ] <hdfsfile>wherein -b <blocksize>to specify a desired data size of slice data, <hdfsfile>a file name of the to-be-stored data to be split.

[0037] Taking the expected data size of the sliced data as an example, which is the default value of 512 MB, if the to-be-stored data after preprocessing has a data size of 10 GB and a file name of order_info.txt, and the to-be-stored data is to be split into 20 data block files, the Linux command corresponds to `hdfs dfs -split -b $(python -c 'print(512 * 1024 * 1024)') order_info.txt`. It should be noted that the expected data size of the sliced data can be configured according to the actual data size of the to-be-stored data, and the present application does not make a specific limitation thereto.

[0038] In an example, after the above data splitting is completed, the big data platform 11 can also store the plurality of sliced data to a temporary directory through a Linux command. Specifically, the directory name of the temporary directory can be added to the Linux command used for splitting the to-be-stored data, so as to automatically store the sliced data after the data splitting is completed to the temporary directory.

[0039] Taking the to-be-stored data with the above file name of order_info.txt as an example, which is split into 20 data block files with a data size of 512 MB, if the name of the temporary directory is tmp, the Linux command corresponds to `hdfs dfs -split -b $(python -c 'print(512 * 1024 * 1024)') order_info.txt tmp / `.

[0040] In the embodiment of the present application, after the plurality of sliced data is successfully stored to the temporary directory, the target storage node 121 of each sliced data is dynamically allocated, so that the storage space of the columnar storage node 12 can be maximally and reasonably applied.

[0041] In a possible implementation manner, at least one target storage node 121 in the columnar storage node 12 can be allocated to each sliced data according to the access frequency of each sliced data.

[0042] In an example, when the access frequency of the sliced data is higher than a first preset threshold, the sliced data can be referred to as hot data, and based thereon, a high-performance storage node in the columnar storage node 12 can be allocated to the hot data, so as to provide a fast read-write function by the optimized storage format in the high-performance storage node, thereby meeting the high-frequency access requirement of the hot data; wherein the high-performance storage node is, for example, a high-speed solid state disk (SSD) storage node.

[0043] In another example, when the access frequency of the slice data is lower than a second preset threshold, the slice data can be referred to as cold data, based on which, a low-performance storage node in the array storage node 12 can be allocated for the cold data, so as to compress and store the slice data in the low-performance storage node, thereby saving storage space; wherein, the low-performance storage node is, for example, a mechanical hard disk storage node.

[0044] It should be noted that the performance of the above-mentioned storage node "high" and "low" can be determined based on the speed of data response of each storage node, the cost, and the storage format. Correspondingly, the faster the data response speed, the higher the cost, and the more optimized the storage format, the higher the performance of the storage node; and the slower the data response speed, the lower the cost, and the more general the storage format, the lower the performance of the storage node.

[0045] In addition, the present application can also allocate the target storage node 121 for the slice data in combination with the space usage, read-write performance, and the like of each target storage node 121.

[0046] Step S202, receiving a target write command, and setting the target storage node 121 to a restricted access state in response to the target write command, wherein the restricted access state only allows the target write command to access the target storage node 121.

[0047] In the embodiment of the present application, the access state of the target storage node 121 is limited by responding to the target write command, so as to avoid the situation that data is disordered due to other write processes in the process of writing multiple slice data to the target storage node 121.

[0048] In a possible implementation, in response to the target write command, the data table of the target storage node 121 is unloaded from the memory, so that the data table is in a separated state, so as to realize the restricted access state of the target storage node 121. It should be noted that although the data table of the target storage node 121 is unloaded from the memory, the data file is retained.

[0049] Specifically, the restricted access state of the target storage node 121 can be understood as a locking operation on the data table in the target storage node 121, wherein the data table of the target storage node 121 refers to a specific table in the target storage node 121 for receiving data.

[0050] In an example, the big data platform 11 can set the target storage node 121 to the restricted access state by running a synchronization script, and correspondingly, the synchronization script contains a locking statement and a locking position (i.e., the node modified to the restricted access state is a certain target storage node 121).

[0051] Based on this, the target storage node 121 whether is set to the access limiting state can be determined by acquiring the current lock information or the lock table, if the target storage node 121 corresponding to the slice data is in the access allowing state, the further operation on the slice data is suspended (i.e., the slice data under the temporary directory is copied to the access directory of the target storage node 121), otherwise the next operation is performed. For example, the current lock information can be acquired by checking the table / row in the access limiting state, checking the transaction waiting for the lock, and checking the transaction level lock information.

[0052] For example, when the target write command is the attach command, the metadata of the target storage node 121 can be locked by executing the detachtable command in response to the attach command, so as to modify the access state of the target storage node 121.

[0053] Correspondingly, when locked by the detach table command, the metadata lock of the target storage node 121 can be acquired by executing the attach command, so as to enable normal data writing to the target storage node 121. It should be noted that in the case of locking by the detachtable command, other businesses cannot perform add, delete, modify, and query operations on the data table, thereby avoiding data disorder.

[0054] In the embodiment of the present application, based on the access limiting state that only allows the target write command to access the target storage node 121, the data disorder caused by multiple slice data corresponding to different to-be-stored data when writing to different data tables of the target storage node 121 is avoided, and the integrity and consistency of the synchronization data are ensured.

[0055] In step S203, the big data platform 11 acquires the slice data from the temporary directory 13, copies the slice data to the access directory of the corresponding target storage node 121, and maps the slice data in the access directory of the target storage node 121 to the temporary partition of the target storage node 121.

[0056] In the embodiment of the present application, based on the columnar storage format of the target storage node 121, the multiple slice data can be copied to the access directory of the target storage node 121 in parallel, so that the target storage node 121 can quickly load the partition data under the access directory (i.e., each slice data under the access directory is directly mapped to each temporary partition).

[0057] In an example, the command line tool (SecureCopy Protocol, referred to as SCP mode) based on the SSH protocol for encrypted file transfer can be used to copy and transmit the multiple slice data to the access directory of the corresponding target storage node 121.

[0058] In a possible implementation, after the multiple slice data is successfully copied to the access directory of the target storage node 121, the consistency of the number of slice data under the access directory and the number of temporary partitions in the target storage node 121 is verified first. If the numbers are consistent, the slice data is mapped to the corresponding temporary partition one by one.

[0059] In an example, the slice data under the access directory can be mapped to the temporary partition of the target storage node 121 by using an Attach command.

[0060] Specifically, the Attach command can be executed to define the data source of the first temporary partition as the first slice data, so as to map the first slice data to the corresponding first temporary partition one by one. The first slice data is any slice data under the access directory of the target storage node 121, and the first temporary partition is any temporary partition of the target storage node 121.

[0061] Taking the target storage node 121 as clickhouse and the slice data under the access directory as the order_info table composed of 20 slice data formed by the step S201, the mapping of the slice data to the corresponding temporary partition can be to execute the Attach command to define the data source of the temporary partition 1 as the slice data 1, the data source of the temporary partition 2 as the slice data 2, and so on, and the data source of the temporary partition 20 as the slice data 20. The Attach command is as follows: ATTACH TABLE order_info PARTITION (part_1) FROM ' / data / order_info / order_info_1.txt'; ATTACH TABLE order_info PARTITION (part_2) FROM ' / data / order_info / order_info_2.txt'; ... ATTACH TABLE order_info PARTITION (part_20) FROM ' / data / order_info / order_info_20.txt'; Correspondingly, after the execution of the above Attach command is completed, it can be indicated that each slice data has been successfully mapped to the temporary partition.

[0062] In a possible implementation, for each target storage node 121, when the slice data is successfully mapped to the temporary partition of the target storage node 121, the target storage node 121 can be modified to be in an allowed access state, so as to restore the operations such as adding, deleting, modifying, and querying of the data table in the target storage node 121 by other service nodes.

[0063] Specifically, modifying the target storage node 121 to be in the allowed access state can be understood as unlocking the data table in the target storage node 121.

[0064] Compared with the existing serial data writing mode of writing data into a data table, the successful mapping of the slice data to the temporary partition in the embodiment of the application completes the data writing, thereby improving the data writing efficiency and writing performance and reducing resource consumption.

[0065] In an embodiment of the application, when the to-be-stored data is divided into a plurality of slice data, the original data needs to be preprocessed into a target data format required by the big data platform 11. For example, the foregoing "dividing the to-be-stored data into a plurality of slice data" includes: preprocessing the original data according to the data format requirement of the big data platform 11 to obtain the to-be-stored data; obtaining a state mark of the to-be-stored data, and when the state mark indicates that the to-be-stored data meets the data format requirement, dividing the to-be-stored data into a plurality of slice data.

[0066] In an example, when the big data platform 11 is a big data platform Yarn established based on a Hadoop big data framework, the Yarn resource can be used to perform data cleaning and pre-computation processing on the original data, so as to generate a distributed file (Hadoop Distributed File System, HDFS data file) suitable for running on general hardware, that is, the big data platform 11 preprocesses the original data into HDFS format to-be-stored data.

[0067] Correspondingly, the big data platform 11 obtains the state mark in the to-be-stored data through a synchronization script, and if the state mark indicates that the to-be-stored data is HDFS format data, it indicates that the big data platform 11 successfully processes the original data into the HDFS data file; if the state mark fails to indicate that the to-be-stored data is HDFS format data, it indicates that the original data fails to be successfully processed into the HDFS data file, at this time, a polling waiting stage is entered (that is, the state mark of the to-be-stored data is obtained again after a preset polling time), until the data format of the to-be-stored data is the HDFS format; it should be noted that the preset polling time is not limited in the application.

[0068] For example, the status mark in the to-be-stored data can be represented as SUCCESS, and when the status mark is SUCCESS, it indicates that the to-be-stored data is HDFS format data, and when the status mark is not SUCCESS, it enters a polling waiting stage.

[0069] For example, when the data format of the to-be-stored data is HDFS format, the to-be-stored data is divided into multiple slice data.

[0070] In the embodiment of the present application, the data format of the to-be-stored data is processed by the Yarn resource to meet the requirements of the big data platform 11, which avoids the situation that the to-be-stored data cannot be adapted to the big data platform 11, thereby ensuring the stability of data transmission.

[0071] In an embodiment of the present application, before copying the multiple slice data to the access directory of the corresponding target storage node 121, the service state of each target storage node 121 in the columnar storage node 12 can also be obtained to determine whether the service state of each target storage node 121 is normal.

[0072] For example, if the service state of each target storage node 121 in the columnar storage node 12 is normal, the subsequent copy operation of the slice data is performed; if individual or partial target storage nodes 121 in the columnar storage node 12 are offline, it enters a polling waiting stage (i.e., the service state of each target storage node 121 is obtained again after a predetermined polling time), until each target storage node 121 is in a normal service state.

[0073] In the embodiment of the present application, the subsequent operation is performed when each target storage node 121 is in a normal service state, so that the remaining idle target storage nodes 121 can provide corresponding services when any target storage node 121 fails, thereby ensuring the service function of the columnar storage node 12.

[0074] In an embodiment of the present application, when the slice data is mapped to the temporary partition of the target storage node 121, in addition to verifying whether the number of temporary partitions is consistent with the number of slice data, it can also verify whether the table structure of the data table is consistent with the data format of the slice data.

[0075] In an example, it can be verified whether the field definition, data type, primary key constraint and other meta information of the data table are completely matched with the data format of the slice data, and if matched, a one-to-one mapping relationship between the slice data and the temporary partition is established, otherwise, the reason for the matching failure is determined based on the matching result of the data table meta information and the data format of the slice data, wherein the reason for the matching failure can include an error in copying and transmitting the to-be-stored data or an error in the slice operation of the to-be-stored data.

[0076] In an embodiment of the present application, each target storage node 121 can also merge the slice data in the temporary partition.

[0077] For example, the target storage node 121 is configured to merge the slice data in all temporary partitions corresponding to the same access directory, and write the merged slice data into the continuous storage space of the target storage node 121.

[0078] Specifically, the target storage node 121 obtains the temporary partitions to be merged by querying the data table of the target storage node 121, and merges the slice data in the temporary partitions to be merged by Merge.

[0079] In an embodiment of the present application, the total data write amount of the target storage node 121 in the columnar storage node 12 can also be obtained to verify whether the total data write amount is consistent with the number of the data to be stored.

[0080] In a possible implementation, the data write amount of each target storage node 121 is obtained, the total data amount is obtained based on the data write amount of each target storage node 121, and it is verified whether all total data amounts are consistent with the number of the data to be stored.

[0081] For example, for each target storage node 121, the data write amount of the target storage node 121 can be determined according to the number of records of each data table in the target storage node 121 after the slice data is written to the temporary partition corresponding to the target storage node 121 in parallel, and then the data write amounts of the target storage nodes 121 are summed to obtain the total data amount.

[0082] Correspondingly, the number of the data to be stored can correspond to the number of the split data of the big data platform 11 (i.e., the total number of the slice data).

[0083] For example, if the total data amount is consistent with the number of the data to be stored, it indicates that the data to be stored is successfully stored in the columnar storage node 12; if the total data amount is not consistent with the number of the data to be stored, a prompt information is generated, which is used to indicate that the data storage fails.

[0084] In the embodiment of the present application, by comparing the total data amount after the data is stored in the database with the number of the data to be stored, potential data problems can be found, and when the number comparison is inconsistent, an alarm is given to notify the data storage failure in time, so as to avoid the data quality problem from disturbing the subsequent data analysis work and ensure the strong consistency of the data.

[0085] In an embodiment, another data storage method is also provided. For example, the application scenario of the data storage method is shown in Figure 1 The big data platform 11 is a big data platform Yarn, Figure 3 is another flowchart of the data storage method provided by the embodiments of the present application, as shown in the figure, the method comprises the following steps: Figure 3 Step S301, the synchronization script checks all mandatory parameter conditions.

[0086] Exemplarily, the mandatory parameter conditions can include the source database of the data to be stored, the storage time, the target storage database, and other storage parameters, etc.

[0087] Step S302, the state marker of the data to be stored is obtained, and it is checked based on the state marker whether the HDFS data file is ready.

[0088] Exemplarily, on the big data platform established by the Hadoop big data framework, the Yarn resource can be used to perform data cleaning and pre-computation processing on the original data to generate the HDFS data file.

[0089] Exemplarily, the synchronization script can check whether the state marker on the HDFS data file is SUCCESS, if the state marker is SUCCESS, it means that the HDFS data file has been generated; if the state marker is not SUCCESS, polling is waited, and the polling duration can be configured autonomously.

[0090] Step S303, it is checked whether the service state of each target storage node 121 in the columnar storage node 12 is normal.

[0091] Exemplarily, if the service state of each target storage node 121 in the columnar storage node 12 is normal, step S304 is executed, otherwise, polling is waited, and the polling duration can be configured autonomously.

[0092] Step S304, the HDFS data file is split, and multiple slice data is output to the temporary directory 13.

[0093] Exemplarily, the total size of the original HDFS data file and the expected file block size after splitting are obtained, and the HDFS data file is split.

[0094] For example, the Linux command `hdfs dfs -split [-b <blocksize> ] <hdfsfile>` Average split of HDFS data files, where -b <blocksize>to specify the size of each partition in bytes, <hdfsfile>A file for specifying original HDFS data files that need to be split.

[0095] Step S305, the data table of the target storage node 121 is locked.

[0096] Exemplarily, the data table corresponding to each target storage node 121 is locked, and for any target storage node 121, if the lock is not obtained, polling is waited; if the lock is obtained, step S306 is continued.

[0097] Step S306, real-time monitoring of the space usage, read-write performance and access frequency of the slice data of each target storage node 121, and at least one target storage node 121 is allocated for the slice data.

[0098] Exemplarily, when new data is warehoused, the appropriate storage device and storage method can be intelligently selected according to the attributes of the data.

[0099] For example, for hot data with high access frequency, it can be allocated to a high-speed solid state disk (SSD) storage; for cold data with low access frequency, it can be allocated to a mechanical hard disk with lower cost.

[0100] Step S307, the slice data in the temporary directory 13 is copied to the access directory of the target storage node 121.

[0101] Exemplarily, the split HDFS data file under the tmp directory can be copied and transmitted to the access directory of the target storage node 121 by the scp method.

[0102] Step S308, the content of the slice data under the access directory is associated with the temporary partition of the target storage node 121.

[0103] Exemplarily, taking the target storage node 121 as a columnar database clickhouse as an example, the above association step specifically includes: First, obtain the table structure of the data table of the target storage node 121, and if the data table does not exist, create the data table; Second, verify whether the table structure of the data table of the target storage node 121 and the format of the slice HDFS data file are completely matched, if matched, execute the third step, otherwise it is easy to cause data inconsistency or error in the loading process; Third, traverse the slice HDFS data file under the access directory, and use the Attach command to associate each slice HDFS data file to the corresponding temporary partition.

[0104] Step S309, the data directory of the data table of the target storage node 121 is unlocked.

[0105] Step S310, merging the slice data in all temporary partitions corresponding to the same access directory.

[0106] Step S311, successfully storing the data table of the target storage node 121 and verifying the amount of written data.

[0107] In the data storage method provided by the present application, on the one hand, the calculation and processing of the original data are completed in advance in the big data platform Yarn, so as not to occupy the memory of the columnar storage node in the process of storing the data to be stored, and only a small part of CPU is used to decompress the data when downloading; on the other hand, compared with the traditional data storage method, the slice data can be quickly stored in the target storage node through the parallel writing process of multiple slice data under the condition of the same data amount, effectively relieving the pressure in the process of data transmission and processing, greatly improving the speed of data storage, for example, for TB level data, it only takes seconds to complete, providing timely data support for subsequent data analysis and application; on the other hand, when merging the slice data in the data table, the temporary partition of the target storage node 121 is dynamically divided to adjust the merging speed, for example, the fewer the number of temporary partitions, the faster the merging; on the other hand, through copying data copies under the access directory and data transmission confirmation mechanism and other ways, the reliability and stability of data processing are effectively improved, for example, in the process of data transmission, even if part of the data is lost or transmission error, the retransmission mechanism can ensure the integrity of the data.

[0108] In summary, the present application adopts the zero-pressure storage technology, based on the task scheduler and Hadoop big data framework, while reducing the time consumption and resource usage of each stage in the calculation, storage and merging stages, to improve the data storage performance; secondly, relying on the scheduling engine to manage and schedule various preprocessing tasks before data storage, to prioritize each data processing task, and according to the resource requirements (such as CPU, memory, etc.) of the task and the current system resource usage, reasonably arrange the task execution order and reasonably allocate resources.

[0109] The following refers to Figure 4 , Figure 4 shows a structural schematic diagram of a computer device suitable for implementing the embodiments of the present application, such as Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401 which can perform various appropriate actions and processes in accordance with programs stored in a read only memory (ROM) 402 or programs loaded from the storage section 409 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation instructions of the system are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0110] Connected to the I / O interface 405 are the following: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as necessary. A removable media 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 410 as necessary, so that a computer program read therefrom is installed into the storage section 408 as necessary.

[0111] In particular, the processes described above with reference to the flowcharts Figure 2 、 Figure 3 of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such an embodiment, the computer program contains program code for executing the methods illustrated by the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network by the communication section 409, and / or installed from the removable media 411. When the computer program is executed by the central processing unit (CPU) 401, the above-described functions defined in the system of the present application are performed.

[0112] It should be noted that the computer-readable medium shown in the application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In this application, the computer-readable signal medium can include a data signal carried in a baseband or as a carrier wave part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0113] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operation instructions of the systems, methods and computer program products according to various embodiments of the application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can also occur in different order from that noted in the drawings. For example, two connected blocks can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operation instructions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0114] The units or modules described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware. The described units or modules can also be arranged in a processor, for example, a processor can be described as including a semantic extraction unit, a weight distribution unit, and a determination unit. In some cases, the names of these units or modules do not constitute a limitation on the units or modules themselves.

[0115] As another aspect, the present application also provides a computer readable storage medium, which can be included in the computer device described in the above embodiments, or can exist separately and not be assembled into the computer device. The computer readable storage medium stores one or more programs, and when the programs are used by one or more processors to execute the methods described in the present application. For example, the steps of the methods shown in Figure 2 、 Figure 3 can be executed.

[0116] The embodiments of the present application provide a computer program product, which includes instructions that, when executed, cause the methods described in the embodiments of the present application to be performed. For example, the steps of the methods shown in Figure 2 、 Figure 3 can be executed.

[0117] The above description is only the preferred embodiments of the present application and the explanation of the applied technical principles. Those skilled in the art should understand that the disclosed range in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present application (but not limited to) having similar functions.< / hdfsfile> < / blocksize> < / hdfsfile> < / blocksize> ​< / hdfsfile> < / blocksize> < / hdfsfile> < / blocksize>

Claims

1. A data storage method, characterized in that, Applied to a data storage system, which includes a big data platform and columnar storage nodes, the method includes: The big data platform divides the data to be stored into multiple slices and stores each slice in a temporary directory. Based on the access frequency of the slice data, it allocates at least one target storage node from the columnar storage nodes to the slice data. Receive a target write command, and in response to the target write command, set the target storage node to a restricted access state, in which only the target write command is allowed to access the target storage node; The big data platform retrieves slice data from the temporary directory, copies the slice data to the access directory of the corresponding target storage node, and maps the slice data in the access directory of the target storage node to the temporary partition of the target storage node to complete the data writing.

2. The method according to claim 1, characterized in that, The step of dividing the data to be stored into multiple slices includes: The raw data is preprocessed according to the data format requirements of the big data platform to obtain the data to be stored; Obtain the status flag of the data to be stored. When the status flag indicates that the data to be stored conforms to the data format requirements, the data to be stored is divided into multiple slices.

3. The method according to claim 1, characterized in that, The step of mapping the slice data under the access directory of the target storage node to the temporary partition of the target storage node to complete the data writing includes: Verify whether the number of temporary partitions of the target storage node is consistent with the number of slice data under the access directory of the target storage node. If they are consistent, map the slice data to the corresponding temporary partitions one by one.

4. The method according to claim 1, characterized in that, The step of receiving the target write command and, in response to the target write command, setting the target storage node to a restricted access state includes: Upon receiving the attach command, in response to the attach command, execute the detach table command to lock the metadata of the target storage node; The attach command is executed to acquire the metadata lock of the target storage node, thereby restricting access to the target storage node to only allow the attach command.

5. The method according to claim 4, characterized in that, The step of mapping the slice data under the access directory of the target storage node to the temporary partition of the target storage node includes: The `attach` command is executed to define the data source of the first temporary partition as the first slice data, so as to map the first slice data to the corresponding first temporary partition one by one; the first slice data is any slice data under the access directory of the target storage node, and the first temporary partition is any temporary partition of the target storage node.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the data write volume of each target storage node, and obtain the total data volume based on the data write volume of each target storage node; Verify whether the total amount of data matches the amount of data to be stored. If they do not match, generate a prompt message to indicate that data storage has failed.

7. The method according to claim 1, characterized in that, After allocating at least one target storage node from the columnar storage nodes to the slice data based on the access frequency of the slice data, the method further includes: Obtain the access status of the target storage node. When the access status is allowed, pause copying the slice data under the temporary directory to the access directory of the target storage node until the target storage node changes to the restricted access status.

8. A data storage system, characterized in that, The data storage system includes a big data platform and columnar storage nodes; The big data platform is used to divide the data to be stored into multiple slices, store each slice in a temporary directory, and allocate at least one target storage node from the columnar storage nodes to the slice based on the access frequency of the slice data. The big data platform is also used to receive a target write command, and in response to the target write command, set the target storage node to a restricted access state, wherein only the target write command is allowed to access the target storage node in the restricted access state. The big data platform is also used to obtain slice data from the temporary directory, copy the slice data to the access directory of the corresponding target storage node, and map the slice data in the access directory of the target storage node to the temporary partition of the target storage node to complete the data writing.

9. The system according to claim 8, characterized in that, The target storage node is used to merge the slice data in all temporary partitions corresponding to the same access directory, and write the merged slice data into the contiguous storage space of the target storage node.

10. A computer program product, characterized in that, The computer program product includes instructions that, when executed, cause the method as described in any one of claims 1-7 to be implemented.

Images