Data processing method, device, electronic device and readable storage medium

Abstract

The embodiments of the present application provide a data processing method, device, electronic device and computer-readable storage medium, which relate to the field of data storage technology. The method includes: obtaining a data storage request, the data storage request includes the data to be stored, and caching the data to be stored in the form of a column array in the memory; when the data cached in the memory reaches a preset size, compressing the cached data to obtain a data block; searching for free storage space of each data file from the space management file, and determining the target data file for storing the data block and the storage location of the data block in the target data file from each data file in combination with the size of the data block; moving the data block to the storage location for storage. The embodiments of the present application avoid the column storage system from generating too much write IO in a frequently updated state, and avoids generating a large number of small files.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and specifically, to a data processing method, device, electronic device, and readable storage medium. Background Art

[0002] In the field of analytical databases, column storage systems are usually used to store data in order to speed up data analysis. In a column storage system, table data is organized by column, for example, one column is stored in one data file. In this way, only the columns involved need to be read during the query process, avoiding reading irrelevant column data.

[0003] Writing a batch of updated data in a column storage system requires an IO number equal to the number of columns. Therefore, the existing data update process has problems such as excessive IOPS and the generation of multiple files, which in turn leads to a deterioration in the writing capability of the column storage system and even causes write stall. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium, and computer program product, which can solve the above-mentioned problems of the prior art. The technical solution is as follows:

[0005] According to one aspect of an embodiment of the present application, a data processing method is provided, the method comprising:

[0006] Obtain a data storage request, the data storage request including the data to be stored, and cache the data to be stored in the form of a column array in the memory;

[0007] When the data cached in the memory reaches a preset size, the cached data is compressed to obtain a data block;

[0008] Searching for free storage space of each data file from the space management file, and determining a target data file for storing the data block and a storage position of the data block in the target data file from the data files in combination with the size of the data block;

[0009] Moving data blocks to storage locations for storage;

[0010] Among them, each data file is located in a persistent storage device.

[0011] As an optional implementation, compressing the cached data to obtain a data block includes:

[0012] Serialize and compress each column array in the memory to obtain the corresponding sub-data block;

[0013] Aggregate all sub-data blocks to obtain a data block.

[0014] As an optional implementation, the data block is moved to a target storage location for storage, and then the method further includes:

[0015] Determining storage-related meta-information of the data block;

[0016] The meta information is stored in a first meta information file in the memory.

[0017] As an optional implementation manner, the meta information includes at least one of the following information:

[0018] A data block identifier of the data block and a corresponding first offset;

[0019] The file identifier of the target data file;

[0020] The sub-data block identifier of each sub-data block and the corresponding second offset;

[0021] The first offset is used to indicate the storage position of the corresponding data block in the target data file; and the second offset is used to indicate the relative position of the corresponding data sub-block in the storage position.

[0022] As an optional implementation, the data block is moved to a target storage location for storage, and then the method further includes:

[0023] receiving a data read request, the data read request including a target sub-data block identifier of a target sub-data block and a target data block identifier of a target data block corresponding to the target sub-data block;

[0024] Searching the first meta information file for target meta information including the target data block identifier;

[0025] Reading a target sub-data block from the target data file into the memory according to the file identifier of the target data file in the target meta information, a first offset corresponding to the target data block identifier, and a second offset corresponding to the target sub-data block identifier;

[0026] The target sub-data block is decompressed in memory and decoded to obtain a column array as a reading result.

[0027] As an optional implementation, the method further includes: storing the meta information in a second meta information file in a persistent storage device.

[0028] As an optional implementation, the space management file is an order-preserving data structure;

[0029] The nodes in the data structure are used to represent data files; the node information of the nodes includes the first offset and the remaining space of the corresponding data file; and each node is sorted according to the corresponding offset.

[0030] According to another aspect of an embodiment of the present application, a data processing device is provided, the device comprising:

[0031] A cache module is used to obtain a data storage request, the data storage request includes data to be stored, and cache the data to be stored in the form of a column array in the memory;

[0032] A compression module is used to compress the cached data to obtain data blocks when the cached data in the memory reaches a preset size;

[0033] A storage location determination module is used to search for free storage space of each data file from the space management file, and determine a target data file for storing the data block and a storage location of the data block in the target data file from each data file in combination with the size of the data block;

[0034] A storage module, used for moving data blocks to a storage location for storage;

[0035] Among them, each data file is located in a persistent storage device.

[0036] As an optional implementation, the compression module includes:

[0037] The column array compression submodule is used to serialize and compress each column array in the memory to obtain the corresponding sub-data block;

[0038] The summary submodule is used to summarize all sub-data blocks to obtain a data block.

[0039] As an optional implementation manner, the data processing device further includes:

[0040] A meta-information determination module, used to determine the storage-related meta-information of the data block;

[0041] The first storage module is used to store the meta information in a first meta information file in the memory.

[0042] As an optional implementation manner, the meta information includes at least one of the following information:

[0043] A data block identifier of the data block and a corresponding first offset;

[0044] The file identifier of the target data file;

[0045] The sub-data block identifier of each sub-data block and the corresponding second offset;

[0046] The first offset is used to indicate the storage position of the corresponding data block in the target data file; and the second offset is used to indicate the relative position of the corresponding data sub-block in the storage position.

[0047] As an optional implementation manner, the data processing device further includes:

[0048] A read request receiving module, used for receiving a data read request, wherein the data read request includes a target sub-data block identifier of a target sub-data block and a target data block identifier of a target data block corresponding to the target sub-data block;

[0049] A meta information search module, used for searching the first meta information file for target meta information including the target data block identifier;

[0050] A sub-data block search module is used to read a target sub-data block from a target data file into a memory according to a file identifier of the target data file in the target meta information, a first offset corresponding to the target data block identifier, and a second offset corresponding to the target sub-data block identifier;

[0051] The decompression module is used for decompressing the target sub-data block in the memory and decoding to obtain a column array as a reading result.

[0052] As an optional implementation manner, the data processing device includes:

[0053] The second storage module is used to store the meta information in a second meta information file in the persistent storage device.

[0054] As an optional implementation, the space management file is an order-preserving data structure;

[0055] The nodes in the data structure are used to represent data files; the node information of the nodes includes the first offset and the remaining space of the corresponding data file; and each node is sorted according to the corresponding offset.

[0056] According to another aspect of an embodiment of the present application, an electronic device is provided. The electronic device includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0057] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0058] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0059] By obtaining a data storage request, wherein the data storage request includes the data to be stored, the data to be stored is cached in the form of a column array, and when the cached data in the memory reaches a preset size, the cached data is compressed to obtain a data block, and the free storage space of each data file is searched from the space management file, and combined with the size of the data block, the target data file for storing the data block and the storage location of the data block in the target data file are determined, and the data block is moved to the storage location for storage, thereby avoiding excessive write IO in the column storage system under the state of frequent updates and avoiding the generation of a large number of small files. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0061] Figure 1 A schematic diagram of writing data to a column storage system provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;

[0063] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of a process for determining a target data file provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the structure of a data processing system of another embodiment provided in the embodiment of the present application;

[0066] Figure 6 A schematic diagram of a data reading process of a data processing method provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0068] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0070] Those skilled in the art will appreciate that, unless specifically stated, the singular forms "one", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we call an element "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or may refer to the one element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B".

[0071] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0072] When writing a batch of updated data in a column storage system, the number of IOs required is equal to the number of columns. Figure 1 , which exemplarily shows a schematic diagram of writing data in a column storage system. The column storage system includes three column arrays, namely ColA, ColB, and ColC. Whenever new data needs to be written into the system, a new data structure needs to be created at the end of the three column arrays. When the new data is written into the newly created data structure, it means that a row of data Row3 is added to the column storage system.

[0073] Since the existing technology generates a new file every time the data is updated, a large number of small files will be generated when the update speed is fast, slowing down the query performance; in order to avoid too many small files, the background needs to continuously merge data, consuming a lot of CPU and IO, affecting system stability.

[0074] The data processing method, device, electronic device, computer-readable storage medium, and computer program product provided in this application are intended to solve the above technical problems in the prior art.

[0075] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0076] The data processing method provided in this application can be applied to Figure 2 The data processing system includes a terminal 110 and a server 120. It should be understood that the data storage system provided in the embodiment of the present application can be a background data storage system of the target application, and the server 120 in the data storage system can communicate with the terminal device 110 running the target application.

[0077] Exemplarily, after the target application running on the terminal device 110 generates data to be stored, the terminal device 110 sends a data storage request to the server 120, and the data storage request includes the data to be stored. The server 120 includes a memory and a persistent storage device such as a disk, a solid-state drive (SSD) or an NVMe solid-state drive (NVMe SSD), etc. Compared with the memory, the persistent storage device is a device for persistently storing data.

[0078] The server 120 caches the data to be stored in the form of a column array in the memory. When the cached data in the memory reaches a preset size, the server 120 compresses the cached data to obtain data blocks, and searches for free storage space for each data file from the space management file on the persistent storage device. It should be understood that the space management file is a file on the persistent storage device for recording free storage space for each data file, and the data file is a file for storing a database. The server 120 further determines the target data file for storing the data block and the storage location of the data block in the target data file based on the size of the data block, and moves the data block to the storage location for storage. The server stores the storage-related metadata of the data block in the first metadata file in the memory, so as to quickly obtain the target data when reading the data.

[0079] The meta information of the embodiment of the present application includes: a data block identifier of a data block and a corresponding first offset; a file identifier of a target data file; a sub-data block identifier of each sub-data block and a corresponding second offset. The first offset is used to indicate the storage position of the corresponding data block in the target data file; and the second offset is used to indicate the relative position of the corresponding data sub-block in the storage position.

[0080] When the target application running on the terminal device 110 needs to read data, the terminal device 110 sends a data read request to the server 120, and the data read request includes a target sub-data block identifier of the target sub-data block and a target data block identifier of the target sub-data block. The server 120 searches for the target meta-information including the target data block identifier in the first meta-information file; reads the target sub-data block from the target data file of the persistent storage device to the memory according to the file identifier of the target data file in the target meta-information, the first offset corresponding to the target data block identifier, and the second offset corresponding to the target sub-data block identifier; decompresses the target sub-data block in the memory, and decodes it to obtain a column array as the reading result.

[0081] See also Figure 3 , which exemplarily shows a flow chart of a data processing method according to an embodiment of the present application, as shown in the figure, including:

[0082] S101, obtaining a data storage request, wherein the data storage request includes data to be stored, and caching the data to be stored in a memory in the form of a column array.

[0083] The embodiment of the present application processes a storage request for data to be stored in the memory, parses the data to be stored into a columnar storage form in the memory, and caches it in the memory. For example, if the memory includes 3 column arrays, if each column array includes 3 sub-data at a certain moment, when a data to be stored is subsequently obtained, the data to be stored will be split into 3 sub-data and stored in order at the end of the 3 column arrays.

[0084] S102: When the data cached in the memory reaches a preset size, compress the cached data to obtain a data block.

[0085] The embodiment of the present application sets a data volume threshold for the data cached in the memory. When the cached data reaches the data volume threshold, the data is compressed. It should be understood that the data volume threshold set by the present application is relatively small. The present application does not need to accumulate too much data, ensuring the timeliness of data update.

[0086] S103, searching for free storage space of each data file from the space management file, and determining a target data file for storing the data block and a storage position of the data block in the target data file from each data file in combination with the size of the data block.

[0087] It should be noted that the space management file of the embodiment of the present application can be located in the memory, so as to improve the efficiency of obtaining the free storage space of the data file, and each data file is located in a persistent storage device such as a disk, a solid-state drive (SSD) or an NVMe solid-state drive (NVMe SSD), wherein the space management file is used to store the free storage space of each data file, and the size of the free storage space determines the size of the data block that can be stored in the data file later. Therefore, after obtaining the free storage space of each data file from the space management file, the data file that can accommodate the data block can be selected in combination with the size of the data block. When there are multiple data files that can accommodate the data block, the target data file can be determined in a random manner.

[0088] S104: Move the data block to a storage location for storage.

[0089] The data processing method of the embodiment of the present application obtains a data storage request, wherein the data storage request includes data to be stored, caches the data to be stored in the form of a column array, and when the cached data in the memory reaches a preset size, compresses the cached data to obtain a data block, searches for free storage space of each data file from the space management file, and determines a target data file for storing the data block and a storage location of the data block in the target data file in combination with the size of the data block, and moves the data block to the storage location for storage, thereby avoiding excessive write IO in a column storage system under a frequently updated state and avoiding the generation of a large number of small files.

[0090] Based on the above embodiments, as an optional embodiment, compressing the cached data to obtain a data block includes:

[0091] Serialize and compress each column array in the memory to obtain the corresponding sub-data block;

[0092] Aggregate all sub-data blocks to obtain a data block.

[0093] by Figure 1 For example, Figure 1 There are three column arrays in total. By serializing and compressing each column array, three sub-data blocks can be obtained. The three sub-data blocks are packaged to obtain a data block.

[0094] Based on the above embodiments, as an optional embodiment, compressing cached data into data blocks includes:

[0095] Determining storage-related meta-information of the data block;

[0096] Specifically, the meta information in the embodiment of the present application may include the following information:

[0097] A data block identifier of the data block and a corresponding first offset, where the first offset is used to indicate a storage position of the corresponding data block in the target data file;

[0098] The file identifier of the target data file;

[0099] A sub-data block identifier of each sub-data block and a corresponding second offset, where the second offset is used to indicate a relative position of the corresponding data sub-block in the storage location.

[0100] The first meta information file storing the meta information in the memory. By caching the meta information in the memory, it is convenient for fast access and ensures reading efficiency. Specifically, the embodiment of the present application can use the data block identifier of the data block in the meta information as the key and use other information in the meta information except the data block identifier as the value for storage.

[0101] Based on the above embodiments, as an optional embodiment, the space management file is an order-preserving data structure; for example, it can be a red-black tree (also known as a self-balancing binary search tree) structure. The nodes in the data structure are used to represent data files; the node information of the node includes the first offset and the remaining space of the corresponding data file; each node is sorted by the corresponding offset.

[0102] See also Figure 4 , which exemplarily shows a flow chart of determining a target data file in an embodiment of the present application. As shown in the figure, the red-black tree includes 3 nodes, wherein the first offset of node 1 is 200 bytes, and the remaining space is 50 bytes, the first offset of node 2 is 0 bytes, and the remaining space is 100 bytes, the first offset of node 3 is 400 bytes, and the remaining space is 300 bytes. If the size of the data block is 200 bytes, then traverse downward from the root node of the red-black tree, that is, node 1, to determine that the remaining space of node 3 satisfies the storage of the data block, then move the data block to the data file corresponding to node 3, and update the first offset and remaining space of the data file in the space management file. Correspondingly, the first offset recorded in node 3 is 600 (400+200), and the remaining space is 100 (300-200).

[0103] See also Figure 5 , which exemplarily shows a structural diagram of a data processing system of another embodiment of the present application. As shown in the figure, the overall structure of the system is divided into three parts, including a memory table MemTable101, a data block storage module BlobStore102 and a metadata storage module PageMetaStore103:

[0104] MemTable 101 is a memory structure used to store the latest written data. The data in MemTable has been parsed and stored in a column-based format (called a column array). When the data in MemTable reaches a preset size, it is flushed to BlobStore 102.

[0105] Before being flushed, the data of MemTable101 will first be serialized and compressed to obtain a data block, which is called a Page. Each Page has a unique ID, that is, the data block identifier PageID. It should be noted that the size of the Page in the embodiment of the present application is determined by the size of the Memtable, so the size of the Page is not fixed.

[0106] As an optional embodiment, MemTable101 adopts a columnar data format, that is, an array is used to store the data of each column, and one array corresponds to one column (column array). Data writing only needs to be appended (Append) to the end of the array. MemTable101 contains two parts: Immutable (an unmodifiable table, which is a temporary form of the Mutable flush phase) and Mutable (a modifiable table, updated data can be written normally). Only Mutable can be written, and Immutable is the intermediate state of Mutable during the Flush process.

[0107] BlobStore 102 is used to actually store the content of Page. BlobStore 102 uses the data file (BlobFile) 1022 on the persistent storage device to actually store data, and records the free space (Free Space) of the corresponding BlobFile 1022 through the space management file (SpaceMap) 1021. BlobStore 102 can contain multiple pairs of SpaceMap 1021 and BlobFile 1022, and BlobFile 1022 has a capacity limit.

[0108] As an optional embodiment, SpaceMap 1021 uses a red-black tree (self-balancing binary search tree) data structure. Each node on the tree records the corresponding free space, and the node information includes (offset Offset, remaining space Length), and is sorted according to Offset.

[0109] PageMetaStore 103 is used to store the meta information (PageMeta) of the Page. The first meta information file (PageDirectory) 1031 in the memory is used to cache PageMeta for quick access, and PageMeta is persistently stored in the second meta information file (PageMate WAL File) 1032 on the persistent storage device.

[0110] As an optional embodiment, PageDirectory 1031 uses a hash table data structure to store meta information, so as to facilitate rapid acquisition of the meta information of the corresponding Page according to the PageID.

[0111] Figure 5 The processing flow of the data processing system shown includes:

[0112] S201 . Process a write request for data to be stored in the memory, and cache the data to be stored in MemTable 101 .

[0113] MemTable is divided into two parts: Mutable and Immutable. When the data fills up the Mutable or the upper layer manually calls it, it will trigger the current Mutable to be converted to Immutable and generate a new Mutable for future writing; the newly generated Immutable will be serialized and compressed into Page.

[0114] Since one of the features of column storage is that it supports reading only some columns, the Page needs to have an index for each column to quickly obtain the Offset of a specific column.

[0115] S202, according to the size of the Page, apply for appropriate FreeSpace from SpaceMap 1021 in BlobStore 102, and generate PageMeta.

[0116] The contents of PageMeta include BlobFile ID, first offset Page Offset, data block size PageSize, second offset Column Offset of each column in Page, and so on.

[0117] S203, write the Page to the specific location of the corresponding BlobFile1022.

[0118] S204, write PageMeta into the PageMeta WAL File 1032 of the persistent storage device.

[0119] S205, cache PageMeta in the PageDirectory 1031 in the memory. By storing the meta information in a persistent storage device, the data can be restored to the PageDirectory in the memory during the startup process, and the information such as PageID, Offset, Size in the PageMetaWAL File can also be used to restore the space management file SpaceMap, providing a backup for system restart.

[0120] See also Figure 6 , which exemplarily shows the data reading process of the data processing method of an embodiment of the present application, including:

[0121] S301, receiving a data read request, the data read request including a target sub-data block identifier of a target sub-data block and a target data block identifier of a target data block corresponding to the target sub-data block;

[0122] S302: Search the first meta information file for target meta information including a target data block identifier.

[0123] When storing the first metadata file, the embodiment of the present application may use the target data block identifier as the key and other metadata as the value for storage. During the data reading phase, the key corresponding to the target database identifier may be searched in the first metadata file to quickly locate the target metadata.

[0124] S303, reading the target sub-data block from the target data file into the memory according to the file identifier of the target data file in the target meta information, the first offset corresponding to the target data block identifier, and the second offset corresponding to the target sub-data block identifier.

[0125] Specifically, according to the file identifier of the target data file in the target metadata, the location of the target data file is determined from the persistent storage device, and the storage location of the target data in the persistent storage device is further determined according to the first offset corresponding to the target data block identifier, and the storage location of the target sub-data block is further determined according to the second offset corresponding to the target sub-data block identifier, and the target sub-data block is read from the target data file to the memory according to the storage location.

[0126] S304, decompress the target sub-data block in the memory and decode it into a column array as a reading result.

[0127] The present application embodiment provides a data processing device, such as Figure 7 As shown, the data processing device may include: a cache module 101, a compression module 102, a storage location determination module 103 and a storage module 103, wherein:

[0128] The cache module 101 is used to obtain a data storage request, the data storage request includes data to be stored, and cache the data to be stored in the form of a column array into the memory;

[0129] The compression module 102 is used to compress the cached data to obtain data blocks when the cached data in the memory reaches a preset size;

[0130] The storage location determination module 103 is used to search the free storage space of each data file from the space management file, and determine the target data file for storing the data block and the storage location of the data block in the target data file from each data file in combination with the size of the data block;

[0131] The storage module 104 is used to move the data block to a storage location for storage;

[0132] Among them, each data file is located in a persistent storage device.

[0133] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0134] As an optional implementation, the compression module includes:

[0135] The column array compression submodule is used to serialize and compress each column array in the memory to obtain the corresponding sub-data block;

[0136] The summary submodule is used to summarize all sub-data blocks to obtain a data block.

[0137] As an optional implementation manner, the data processing device further includes:

[0138] A meta-information determination module, used to determine the storage-related meta-information of the data block;

[0139] The first storage module is used to store the meta information in a first meta information file in the memory.

[0140] As an optional implementation manner, the meta information includes at least one of the following information:

[0141] A data block identifier of the data block and a corresponding first offset;

[0142] The file identifier of the target data file;

[0143] The sub-data block identifier of each sub-data block and the corresponding second offset;

[0144] The first offset is used to indicate the storage position of the corresponding data block in the target data file; and the second offset is used to indicate the relative position of the corresponding data sub-block in the storage position.

[0145] As an optional implementation manner, the data processing device further includes:

[0146] A read request receiving module, used for receiving a data read request, wherein the data read request includes a target sub-data block identifier of a target sub-data block and a target data block identifier of a target data block corresponding to the target sub-data block;

[0147] A meta information search module, used for searching the first meta information file for target meta information including the target data block identifier;

[0148] A sub-data block search module is used to read a target sub-data block from a target data file into a memory according to a file identifier of the target data file in the target meta information, a first offset corresponding to the target data block identifier, and a second offset corresponding to the target sub-data block identifier;

[0149] The decompression module is used for decompressing the target sub-data block in the memory and decoding to obtain a column array as a reading result.

[0150] As an optional implementation manner, the data processing device includes:

[0151] The second storage module is used to store the meta information in a second meta information file in the persistent storage device.

[0152] As an optional implementation, the space management file is an order-preserving data structure;

[0153] The nodes in the data structure are used to represent data files; the node information of the nodes includes the first offset and the remaining space of the corresponding data file; and each node is sorted according to the corresponding offset.

[0154] In an embodiment of the present application, an electronic device is provided, including a memory, a processor and a computer program stored in the memory, the processor executes the above-mentioned computer program to implement the steps of the data processing method, which can be implemented compared with the related art: by obtaining a data storage request, wherein the data storage request includes data to be stored, the data to be stored is cached in the form of a column array, when the cached data in the memory reaches a preset size, the cached data is compressed to obtain a data block, the free storage space of each data file is searched from the space management file, and combined with the size of the data block, the target data file for storing the data block and the storage location of the data block in the target data file are determined, and the data block is moved to the storage location for storage, thereby avoiding excessive write IO in the column storage system under frequent updates and avoiding the generation of a large number of small files.

[0155] In an alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0156] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0157] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0158] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0159] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0160] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0161] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0162] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0163] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0164] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: Obtaining a data storage request, wherein the data storage request includes data to be stored, and caching the data to be stored in a memory in the form of a column array; When the data cached in the memory reaches a preset size, compressing the cached data to obtain a data block; Searching for free storage space of each data file from a space management file on a persistent storage device, and determining a target data file for storing the data block and a storage position of the data block in the target data file from the data files in combination with the size of the data block; Moving the data block to the storage location for storage; Wherein, each of the data files is located in a persistent storage device; Compressing the cached data to obtain a data block includes: Serializing and compressing each column array in the memory to obtain a corresponding sub-data block; All the sub-data blocks are aggregated to obtain the data block.

2. The data processing method according to claim 1, characterized in that: The step of moving the data block to the storage location for storage further includes: Determining storage-related meta-information of the data block; The meta information is stored in a first meta information file in the memory.

3. The data processing method according to claim 2, characterized in that: The meta information includes at least one of the following information: A data block identifier of the data block and a corresponding first offset; The file identifier of the target data file; The sub-data block identifier of each sub-data block and the corresponding second offset; The first offset is used to indicate the storage position of the corresponding data block in the target data file; and the second offset is used to indicate the relative position of the corresponding data sub-block in the storage position.

4. The data processing method according to claim 3, characterized in that: The step of moving the data block to the target storage location for storage further includes: receiving a data read request, the data read request including a target sub-data block identifier of a target sub-data block and a target data block identifier of a target data block corresponding to the target sub-data block; Searching the first meta information file for target meta information including the target data block identifier; Reading the target sub-data block from the target data file to the memory according to the file identifier of the target data file in the target meta information, the first offset corresponding to the target data block identifier, and the second offset corresponding to the target sub-data block identifier; The target sub-data block is decompressed in the memory and decoded to obtain a column array as a reading result.

5. The data processing method according to claim 2, characterized in that: Also includes: The meta information is stored in a second meta information file in the persistent storage device.

6. The data processing method according to claim 1, characterized in that: The space management file is an order-preserving data structure; The nodes in the data structure are used to represent the data files; the node information of the nodes includes the first offset and the remaining space of the corresponding data file; and each of the nodes is sorted according to the corresponding offset.

7. A data processing device, characterized in that: include: A cache module, used for obtaining a data storage request, wherein the data storage request includes data to be stored, and caching the data to be stored in a memory in the form of a column array; A compression module, configured to compress the cached data to obtain a data block when the data cached in the memory reaches a preset size; wherein compressing the cached data to obtain a data block comprises: serializing and compressing each column array in the memory to obtain a corresponding sub-data block; and aggregating all the sub-data blocks to obtain the data block; A storage location determination module, used to search for free storage space of each data file from a space management file on a persistent storage device, and determine a target data file for storing the data block and a storage location of the data block in the target data file from the data files in combination with the size of the data block; A storage module, used for moving the data block to the storage location for storage; Wherein, each of the data files is located in a persistent storage device.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images