Data processing method and apparatus, storage system, computer readable storage medium, and computer program product

Abstract

The present application provides a data processing method and apparatus, a storage system, a computer readable storage medium and a computer program product. The method comprises: acquiring from a first data server multiple pieces of data to be processed, the data to be processed comprising keywords to be processed and data values to be processed; generating an object data file on the basis of the multiple pieces of data to be processed, the object data file at least comprising the data to be processed and storage position information of the data to be processed in a second data server, and the second data server being used for storing data with the access frequency lower than a preset frequency threshold and the degree-of-importance value smaller than a preset degree-of-importance threshold; storing at least one object data file to the second data server; and updating the data values to be processed corresponding to the keywords to be processed in the first data server to position values corresponding to the storage position information.

Description

Data processing method, device, storage system, computer-readable storage medium, and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202311166312.5 and application date September 11, 2023, and claims the priority of the above Chinese patent application. The entire content of the above Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to data processing technology, and in particular to data processing methods, devices, storage systems, computer-readable storage media, and computer program products. Background Art

[0004] Distributed storage systems store data across multiple independent data servers, creating multiple replicas to achieve high data reliability and system availability. Over time, data access popularity gradually decreases. Storage systems refer to data with low access density as cold data, while data with high access density is called hot data. For cost considerations, industry vendors typically store cold data on low-cost hardware that doesn't require high throughput, such as local mechanical hard drives or inexpensive cloud storage offering block or object storage. In contrast, hot data is typically stored on hardware that offers high throughput but is more expensive, such as solid-state drives and non-volatile memory. The process of separating data from hot data storage and storing it on cold storage is called data cooling. Related technologies utilize a bypass cooling module that scans and filters data, writing the filtered cold data to inexpensive cold storage media and deleting the cold data from the more expensive hot storage media. However, bypass cooling requires additional resources to support the bypass cooling module, which can become a performance bottleneck. Furthermore, this approach can result in a high number of empty reads of cold data, reducing data read efficiency.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a data processing method, device, storage system, computer-readable storage medium, and computer program product, which can reduce data storage costs while ensuring availability and data reliability.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] An embodiment of the present application provides a data processing method, which is applied to a storage system, wherein the storage system includes at least a first data server and a second data server. The method includes:

[0009] Acquire a plurality of to-be-processed data from a first data server, wherein the to-be-processed data include to-be-processed keywords and to-be-processed data values ​​corresponding to the to-be-processed keywords;

[0010] generating at least one object data file based on the plurality of data to be processed, the object data file including at least the data to be processed and storage location information of the data to be processed in a second data server, the second data server being configured to store data with an access frequency lower than a preset frequency threshold and an importance value lower than a preset importance threshold;

[0011] storing the at least one object data file to a second data server;

[0012] The to-be-processed data value corresponding to the to-be-processed keyword in the first data server is updated to the location value corresponding to the storage location information.

[0013] An embodiment of the present application provides a data processing device, including:

[0014] A first acquisition module is configured to acquire a plurality of to-be-processed data from a first data server, wherein the to-be-processed data includes to-be-processed keywords and to-be-processed data values ​​corresponding to the to-be-processed keywords;

[0015] A file generation module is configured to generate at least one object data file based on the plurality of data to be processed, wherein the object data file includes at least the data to be processed and storage location information of the data value to be processed in the second data server, the second data server being used to store data having an access frequency lower than a preset frequency threshold and an importance value lower than a preset importance threshold;

[0016] a first storage module configured to store the at least one object data file to a second data server;

[0017] The data updating module is configured to update the to-be-processed data value corresponding to the to-be-processed keyword in the first data server to the location value corresponding to the storage location information.

[0018] An embodiment of the present application provides a storage system, which includes at least: a first data server and a second data server, wherein the first data server is configured to store data having an access frequency greater than or equal to a preset frequency threshold and data having an importance value greater than or equal to a preset importance threshold, and the second data server is configured to store data having an access frequency less than the preset frequency threshold and an importance value less than the preset importance threshold;

[0019] The first data server includes a first memory and a first processor, and the second data server includes a first memory and a second processor. The first memory and the second memory are used to store computer-executable instructions; when the first processor and the second processor are used to execute the computer-executable instructions, the data processing method provided in the embodiment of the present application is implemented.

[0020] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in the embodiment of the present application when executed by a processor.

[0021] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the data processing method provided in the embodiment of the present application is implemented.

[0022] The embodiments of the present application have the following advantageous effects: after obtaining a plurality of data to be processed from a first data server, an object data file is generated based on the plurality of data to be processed. The object data file includes at least a device identifier of a second data server for storing the data to be processed and storage location information of the data to be processed in the second data server. The second data server is used to store data with an access frequency lower than a preset frequency threshold and an importance value lower than a preset importance threshold, that is, the second data server is used to store data with low access density and unimportant data. At least one object data file is stored in the second data server corresponding to the device identifier, and the data value to be processed corresponding to the keyword to be processed in the first data server is updated to the location value corresponding to the storage location information. In other words, the first data server stores data with high access density or important data (hot data), as well as keywords and location values ​​of data values ​​corresponding to the keywords for data with low access density and unimportant data (cold data). This ensures that the keywords and data values ​​of the cold data can also be read by the first data server, thereby ensuring availability and data reliability. In addition, because the space occupied by the location value of the data value corresponding to the keyword is much smaller than the space occupied by the data value itself, the space occupied by the cold data in the first data server can be reduced, thereby reducing storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG1 is a schematic diagram of a network architecture of a storage system provided in an embodiment of the present application;

[0024] FIG2 is a schematic diagram of the structure of a first data server 400-1 provided in an embodiment of the present application;

[0025] FIG3A is a schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application;

[0026] FIG3B is a schematic diagram of an implementation flow of generating an object data file according to an embodiment of the present application;

[0027] FIG3C is a schematic diagram of an implementation flow of checking data consistency provided in an embodiment of the present application;

[0028] FIG4A is a schematic diagram of a process for implementing on-demand data recovery according to an embodiment of the present application;

[0029] FIG4B is a schematic diagram of a flow chart illustrating the total capacity and invalid data capacity of an object data file according to an embodiment of the present application;

[0030] FIG4C is a schematic diagram of a process for implementing periodic data recovery according to an embodiment of the present application;

[0031] FIG5 is a schematic diagram of an implementation flow of a data reading method provided in an embodiment of the present application;

[0032] FIG6 is a schematic diagram of an implementation flow of a data writing method provided in an embodiment of the present application;

[0033] FIG7 is another schematic diagram of the network structure of the distributed storage system provided in an embodiment of the present application;

[0034] FIG8 is a schematic diagram of the structure of a Blob file provided in an embodiment of the present application;

[0035] FIG9 is a schematic diagram of a double-chain cache queue provided in an embodiment of the present application;

[0036] FIG10 is a schematic diagram of a process flow for implementing write data provided in an embodiment of the present application;

[0037] FIG11 is a schematic diagram of the implementation flow of reading data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0041] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0042] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0043] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0044] 1) RocksDB is a Log-Structured Merge Tree (LSM-tree) architecture engine developed based on LevelDB that provides key-value storage and read-write capabilities.

[0045] 2) Sorted Sequence Table (SST) file is a file in which all data is sorted and stored in the form of key-value in the order of the data key.

[0046] 3) Compaction: In the LSM storage structure, the process of merging and compressing data from different layers.

[0047] 4) Raft is a consensus algorithm for achieving consistency among multiple replicas of data in a distributed system.

[0048] 5) Blob file refers to an object data file organized according to a specific storage structure.

[0049] 6) Garbage Collection (GC) refers to the space recovery of garbage data in the storage system.

[0050] 7) Hot data refers to data that is frequently accessed and critical to business and applications. This data typically requires fast and efficient access and processing, and therefore needs to be stored on high-performance, low-latency storage devices such as SSDs and memory.

[0051] 8) Cold data refers to data that is accessed less frequently and is not critical to business and applications. This data typically needs to be stored long-term but does not require frequent access or processing. Therefore, it can be stored on lower-cost, higher-capacity storage devices, such as tape libraries.

[0052] 9) Data cooling is the process of separating data from hot data storage media and storing it in cold data storage media.

[0053] 10) Serialization processing refers to converting data in a specific format into a recoverable string sequence. In the embodiment of the present application, it is to convert storage location information into a location value represented by a string sequence.

[0054] 11) Deserialization processing is the process of restoring the string sequence obtained by serialization to the original format data. In the embodiment of the present application, the position value represented by the string sequence is converted into storage position information.

[0055] In order to better understand the data processing method provided in the embodiment of the present application, the data processing method for separating hot and cold data in the related art and the existing shortcomings are first explained.

[0056] There are two data processing methods for hot and cold data separation in related technologies:

[0057] First, bypass cooling. The process of separating data from hot data media and storing it on cold data media is called data cooling. This method utilizes a bypass cooling module to scan and filter data, writing the filtered cold data to inexpensive cold storage media and deleting the cold data from the expensive hot media.

[0058] Second, use RocksDB's SST files to reduce cold data. Based on write time characteristics, try to concentrate data in the same SST file and write RocksDB's cold data SST files to inexpensive storage media.

[0059] Among them, the bypass cooling method requires additional resources to support the bypass cooling module, which will cause the cooling module to become a performance bottleneck. In addition, this method will cause a large number of empty reads in cold data reading. The cooling method using Rocksdb's SST file needs to be sorted according to the data write time. It requires a unique Rocksdb Comparator strategy to concentrate keywords with close time in one SST file, and the cooling strategy cannot be expanded. The fixed SST file format cannot cope with the scenario of splitting and merging replicas. In a distributed storage system, the SST files of multiple replicas are independent of each other. The differences in SST files between replicas will lead to the problem of inconsistency of cooling data between replicas.

[0060] This approach uses self-aggregated data to organize cooling files. During RocksDB's data compaction process, custom cooling strategies are used to filter cold data for cooling. Cooling strategies can be expanded based on user needs. The cooling file metadata records the key range and corresponding replica IDs routed to the data replica, serving as a replica metadata verification tool. In the event of replica splits and merges, the cooling blob file data can be automatically adjusted and organized. In this method, cooling is performed only on the primary replica for multiple data replicas. The post-cooling location information is replicated to other replicas via the Raft protocol, ensuring consistency in the cooling data across multiple replicas.

[0061] The embodiments of the present application provide a data processing method, device, storage system, computer-readable storage medium and computer program product, which retains the key with small storage space consumption in the hot data storage cluster, and transfers the value with large storage space consumption to the cold data storage cluster. In this way, the hot data storage cluster only stores the location information of the key and value with small data volume in the cold data storage cluster. During the read access process of the business, there will be no empty reads, and it can accurately judge whether the data exists and the actual content of the data, and it can also ensure the consistency of multiple copies of data. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Below, an exemplary application when the device is implemented as a server will be described.

[0062] Refer to Figure 1, which is an architectural diagram of a distributed storage system 100 provided in an embodiment of the present application. As shown in Figure 1, the system includes a terminal 200, a first data server cluster (exemplarily showing a first data server 400-1 as a master copy, a third data server 400-2 as a slave copy, and a third data server 400-3 as a slave copy), and a second data server cluster (exemplarily showing a second data server 300-1, a second data server 300-2, ..., a second data server 300-K), wherein the terminal 200 and the first data server cluster, and the first data server cluster and the second data server cluster are connected through a network (not shown in Figure 1), and the network can be a wide area network or a local area network, or a combination of the two.

[0063] Each server in the first data server cluster is used to store data with an access frequency higher than or equal to a preset frequency threshold, and data with an importance value greater than or equal to a preset importance threshold (in some embodiments, this may also be referred to as hot data), and the second data server cluster is used to store data with an access frequency lower than the preset frequency threshold and an importance value lower than the preset importance threshold (in some embodiments, this may also be referred to as cold data).

[0064] During the data merging and compression process, the first data server 400-1 obtains multiple data to be processed from the stored data through a filter. In this embodiment of the present application, the data to be processed may be data with an access frequency lower than a preset frequency threshold and an importance value lower than a preset importance threshold, i.e., cold data. The data to be processed includes a keyword to be processed and a data value to be processed corresponding to the keyword to be processed. The first data server 400-1 generates at least one object data file based on the multiple data to be processed. The object data file includes at least the device identifier of the second data server storing the data to be processed and the storage location information of the data to be processed in the second data server. Then, based on the storage location information, the first data server 400-1 stores the at least one object data file to the second data server corresponding to the device identifier, and updates the data value to be processed corresponding to the keyword to be processed in the first data server to the location value corresponding to the storage location information. The location value is obtained by serializing the storage location information. The storage location information can be converted using a preset serialization function to obtain a location value in string format. In some embodiments, the first data server 400-1 sends the storage location information to the third data server 400-2 serving as a slave copy and the third server 400-3 serving as a slave copy, so as to update the to-be-processed data value corresponding to the to-be-processed keyword in the third data server 400-2 to the location value corresponding to the storage location information, and update the to-be-processed data value corresponding to the to-be-processed keyword in the third data server 400-3 to the location value corresponding to the storage location information, thereby achieving data consistency between the master and slave copies.

[0065] In an embodiment of the present application, the first data server stores keywords of hot data and data values ​​corresponding to the keywords, as well as keywords of cold data and location information of data values ​​corresponding to the keywords. This can reduce data storage costs while ensuring availability and data reliability.

[0066] Upon receiving a read data operation, terminal 200 sends a read data request to first data server 400-1, carrying a to-be-read keyword for the data to be read. Upon determining that the to-be-read keyword is stored in first data server 400-1, terminal 200 obtains attribute information of the data value corresponding to the to-be-read keyword. If the attribute information indicates that the first data server stores a location value corresponding to storage location information of the to-be-read data value in the second data server, terminal 200 deserializes the location value to obtain storage location information. Based on the storage location information, terminal 200 obtains the to-be-read data value from the second data server and sends the to-be-read data value to terminal 200. In this way, when cold data needs to be read, the to-be-read data value can be obtained based on the storage location information of the cold data value in the first data server, thereby avoiding empty data reads and improving data reading efficiency.

[0067] In some embodiments, the first data server, the second data server, and the third data server can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 200 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0068] Referring to FIG. 2 , FIG. 2 is a schematic diagram of the structure of a first data server 400 - 1 provided in an embodiment of the present application. The first data server 400 - 1 shown in FIG. 2 includes: at least one processor 410 , a memory 450 , at least one network interface 420 , and a user interface 430 . The various components in the first data server 400 - 1 are coupled together via a bus system 440 . It will be appreciated that the bus system 440 is used to enable connection and communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity of explanation, in FIG. 2 , all of the various buses are labeled as the bus system 440 .

[0069] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0070] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0071] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more data servers physically located away from the processor 410.

[0072] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0073] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0074] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0075] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0076] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0077] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0078] In some embodiments, the apparatus provided in the embodiments of the present application can be implemented in software. FIG2 shows a data processing device 455 stored in a memory 450. The data processing device 455 can be software in the form of a program or plug-in, and includes the following software modules: a first acquisition module 4551, a file generation module 4552, a first storage module 4553, and a data update module 4554. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0079] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0080] The data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0081] 3A is a schematic diagram illustrating an implementation flow of a data processing method provided in an embodiment of the present application. The steps shown in FIG3A are described below. The subject of each step in FIG3A is a first data server. The first data server can be a data server serving as a primary replica in a storage system.

[0082] In step 101, a plurality of data to be processed is obtained from a first data server.

[0083] The data to be processed includes a keyword to be processed and the data value to be processed corresponding to the keyword to be processed. In some embodiments, when determining that a data screening opportunity has arrived, the first data server obtains the access frequency and importance value of each data item in the first data server. The data screening opportunity is determined to have arrived when a preset interval has elapsed between the current time and the last time data was processed, or when the remaining storage space on the first data server is less than a preset space threshold. The access frequency of data can be represented by the number of times the data is accessed within a preset time period. For example, the preset time period can be 1 day, 7 days, 10 days, etc. The access frequency can be 3 accesses within 1 day, 5 accesses within 7 days, etc. The importance value of the data can be determined based on the data type. Importance values ​​corresponding to different data types can be pre-set, and then the importance value of the data can be determined based on the data type. In some embodiments, the importance value of the data can also be manually set when the data is written. After obtaining the access frequency and importance value of each data item, the first data server determines the data that meets the preset cooling conditions as the data to be processed based on the access frequency and importance value of each data item. Among them, the preset cooling condition corresponds to the cooling strategy in other embodiments. The preset cooling condition can be set according to the actual storage demand. For example, the access frequency can be less than the preset frequency threshold, or the importance value can be less than the preset importance threshold. In some embodiments, the preset cooling condition can also be that the storage duration exceeds the preset duration threshold. In the embodiment of the present application, the data that meets the preset cooling condition is determined as the data to be processed, that is, the data to be stored in the second data server. The second data server is used to store data with an access frequency lower than the preset frequency threshold and an importance value less than the preset importance threshold. It can also be understood that the second data server is used to store cold data.

[0084] In step 102, at least one object data file is generated based on the plurality of data to be processed.

[0085] In some embodiments, referring to FIG. 3B , step 102 may be implemented using steps 1021 to 1023 , which are described in detail below.

[0086] In step 1021 , a plurality of data to be processed is added to a queue of data to be processed.

[0087] It can be understood that the queue for data to be processed can be a first-in-first-out multi-producer single-consumer chain cache queue, that is, the queue for data to be processed can store data to be processed sent from multiple different first data servers, and the data to be processed in the queue for data to be processed is consumed by the aggregator, that is, the aggregator is used to aggregate the data to be processed.

[0088] In step 1022, at least one to-be-processed data constituting the object data file is determined from the to-be-processed data queue.

[0089] In some embodiments, the file size of the object data file (the size of the space available for storing data in the file) is pre-set. Therefore, in this step, at least one to-be-processed data constituting an object data file is determined based on the data size of each to-be-processed data in the to-be-processed data queue (i.e., the size of the space occupied by the data), wherein the total size of the space occupied by the at least one to-be-processed data constituting an object data file cannot be greater than the size of the space available for storing data in the object data file.

[0090] Exemplarily, the space size for storing data in an object data file is 3MB, so the data size of the data to be processed is determined sequentially from the queue of data to be processed. Assume that there are 10 data to be processed stored in the current queue of data to be processed, among which the data size of the 10th data to be processed is 1MB, the data size of the 9th data to be processed is 1MB, the data size of the 8th data to be processed is 2MB, and the data size of the 7th data to be processed is 0.5MB. Then, since the total data size of the 10th data to be processed and the 9th data to be processed is 2MB, which is smaller than the space size of 3MB for storing data in the object data file, and the total data size of the 10th data to be processed, the 9th data to be processed and the 8th data to be processed is 4MB, which is larger than 3MB, the total data size of the 10th data to be processed and the 9th data to be processed is 2MB, which is determined to be the data to be processed constituting an object data file.

[0091] In step 1023, aggregation processing is performed on at least one piece of data to be processed to generate an object data file.

[0092] The object data file includes at least the device identifier of a second data server storing the data to be processed and the storage location information of the data to be processed in the second data server. The second data server is used to store data whose access frequency is lower than a preset frequency threshold and whose importance value is lower than a preset importance threshold. In some embodiments, data to be processed for different businesses is stored in different second data servers. Therefore, when performing aggregation processing on at least one data to be processed, the business identifier of the data to be processed can be obtained, and then the device identifier of the second data server storing the data to be processed can be determined based on the business identifier of the data to be processed.

[0093] In some embodiments, at least one data to be processed can be aggregated by an aggregator. First, an aggregator instance is created and initialized, and then at least one data to be processed is passed to the processing module of the aggregator. After receiving at least one data to be processed, the aggregator processing module aggregates the data according to the aggregation logic to obtain an object data file.

[0094] It should be noted that the object data file includes a header area, an index area, a data area, and a trailer area. The header area is used to store attribute information of the object data file, including but not limited to: the file offset size of the index area, the file offset size of the data area, the file offset size of the trailer area, and the CRC32 checksum information of the object data file. The index area is used to store the storage location information of the data value to be processed on the second data server and the keyword to be processed. The storage information of the data value to be processed may include the data offset of the data value to be processed and the space occupied by the data value to be processed. In some embodiments, the index area is also used to store the hash value of the keyword to be processed, the second index number corresponding to the keyword to be processed, the write time, expiration time, and checksum information of the data to be processed. The data area is used to store the data value to be processed. The trailer area is used to store at least the device identification of the second data server. The trailer area may also store the start and end values ​​of the keyword stored in the object data file, as well as the device identification of the third data server acting as a slave replica, the file creation time, the file size, and other information.

[0095] In step 103, at least one object data file is stored in the second data server.

[0096] In some embodiments, the aggregator consumes the pending data in the pending data queue, and at least one object data file generated is added to a first-in-first-out multi-producer single-consumer chain cache queue, which is the cooling task queue in other embodiments. The cooling task queue can store object data files generated by aggregators from multiple different first data servers. The object data files in the cooling task queue can be consumed by the executor, that is, the executor stores the object data files in the cooling task queue to the second data server corresponding to the device identifier.

[0097] In some embodiments, as shown in FIG3C , after step 103 , data consistency may be further ensured through the following steps 201 to 204 , which will be described below in conjunction with FIG3C .

[0098] In step 201, an asynchronous lock is added to the keyword to be processed.

[0099] In some embodiments, an asynchronous lock can be added to the pending keyword through an asynchronous lock function. After the asynchronous lock is added, another process can be moved only after a ready-made run ends. In an embodiment of the present application, after the asynchronous lock is added to the pending keyword, the pending data value corresponding to the pending keyword can be read and written only after executing steps 202 to 204 or steps 202 to 104, thereby avoiding data inconsistency caused by changes in the pending data value after generating the object data file and before updating the pending data value to the location value corresponding to the storage location information.

[0100] In step 202, a first serial number of a keyword to be processed in a first data server is obtained, and a second serial number of a keyword to be processed in a second data server is obtained.

[0101] In an embodiment of the present application, a first data server stores a first sequence number corresponding to each keyword. The sequence number is globally unique and gradually increases with each write request. When an object data file is generated based on the data to be processed, the index area of ​​the object data file records the second index number of each keyword to be processed. This second index number is obtained from the first data server when the object data file is generated. If a write operation is performed on the data value to be processed corresponding to the keyword to be processed during the object data file generation process, the first index number of the keyword to be processed in the first data server will change. For example, assuming that the first index number corresponding to the keyword to be processed obtained from the first data server when the object data file is generated is 25, then the second index number of the keyword to be processed stored in the index area of ​​the generated object data file is 25. If two write operations are performed on the data value to be processed corresponding to the keyword to be processed during the object data file generation process, the first index number of the keyword to be processed in the first data server will change to 27.

[0102] In step 203, it is determined whether the first serial number is the same as the second serial number.

[0103] If the first sequence number and the second sequence number are the same, it indicates that the data value to be processed in the second data server is not outdated, and the process proceeds to step 104, where the data value to be processed corresponding to the keyword to be processed in the first data server is updated to the location value corresponding to the storage location information. If the first sequence number and the second sequence number are different, the process proceeds to step 204.

[0104] In step 204, the value of the data to be processed corresponding to the keyword to be processed in the first data server is kept unchanged.

[0105] When the first serial number is different from the second serial number, it means that the value of the to-be-processed data corresponding to the to-be-processed keyword in the first data server has changed during the process of processing the to-be-processed data, generating the object data file, and storing the object data file in the second data server. Since the value stored in the second data server is the value before the change, that is, the value stored in the second data server is outdated, the value of the to-be-processed data corresponding to the to-be-processed keyword in the first data server is no longer updated to the location value corresponding to the storage location information, that is, the value of the to-be-processed data corresponding to the to-be-processed keyword in the first data server remains unchanged. In addition, since the value of the to-be-processed data corresponding to the to-be-processed keyword in the first data server has changed during the process of processing the to-be-processed data, generating the object data file, and storing the object data file in the second data server, it means that the to-be-processed data has been accessed during this period. Therefore, it can be considered that the to-be-processed data is still hot data that will be accessed, and therefore the to-be-processed data will not be stored in the second data server for the time being.

[0106] Continuing to refer to FIG. 3A , the following description proceeds to step 103 .

[0107] In step 104, the to-be-processed data value corresponding to the to-be-processed keyword in the first data server is updated to the location value corresponding to the storage location information.

[0108] In some embodiments, the storage location information of the data value to be processed includes the file name of the object data file, the data offset of the data value to be processed in the object data file, the data space size of the data value to be processed, and the verification information of the data value to be processed. The first data server originally stores the keyword to be processed and the data value to be processed corresponding to the keyword to be processed. The data value to be processed is generally a string. Therefore, it is necessary to serialize the storage location information of the data value to be processed to obtain the location value corresponding to the storage location information, and update the data value to be processed corresponding to the keyword to be processed in the first data server to the location value corresponding to the storage location information. In this way, the first data server stores the key-value pairs of hot data and the location values ​​of the storage location information of the key and value of cold data. Among them, serialization processing refers to converting data in a specific format into a recoverable string sequence. In the embodiment of the present application, the storage location information is converted into a position value represented by a string sequence. In this way, not only can the serialized string location information be stored in a file or saved in a text field of a database, thereby realizing persistent data storage, but also compared with the original storage location information, the string location information can reduce the storage space occupied, and can also improve the readability, security, compatibility and scalability of the data.

[0109] In a data processing method provided in an embodiment of the present application, after obtaining a plurality of data to be processed that require cooling from a first data server, an object data file is generated based on the plurality of data to be processed. The object data file includes at least storage location information for the data to be processed and the data value to be processed in a second data server. The second data server is configured to store data with an access frequency lower than a preset frequency threshold and an importance value lower than a preset importance threshold, i.e., the second data server is configured to store data with a low access density and unimportant data. At least one object data file is stored in the second data server, and the data value to be processed corresponding to the keyword to be processed in the first data server is updated to the location value corresponding to the storage location information. In other words, the first data server stores data with high access density or important data (hot data), as well as keywords and location values ​​of data values ​​corresponding to the keywords for data with low access density and unimportant data (cold data). This ensures that the keywords and data values ​​of the cold data can also be read by the first data server, thereby ensuring availability and data reliability. In addition, because the space occupied by the location value of the data value corresponding to the keyword is much smaller than the space occupied by the data value itself, the space occupied by the cold data in the first data server can be reduced, thereby reducing storage costs. In addition, before updating the pending data value corresponding to the pending keyword in the first data server to the position value corresponding to the storage location information, data consistency is ensured by adding an asynchronous lock to the pending keyword and by using the first serial number of the pending keyword in the first data server and the second serial number in the second data server.

[0110] In some embodiments, space recovery of invalid data in the second data server can be performed in the following two ways: one is on-demand recovery when the fragmentation rate of the object data file reaches a preset fragmentation rate threshold; the other is scheduled inspection recovery when a preset scheduled recovery period is reached. The two data recovery methods are explained below.

[0111] Figure 4A is a schematic diagram of the implementation process of on-demand data recovery provided in an embodiment of the present application. As shown in Figure 4A, data recovery can be performed on demand through steps 301A to 304A shown in Figure 4A. The executor of each step shown in Figure 4A is the second data server, which is explained in detail below.

[0112] In step 301A, the total capacity and invalid data capacity of each target data file in the second data server are determined.

[0113] In some embodiments, steps 1A1 to 1A3 shown in FIG. 4B are executed for each object data file in the second data server to obtain the total capacity and invalid data capacity of each object data file, which will be described below in conjunction with FIG. 4B .

[0114] In step 1A1, the total capacity of the target data file is acquired from the attribute information of the target data file.

[0115] In some embodiments, a property value corresponding to the file size field may be acquired from the property information of the object data file, and the property value corresponding to the file size field may be determined as the total capacity of the object data file.

[0116] In step 1A2, each keyword included in the object data file is acquired, and an invalid keyword is determined from among the keywords.

[0117] If the storage system receives a data deletion instruction and deletes data from the first data server, the keyword and the data value corresponding to the keyword, or the position value of the data value corresponding to the keyword, in the first data server may be deleted. However, the data deleted from the first data server may still exist in the second data server. In this case, the data in the second data server is invalid. Based on this, when implementing step 1A2, each keyword included in the object data file is first searched on the first data server. If the keyword in the object data file does not exist on the first data server, it means that the keyword and the data value corresponding to the keyword have been deleted from the first data server. Therefore, the data corresponding to the keyword will no longer be accessed. Therefore, the keyword that does not exist on the first data server is determined to be an invalid keyword.

[0118] When the keyword included in the object data file also exists in the first data server, attribute information of the data value corresponding to the keyword is obtained. When the attribute information indicates that the data value corresponding to the keyword is stored in the first data server, then when reading and writing the data value corresponding to the keyword, there is no need to obtain the data value from the second data server. In other words, the data value of the keyword stored in the second data server is outdated data, and therefore the keyword is also determined to be an invalid keyword.

[0119] In step 1A3, the first number of invalid keywords included in the object data file is determined, and based on the first number and the size of a preset data value occupied space, the invalid data capacity of the object data file is determined.

[0120] In some embodiments, the size of the space occupied by the data value can be obtained from the index area of ​​the object data file, and the product of the first number and the size of the space occupied by the data value is determined as the invalid data capacity of the object data file.

[0121] In step 302A, the fragmentation rate of each target data file is determined based on the total capacity and invalid data capacity of each target data file.

[0122] In some embodiments, the invalid data capacity is divided by the total capacity to obtain the fragmentation rate of the object data file. For example, if the total capacity of an object data file is 10MB and the invalid data capacity is 6MB, then the fragmentation rate of the object data file is 60%.

[0123] In step 303A, the object data file whose fragmentation rate is greater than a preset fragmentation rate threshold is determined as a file to be recycled.

[0124] In some embodiments, the fragmentation rate threshold is a real number between 0 and 1. For example, the fragmentation rate threshold may be 0.4, i.e., 40%. When the fragmentation rate of the object data file is greater than the fragmentation rate threshold, it indicates that the object data file contains a large amount of invalid data and needs to be processed for fragment recovery. The object data file is then determined as a file to be recovered.

[0125] In step 304A, the files to be recycled are defragmented to obtain recycled files.

[0126] In some embodiments, when step 304A is implemented, the data value corresponding to the invalid keyword is first deleted from the data area of ​​the file to be recycled, and then the index information of the invalid keyword is deleted from the index area of ​​the file to be recycled to obtain the recycled file.

[0127] It should be noted that each keyword in the index area of ​​the object data file corresponds to an index information, which includes the hash value of the keyword, the second serial number of the keyword, the storage location information of the data value corresponding to the keyword, the data writing time, the data expiration time, the verification information, etc.

[0128] Through the above steps 301A to 304A, after first determining the fragmentation rate of each object data file, the to-be-reclaimed files whose fragmentation rate exceeds the fragmentation rate threshold are defragmented, that is, the index information of invalid keywords and the data values ​​corresponding to the invalid keywords in the to-be-reclaimed files are deleted to obtain the recovered files. In this way, invalid data can be cleaned up in a timely manner, thereby improving the validity of the data stored in the second data server. In addition, in some embodiments, after obtaining the recovered files, the recovered files can be verified. For example, according to a preset data verification algorithm, the to-be-verified information of each data value in the data area of ​​the recovered files can be generated. The to-be-verified information of each data value is then matched with the verification information of each data value in the index area. If the to-be-verified information of the data value is the same as the verification information of the data value in the index area, it means that the data value has passed the verification. If the to-be-verified information of the data value is different from the verification information of the data value in the index area, it means that the data value has failed the verification. In this case, the data value can be deleted from the data area, and the index information of the keyword corresponding to the data value can be deleted from the index area, thereby further increasing the available space in the second data server and improving data reliability.

[0129] FIG4C is a schematic diagram of the implementation process of periodically recovering data provided in an embodiment of the present application. As shown in FIG4C , data recovery can be performed regularly through steps 301B to 303B shown in FIG4C , which will be described in detail below.

[0130] In step 301B, when it is determined that the preset periodic debris recycling opportunity has been reached, invalid keywords are determined from the keywords included in each object data file in the second data server.

[0131] In some embodiments, the time period for performing debris recycling can be preset, for example, it can be set to 30 days or 14 days. When step 301B is implemented, the time of the last regular debris recycling is obtained, and based on the time of the last regular debris recycling and the preset time period, the time for this regular debris recycling is determined. When it is determined that the time for this regular debris recycling has arrived, it is determined that the timing of regular debris recycling has arrived, and at this time, invalid keywords are determined from each object data file in the second data server. First, each keyword included in each object data file is searched from the first data server, and keywords that do not exist in the first data server are determined as invalid keywords. When the keyword included in the object data file also exists in the first data server, the attribute information of the data value corresponding to the keyword is obtained. When the attribute information indicates that the data value corresponding to the keyword is stored in the first data server, the keyword is also determined to be an invalid keyword.

[0132] In step 302B, the data value corresponding to the invalid keyword is deleted from the data area of ​​the target data file.

[0133] In some embodiments, the offset information of the data value corresponding to the invalid keyword in the data area can be obtained from the index area in the object data file based on the invalid keyword, and the storage location of the data value can be determined from the data area based on the offset information, and then the data value stored in the storage location can be deleted.

[0134] In step 303B, the index information of the invalid keyword is deleted from the index area of ​​the target data file.

[0135] In some embodiments, since the data value corresponding to the invalid keyword has been deleted from the data area, the index information corresponding to the invalid keyword in the index area is also called invalid information. At this time, in order to further increase the effective space in the object data file, the index information of the invalid keyword is also deleted.

[0136] In the above steps 301B to 303B, when the preset periodic debris recovery time is reached, each object data file in the second data server is screened for invalid keywords, and the data values ​​and index information corresponding to the invalid keywords are deleted, that is, the entire data is subjected to debris recovery and defragmentation. In this way, even object data files that have not reached the fragmentation rate threshold are also subjected to debris recovery and defragmentation, freeing up the storage space occupied by invalid data in all object files, thereby maximizing the available space of the second data server. In some embodiments, after all object data files are defragmented and recovered, the data values ​​of valid keywords in all object data files can also be verified, and data that fails the verification can be deleted. This not only increases the available space of the second data server, but also ensures the reliability of the data.

[0137] Based on the aforementioned embodiments, the present application provides a data reading method. FIG5 is a schematic diagram of the implementation flow of the data reading method provided in the present application. The method will be described below in conjunction with FIG5 .

[0138] In step 401, a first data server receives a data read request sent by a terminal.

[0139] In some embodiments, in response to the received data read operation, the terminal sends a data read request to the first data server, wherein the data read request carries a to-be-read keyword of the data to be read.

[0140] In step 402, the first data server determines whether there is a keyword to be read stored.

[0141] If the first data server does not store the keyword to be read, the process proceeds to step 403 ; if the first data server does store the keyword to be read, the process proceeds to step 404 .

[0142] In step 403, the first data server sends a notification message of read failure to the terminal.

[0143] In step 404, the first data server obtains attribute information of the data value corresponding to the keyword to be read.

[0144] In some embodiments, the attribute information of the data value corresponding to the keyword to be read is used to represent that the data value corresponding to the keyword to be read is stored in the first data server, or is used to represent the location value of the storage location information of the data value corresponding to the keyword to be read is stored in the first data server.

[0145] In step 405, the first data server determines whether the attribute information represents that the data value stored in the first data server is itself.

[0146] Among them, if the attribute information represents that what is stored in the first data server is the data value to be read itself, go to step 406; if the attribute information represents that what is stored in the first data server is not the data value to be read itself, that is, what is stored in the first data server is the location value corresponding to the storage location information of the data value to be read in the second data server, go to step 407.

[0147] In step 406, the first data server sends the data value to be read to the terminal.

[0148] In step 407, the first data server deserializes the location value to obtain storage location information.

[0149] In some embodiments, since the first data server stores a location value obtained by serializing the storage location information, in this step, the location value needs to be deserialized to obtain the storage location information.

[0150] In step 408 , the first data server obtains the data value to be read from the second data server based on the storage location information.

[0151] Since the storage location information includes the file identifier of the object data file, the data offset of the data value to be processed, and the size of the space occupied by the data value to be processed, the first data server can send a data acquisition request to the second data server. The data acquisition request carries the storage location information of the data value to be read. After receiving the data acquisition request, the second data server obtains the data value to be read based on the storage location information and sends the data value to be read to the first data server.

[0152] In step 409 , the first data server sends the data value to be read to the terminal.

[0153] In the embodiment of the above steps 401 to 409, after receiving the read data request, if the first data server stores the data value to be read corresponding to the keyword to be read carried in the read data request, the first data server sends the data value to be read to the terminal; if the first data server stores the position value of the data value to be read corresponding to the keyword to be read, the position value is first deserialized to obtain the storage position information, and based on the storage position information, the data value to be read is obtained from the second data server, and the data value to be read is returned to the terminal. In this way, after receiving the read data request, the first data server can eventually return the data value to be read to the terminal regardless of whether the data value to be read is stored locally or not, thereby avoiding empty reading of data and improving data reading efficiency.

[0154] An embodiment of the present application provides a data writing method. FIG6 is a schematic diagram of the implementation flow of the data writing method provided in the embodiment of the present application, which will be described below in conjunction with FIG6 .

[0155] In step 501, a first data server receives a data write request sent by a terminal.

[0156] The write data request carries the data to be written, and the data to be written includes a keyword to be written and a data value to be written.

[0157] In step 502, the first data server writes the data to be written into the local storage space.

[0158] In step 503, the first data server sends the data to be written to the third data server.

[0159] In a distributed storage system, to improve disaster recovery capabilities, data on a first data server, acting as a primary replica, is backed up to at least one third data server, acting as a secondary replica. In other words, N is a positive integer. In practical applications, during implementation of step 503, the first data server sends the data to be written to each of the N third data servers, acting as secondary replicas. To improve the success rate of data writes and disaster recovery capabilities, N can be an integer greater than 1, such as 2 or 4. When N is 2, indicating that there are two third data servers acting as secondary replicas, the first data server and the two third data servers constitute a three-replica data service cluster.

[0160] In some embodiments, the first data server sends the data to be written to N third data servers serving as slave replicas, instructing the third data servers to back up the data to be written to their own local storage spaces. When the third data server successfully backs up the data to be written to its own local storage space, it sends a write success notification message to the first data server. When the third data server fails to successfully back up the data to be written to its own local storage space, it sends a write failure notification message to the first data server.

[0161] In step 504, the first data server receives a notification message of successful writing sent by the third data server.

[0162] In some embodiments, if the data to be written is successfully written to the third data server serving as a slave replica, the third data server will send a notification message of successful writing to the first data server.

[0163] In step 505 , the first data server determines whether the data to be written is successfully written into the local storage space and is successfully written to at least N / 2 third data servers serving as slave copies.

[0164] Among them, if the data to be written is successfully written into the local storage space of the first data server, and is successfully written into at least N / 2 third data servers serving as slave copies, it means that more than half of the data servers have successfully written the data to be written, and step 506 is entered at this time; if the data to be written is not successfully written into the local storage space, or is not successfully written into at least N / 2 third data servers serving as slave copies, it means that more than half of the data servers have not successfully written the data to be written, and step 507 is entered at this time.

[0165] In step 506, the first data server sends a notification message of successful writing to the terminal.

[0166] In step 507, the first data server sends a write failure notification message to the terminal.

[0167] In the embodiment of the above steps 501 to 507, after receiving the write data request, the first data server writes the data to be written into the local storage space and sends it to multiple third data servers serving as slave copies. Only when the data to be written is successfully written into the local storage space of the first data server and is successfully written into at least N / 2 third data servers serving as slave copies, is the data write successful determined, and a notification message of successful writing is sent to the terminal. This ensures that the data is stored in multiple copies in the storage system, thereby improving the disaster recovery capability of the storage system.

[0168] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0169] The data processing method provided in the embodiment of the present application can be applied to a data storage platform, for example, it can be applied to a database that supports multimodality, multi-tenancy, and storage and computing separation. The data processing method provided in the embodiment of the present application can be used to perform hot and cold screening and storage separation on the data, thereby saving storage costs and maximizing resource utilization.

[0170] FIG7 is another network structure diagram of the distributed storage system provided by an embodiment of the present application. As shown in FIG7 , a user terminal 701 writes data to the distributed storage system through a data write interface, and does not directly perceive the hierarchical storage format of hot and cold data. The distributed storage system stores the key and data value of hot data in a hot data storage cluster 702. A Blob file (corresponding to the object data file in other embodiments) is generated based on the data value of the cold data and stored in a cold data storage cluster 703. The key of the cold data and the location information of the key pointing to the cold data cluster are retained in the hot data storage cluster 702. The location information includes the Blob file name, the data offset and data size of the data value in the Blob file, and the verification information of the data value.

[0171] FIG8 is a schematic diagram of the structure of a Blob file provided in an embodiment of the present application. As shown in FIG8 , a Blob file includes four parts: a header area (Header) 801, an index area (Inode) 802, a data area (Data) 803, and a footer area (Footer) 804, wherein:

[0172] The header area 801 of the Blob file is used to store the attribute information of the Blob file, which occupies a total of 16 bytes. The attribute information includes the file offset size of the index area, the file offset size of the data area, the file offset size of the tail area and the CRC32 check information of this Blob file.

[0173] The index area 802 of the Blob file is used to store metadata of each data value dropped into the Blob file, including the hash value of the keyword, the offset position of the data value, the size of the data value, the timestamp and the CRC32 checksum information of the data value.

[0174] The data area 803 of the Blob file is used to store each actual data value dropped into the Blob file.

[0175] The tail area 804 of the Blob file is used to store the logical description information of the Blob file in the storage system, such as the replica ID, the hash range of the keyword stored in the replica, the cluster ID of the cold data cluster, and other information.

[0176] In the data processing method provided in the embodiment of the present application, a double-chain cache queue task model is used for data separation. Figure 9 is a schematic diagram of the double-chain cache queue provided in the embodiment of the present application. As shown in Figure 9, in this task model, two first-in-first-out multi-producer single-consumer chain cache queues are used: a record data queue 901 and a cooling task queue 902. The record data queue 901 is used as a data initial screening cache. During the Rocksdb compression (Compaction) process, a filter is used to filter out cold data that meets the cooling strategy conditions, and the cold data is stored in the record data queue 901. The data cached in the record data queue is then consumed by the aggregator (Aggregator) and formed into a Blob file of a specific data structure. The Blob file is then stored in the cooling task queue 902. The cooling Blob file is generated by the aggregator and written to the cold data cluster by the executor (Executor).

[0177] During the application of the data processing method provided in the embodiment of the present application, the frequency of cooling data screening can be controlled by adjusting the LSM tree size of Rocksdb and the cycle time for executing data compression in combination with business characteristics.

[0178] In the data processing method provided in the embodiment of the present application, space recovery of garbage data in the cold data cluster is provided, and a dual garbage recovery mechanism is provided, namely, on-demand recovery and scheduled inspection recovery. The storage system determines the fragmentation rate of the Blob file by recording the effective capacity and total capacity of the Blob file. The total capacity is the total size of the data area when the Blob file is cooled to the cold cluster, and the effective capacity of the Blob decreases as the business data is deleted. The fragmentation rate of the Blob file can be determined by the total capacity and effective capacity of the Blob file. The on-demand recovery mechanism sets a specific recovery threshold, and when the fragmentation rate reaches the recovery threshold, the Blob file is subjected to fragment recovery and file verification. The scheduled inspection recovery mechanism can regularly perform full inspections on Blob files, regularly recover fragmented files, perform file verification, and promptly correct capacity information.

[0179] In an embodiment of the present application, after cooling the data, the process of replacing the data value information of the cold data with the cold cluster location information is called cooling write-back. During the cooling write-back process, the data consistency of multiple copies needs to be ensured. The location information of the data value e is copied from the master copy to other slave copies through the Raft copy. During the write-back process, if the keyword has business replication, resulting in a change in the data value corresponding to the keyword, then the data value corresponding to the location information is different from the latest data value, causing the problem of data obsolescence. At this time, the data needs to be checked for consistency. In an embodiment of the present application, the data consistency check can be achieved through the following two steps:

[0180] First, the atomicity of the write-back process is guaranteed through asynchronous locks.

[0181] In the embodiment of the present application, the storage system routes write operations for the same keyword to the same thread based on the keyword's hash value. After the data is cooled to the cold data cluster, the storage system uses asynchronous locking to lock the cooled keyword to prevent conflicts with the same keyword.

[0182] Second, add data serial numbers to prevent duplication of outdated data by verifying the version number.

[0183] In an embodiment of the present application, the serial number (seq) of the data is filled in the index area of ​​each cooled record data. The serial number of the data is globally unique in the replica and gradually increases with the write request. In the process of cooling and writing back, the serial number of the locally stored record data and the serial number of the record data in the cooling index area are obtained, and the two serial numbers are compared. If the two serial numbers are the same, it means that the write-back location information is not outdated. At this time, the location information of the data value is written to the hot data storage cluster. If the two serial numbers are different, it means that the write-back location information is outdated. At this time, the outdated location information will be discarded, and the data garbage generated by the outdated location information in the cold storage cluster will be recycled with the garbage collection mechanism.

[0184] FIG10 is a schematic diagram of the data writing process according to an embodiment of the present application. As shown in FIG10 , the data writing process is divided into two parts: one part is the process 1001 of the user terminal writing data to the local RocksDB. The other part is the asynchronous data cooling process 1002 of the storage system, which is transparent to the user.

[0185] The process 1001 of the user terminal writing data to the local Rocksdb includes the following steps:

[0186] Step 10011, receiving a write request.

[0187] The write request carries key-value information, and is used to request a write operation to be performed on a data server serving as a primary copy in the storage system.

[0188] Step 10012, determine whether the writing is successful.

[0189] In this embodiment of the present application, the data server serving as the master replica replicates data to the other two data servers serving as slave replicas using the Raft protocol, and then writes to its own local RocksDB storage engine. If Raft fails to reach a quorum during the replication process or writing to the local RocksDB engine fails, the write is determined to be unsuccessful, and the process proceeds to step 10013. If Raft reaches a quorum during the replication process and writes to the local RocksDB engine succeeds, the write is determined to be successful, and the process proceeds to step 10014.

[0190] Step 10013, returns to the user that the write failed.

[0191] Step 10014, returns to the user that the writing is successful.

[0192] The asynchronous data cooling process includes the following steps:

[0193] Step 10021, determine whether the keyword-data value meets the cooling condition.

[0194] In some embodiments, during the execution of data compression, Rocksdb in the data server serving as the primary copy screens the keyword-data value through a filter to determine whether the keyword-data value meets the cooling condition. If the cooling condition is not met, step 10023 is entered; if the cooling condition is met, step 10022 is entered.

[0195] Step 10022: cool down the keyword-data value.

[0196] In some embodiments, the keyword-data value may be added to the record data queue, and the cooling process may be performed using the task model of the double-chain cache queue shown in FIG8 .

[0197] Step 10023, ignore the keyword-data value.

[0198] Ignore the keyword-data value, that is, do not perform cooling processing on the keyword-data value.

[0199] FIG11 is a schematic diagram of a data reading implementation process provided by an embodiment of the present application. As shown in FIG11 , the data reading process includes the following steps:

[0200] Step 1101: Receive a read request sent by a user terminal.

[0201] In some embodiments, in response to a received read operation, the user terminal initiates a read request to the storage system, where the read request carries a keyword for data to be read.

[0202] Step 1102: Determine whether the keyword can be read.

[0203] The storage system first determines the hash value of the keyword and routes it to the corresponding worker thread to read the local Rocksdb to determine whether the keyword can be read. If the keyword cannot be read, the process proceeds to step 1103. If the keyword can be read, the process proceeds to step 1104.

[0204] Step 1103: Return that the user data does not exist.

[0205] Step 1104 , determining whether the read data value is location information of the data value.

[0206] In some embodiments, attribute information of the read data value is obtained. If the attribute information represents that the data value is location information, step 1106 is entered; if the attribute information represents that the data value is the original data value, that is, the attribute information represents that the data value is not the location information of the data value, step 1105 is entered.

[0207] Step 1105: Return the data value read by the user.

[0208] Step 1106 : Read data from the cold data storage cluster according to the location information of the data value.

[0209] In some embodiments, the location information of the data value can be deserialized to obtain the cold storage cluster ID storing the data value, the Blob file name where the data value is located, and the offset and size of the data value of the keyword in the Blob file.

[0210] Step 1107, determine whether the reading is successful and the CRC is correct.

[0211] In some embodiments, based on the cold storage cluster ID storing the data value, the Blob file name where the data value is located, and the offset and size of the keyword data value in the Blob file, data is read from the cold storage cluster, and it is determined whether the read is successful and the CRC is correct. If the read is successful and the CRC is correct, go to step 1108; if the read fails or the CRC is wrong, go to step 1109.

[0212] Step 1108, return the data value.

[0213] Step 1109, return read error.

[0214] In an embodiment of the present application, during the Rocksdb data compression and compaction process, cold data that needs to be cooled is screened according to a custom cooling strategy. This allows the cooling strategy to be expanded according to user needs, improving the flexibility of cooling data. The cooling file metadata records the range of keywords routed to the data replica and the corresponding replica ID, which is used as replica metadata verification. In the face of replica splits and merges, the data of the cooled Blob file can be automatically adjusted and organized. In addition, multiple data replicas are previously only cooled by the primary replica. The location information after cooling is copied to other replicas via the Raft protocol, which can maintain the consistency of the cooled data of multiple replicas. In the data processing method provided in an embodiment of the present application, the keywords of cold data with low storage space consumption are retained in the hot storage, and the data values ​​of cold data with high storage space consumption are transferred to the cold storage cluster. In this way, the hot data storage cluster only stores the location information of keywords and data values ​​with small data volume in the cold data storage cluster, which can reduce data storage costs without affecting availability and data reliability. During the read access process of the business, there will be no idle reads, thereby improving data reading efficiency.

[0215] The following continues to describe an exemplary structure in which the data processing device 455 provided in an embodiment of the present application is implemented as a software module. In some embodiments, as shown in Figure 2, the software modules stored in the data processing device 455 of the memory 450 may include: a first acquisition module 4551, configured to acquire a plurality of data to be processed from a first data server, wherein the data to be processed include a keyword to be processed and a data value to be processed corresponding to the keyword to be processed; a file generation module 4552, configured to generate at least one object data file based on the plurality of data to be processed, wherein the object data file includes at least storage location information of the data to be processed and the data value to be processed in a second data server, wherein the second data server is used to store data having an access frequency lower than a preset frequency threshold and an importance value lower than a preset importance threshold; a first storage module 4553, configured to store the at least one object data file to the second data server; and a data update module 4554, configured to update the data value to be processed corresponding to the keyword to be processed in the first data server to the location value corresponding to the storage location information.

[0216] In some embodiments, the file generation module 4552 is further configured to: add the multiple data to be processed to a queue of data to be processed; determine at least one data to be processed that constitutes an object data file from the queue of data to be processed; perform aggregation processing on the at least one data to be processed to generate an object data file, the object data file including a header area, an index area, a data area and a tail area, the header area is used to store attribute information of the object data file, the index area is used to store storage location information of the data value to be processed on the second data server, the data area is used to store the data value to be processed, and the tail area is used to store at least the device identification of the second data server.

[0217] In some embodiments, the device also includes: an asynchronous lock module, configured to add an asynchronous lock to the to-be-processed keyword; a second acquisition module, configured to obtain the first serial number of the to-be-processed keyword in the first data server, and obtain the second serial number of the to-be-processed keyword in the second data server; a serial number judgment module, configured to judge whether the first serial number is the same as the second serial number, wherein when the first serial number is the same as the second serial number, the to-be-processed data value corresponding to the to-be-processed keyword in the first data server is updated to the position value corresponding to the storage location information.

[0218] In some embodiments, the device further includes: a first determination module, configured to obtain the total capacity and invalid data capacity of each object data file in the second data server; a second determination module, configured to determine the fragmentation rate of each object data file based on the total capacity and invalid data capacity of each object data file; a third determination module, configured to determine the object data file whose fragmentation rate is greater than a preset fragmentation rate threshold as a file to be recycled; and a defragmentation module, configured to defragment the file to be recycled to obtain a recycled file.

[0219] In some embodiments, the first determination module is further configured to: perform the following operations on each object data file in the second data server: obtain the total capacity of the object data file from the attribute information of the object data file; obtain each keyword included in the object data file, and determine invalid keywords from the each keyword; determine a first number of invalid keywords included in the object data file, and determine the invalid data capacity of the object data file based on the first number and a preset data value occupied space size.

[0220] In some embodiments, the first determination module is further configured to: search for each keyword included in the object data file from the first data server, and determine the keyword that does not exist in the first data server as an invalid keyword; when the keyword included in the object data file also exists in the first data server, obtain attribute information of the data value corresponding to the keyword; when the attribute information indicates that the data value corresponding to the keyword is stored in the first data server, determine the keyword as an invalid keyword.

[0221] In some embodiments, the defragmentation module is further configured to: delete the data value corresponding to the invalid keyword from the data area of ​​the file to be recycled; delete the index information of the invalid keyword from the index area of ​​the file to be recycled to obtain the recycled file.

[0222] In some embodiments, the device further includes: a third acquisition module, configured to acquire each keyword included in the object data file and determine an invalid keyword from the each keyword when it is determined that a preset periodic debris recovery time has been reached; a data value deletion module, configured to delete the data value corresponding to the invalid keyword from the data area of ​​the object data file; and an index deletion module, configured to delete the index information of the invalid keyword from the index area of ​​the object data file.

[0223] In some embodiments, the device also includes: a first receiving module, configured to receive a read data request sent by a terminal, wherein the read data request carries a to-be-read keyword of the data to be read; a fourth acquisition module, configured to obtain attribute information of the data value corresponding to the to-be-read keyword when the to-be-read keyword is stored in the first data server; a deserialization module, configured to deserialize the position value to obtain the storage position information when the attribute information represents that the first data server stores a position value corresponding to the storage position information of the to-be-read data value in the second data server; a fifth acquisition module, configured to obtain the to-be-read data value from the second data server based on the storage position information; and a first sending module, configured to send the to-be-read data value to the terminal.

[0224] In some embodiments, the apparatus further includes: a second sending module configured to send the data value to be read to the terminal when the attribute information indicates that the data value to be read is stored in the first data server.

[0225] In some embodiments, the device also includes: a second receiving module, configured to receive a write data request sent by a terminal, wherein the write data request carries the data to be written; a data writing module, configured to write the data to be written into the first data server, and copy the data to be written into N third data servers as slave copies, where N is a positive integer; a third sending module, configured to send a write success notification message to the terminal when it is determined that the data to be written is successfully written into the first data server, and is successfully written into at least N / 2 third data servers as slave copies.

[0226] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in the embodiment of the present application.

[0227] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the data processing method provided by an embodiment of the present application, for example, the data processing method shown in Figure 3A.

[0228] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0229] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0230] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, e.g., in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0231] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0232] To sum up, through the embodiments of the present application, cold data and hot data are stored in layers in the first data server used to store hot data, that is, the first data server stores key-value pairs of hot data, as well as keywords of cold data and location values ​​of storage location information of data values ​​in the second data server. This not only reduces storage costs while ensuring availability and data reliability, but also avoids empty reading of business operations when reading data, thereby improving data reading efficiency.

[0233] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data processing method, applied to a storage system, wherein the storage system comprises at least a first data server and a second data server, the method comprising: Acquire a plurality of to-be-processed data from the first data server, wherein the to-be-processed data include to-be-processed keywords and to-be-processed data values ​​corresponding to the to-be-processed keywords; Based on the plurality of data to be processed, at least one object data file is generated, wherein the object data file at least includes the data to be processed and storage location information of the data value to be processed in a second data server, wherein the second data server is used to store data whose access frequency is lower than a preset frequency threshold and whose importance value is lower than a preset importance threshold; storing the at least one object data file to the second data server; The to-be-processed data value corresponding to the to-be-processed keyword in the first data server is updated to the position value corresponding to the storage position information.

2. The method according to claim 1, wherein: The step of generating at least one object data file based on the plurality of data to be processed comprises: Adding the plurality of to-be-processed data to a to-be-processed data queue; Determine at least one to-be-processed data constituting an object data file from the to-be-processed data queue; Aggregation processing is performed on the at least one data to be processed to generate an object data file, wherein the object data file includes a header area, an index area, a data area and a tail area, wherein the header area is used to store attribute information of the object data file, the index area is used to store storage location information of the data to be processed on the second data server, the data area is used to store the value of the data to be processed, and the tail area is used to store at least a device identifier of the second data server.

3. The method according to claim 1, wherein: After storing the to-be-processed data in the second data server, the method further includes: Adding an asynchronous lock to the keyword to be processed; Obtaining a first serial number of the keyword to be processed in the first data server, and obtaining a second serial number of the keyword to be processed in the second data server; When the first serial number is the same as the second serial number, the to-be-processed data value corresponding to the to-be-processed keyword in the first data server is updated to the position value corresponding to the storage position information.

4. The method according to any one of claims 1 to 3, wherein: The method further comprises: determining a total capacity and an invalid data capacity of each object data file in the second data server; Determining the fragmentation rate of each object data file based on the total capacity and invalid data capacity of each object data file; Determine the object data file whose fragmentation rate is greater than a preset fragmentation rate threshold as a file to be recycled; The files to be recycled are defragmented to obtain recycled files.

5. The method according to claim 4, wherein: The determining of the total capacity and invalid capacity of each object data file in the second data server includes: The following operations are performed on each object data file in the second data server: Acquire the total capacity of the object data file from the attribute information of the object data file; Acquire each keyword included in the object data file, and determine invalid keywords from the each keyword; The first number of invalid keywords included in the object data file is determined, and based on the first number and a preset data value occupied space size, the invalid data capacity of the object data file is determined.

6. The method according to claim 5, wherein: The step of determining the invalid keywords from the keywords includes: searching the first data server for each keyword included in the object data file, and determining a keyword not existing in the first data server as an invalid keyword; When the keyword included in the object data file also exists in the first data server, acquiring attribute information of the data value corresponding to the keyword; When the attribute information indicates that what is stored in the first data server is the data value corresponding to the keyword, the keyword is determined to be an invalid keyword.

7. The method according to claim 5, wherein: The defragmenting the to-be-recycled files to obtain the recycled files includes: Deleting the data value corresponding to the invalid keyword from the data area of ​​the to-be-reclaimed file; The index information of the invalid keyword is deleted from the index area of ​​the file to be recycled to obtain the recycled file.

8. The method according to any one of claims 1 to 3, wherein: The method further comprises: When it is determined that the preset periodic debris recovery time has been reached, the following operations are performed on each object data file in the second data server: Acquire each keyword included in the object data file, and determine invalid keywords from the each keyword; Deleting the data value corresponding to the invalid keyword from the data area of ​​the object data file; The index information of the invalid keyword is deleted from the index area of ​​the object data file.

9. The method according to any one of claims 1 to 3, wherein: The method further comprises: A data read request sent by a receiving terminal, wherein the data read request carries a to-be-read keyword of the data to be read; When the keyword to be read is stored in the first data server, acquiring attribute information of the data value corresponding to the keyword to be read; When the attribute information indicates that what is stored in the first data server is a position value corresponding to the storage position information of the data value to be read in the second data server, deserializing the position value to obtain the storage position information; Based on the storage location information, obtaining the to-be-read data value from the second data server; The to-be-read data value is sent to the terminal.

10. The method according to claim 9, wherein: The method further comprises: When the attribute information indicates that what is stored in the first data server is the data value to be read, the data value to be read is sent to the terminal.

11. The method according to any one of claims 1 to 3, wherein: The method further comprises: A write data request sent by a receiving terminal, wherein the write data request carries data to be written; Writing the data to be written into the first data server, and copying the data to be written into N third data servers serving as slave copies, where N is a positive integer; When it is determined that the data to be written is successfully written into the first data server, and is successfully written into at least N / 2 third data servers serving as slave copies, a notification message of successful writing is sent to the terminal.

12. A data processing device, comprising: A first acquisition module is configured to acquire a plurality of to-be-processed data from a first data server, wherein the to-be-processed data includes to-be-processed keywords and to-be-processed data values ​​corresponding to the to-be-processed keywords; A file generation module is configured to generate at least one object data file based on the plurality of data to be processed, wherein the object data file includes at least the data to be processed and storage location information of the data to be processed in a second data server, wherein the second data server is used to store data whose access frequency is lower than a preset frequency threshold and whose importance value is lower than a preset importance threshold; a first storage module, configured to store the at least one object data file to the second data server; The data updating module is configured to update the to-be-processed data value corresponding to the to-be-processed keyword in the first data server to the position value corresponding to the storage position information.

13. A storage system, the storage system comprising at least: a first data server and a second data server, wherein the second data server is used to store data whose access frequency is lower than a preset frequency threshold and whose importance value is lower than a preset importance threshold; The first data server includes a first memory and a first processor, and the second data server includes a second memory and a second processor, the first memory and the second memory are used to store computer-executable instructions; when the first processor and the second processor are used to execute the computer-executable instructions, the method described in any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program, when executed by a processor, implement the method according to any one of claims 1 to 11.

15. A computer program product, comprising computer executable instructions or a computer program, wherein the computer executable instructions or the computer program, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images