Data processing method and system

Abstract

The application discloses a data processing method and system, and belongs to the technical field of databases. The data processing method can obtain the data volume of received to-be-processed data, configure a corresponding storage mode according to the data volume, and write the to-be-processed data into a target file based on the configured storage mode, so that the persistent file storage position is no longer changed, the file quantity is no longer increased or decreased, different-capacity to-be-processed data can be supported, different storage modes are configured according to different magnitudes of data volume, the storage speed is improved, and the volume is small, light and handy.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to data processing methods and systems. Background Technology

[0002] Existing key-value repositories mainly include open-source key-value repositories (such as LevelDB, Berkeley DB, and LMDB), caching databases (such as Redis and Memcached), and repositories based on relational databases (such as MySQL and SQLite). However, the implementation of the above-mentioned repository functions generally relies on open-source libraries and is subject to the corresponding open-source licenses, posing significant risks. Furthermore, these repositories are large in size and difficult to deploy; in particular, repositories based on relational databases suffer from low read / write execution efficiency. Summary of the Invention

[0003] To address the issues of large size and low read / write efficiency in existing key-value storage systems, a data processing method and system designed to be compact, lightweight, and fast in read / write operations is provided.

[0004] This invention provides a data processing method, comprising:

[0005] Receive the data to be processed and obtain the data volume of the data to be processed;

[0006] Configure the corresponding storage method according to the amount of data to be processed, and write the data to be processed into the target file based on the configured storage method.

[0007] Preferably, a corresponding storage method is configured according to the amount of data to be processed, and the data to be processed is written to a target file based on the configured storage method, including:

[0008] When the amount of data to be processed is less than or equal to the first threshold, the data to be processed is analyzed to obtain first management data, second management data, index data and key-value data, and the first management data, second management data, index data and key-value data are written into the target file, where the target file is a single physical page;

[0009] The first threshold is the amount of data stored in a single physical page.

[0010] Preferably, a corresponding storage method is configured according to the amount of data to be processed, and the data to be processed is written to a target file based on the configured storage method, including:

[0011] When the amount of data to be processed is greater than the first threshold and less than or equal to the second threshold, the data to be processed is analyzed to obtain the first management data, the second management data, the index data, and the key-value data.

[0012] The first management data, the second management data, and the index data are mixed and written into a single physical page;

[0013] Identify whether the value in the key-value data is greater than a first threshold;

[0014] If not, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file. Each first data block corresponds to an address information. The target file includes the physical page and the first data block.

[0015] If so, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file; the value is paginated to generate a second data block, the value is written to the second data block, and the second data block is written to the target file. Each second data block corresponds to an address information. The target file includes the physical page, the first data block, and the second data block.

[0016] Preferably, a corresponding storage method is configured according to the amount of data to be processed, and the data to be processed is written to a target file based on the configured storage method, including:

[0017] When the amount of data to be processed is greater than the second threshold, the data to be processed is analyzed to obtain first management data, second management data, index data and key-value data. The first management data is written into the first management block, the second management data is written into the second management block, the second management block corresponds to an address information, and the index data is written into the index block, the index block corresponds to an address information.

[0018] Identify whether the value in the key-value data is greater than a first threshold;

[0019] If not, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file, the target file including the first data block, the first management block, the second management block and the index block;

[0020] If so, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file; the value is paginated to generate a second data block, the value is written to the second data block, and the second data block is written to the target file, wherein the target file includes the first data block, the second data block, the first management block, the second management block, and the index block.

[0021] Preferably, before configuring a corresponding storage method based on the data volume of the data to be processed, and writing the data to be processed into the target file based on the configured storage method, the method further includes:

[0022] Provide a redundant storage method;

[0023] Configure the corresponding storage method according to the amount of data to be processed, and write the data to be processed into the target file based on the configured storage method and the redundant storage method.

[0024] Preferably, the first management data record contains the address information of the second management data and the address information of the index data;

[0025] The second management data record includes the address information associated with the key-value pairs, as well as the corresponding redundant data addresses;

[0026] The index data records the address information of the key mapping.

[0027] Preferred options also include:

[0028] Receive read command;

[0029] According to the read instruction, query the first management data in the target file, and obtain the address information of the second management data and the address information of the index data from the first management data;

[0030] Based on the address information of the second management data and the address information of the index data, the corresponding key-value data is read.

[0031] Preferably, reading the corresponding key-value data based on the address information of the second management data and the address information of the index data includes:

[0032] Obtain the address information of the key mapping of the index data record;

[0033] Based on the address information to which the key belongs, query the key-value pair information in the second management data that corresponds to the address information;

[0034] Query the data block corresponding to the key-value pair information;

[0035] Read the value data from the data block and verify the value data;

[0036] If the verification passes, output the value data;

[0037] If the verification fails, the redundant data associated with the value data is verified based on the redundant data address. If the redundant data passes the verification, the redundant data replaces the value data that failed the verification, and the redundant data is output.

[0038] The present invention also provides a data processing system, comprising:

[0039] The receiving unit is used to receive the data to be processed and to obtain the data volume of the data to be processed.

[0040] The writing unit is used to configure a corresponding storage method according to the amount of data to be processed, and write the data to be processed into a target file based on the configured storage method.

[0041] Preferably, the receiving unit is further configured to receive a read instruction;

[0042] Also includes:

[0043] The query unit is used to query the first management data in the target file according to the read instruction, and obtain the address information of the second management data and the address information of the index data from the first management data;

[0044] The reading unit is used to read the corresponding key-value data based on the address information of the second management data and the address information of the index data.

[0045] The beneficial effects of the above technical solution are as follows:

[0046] In this technical solution, the data processing method of the present invention can obtain the data volume of the received data to be processed, configure the corresponding storage method according to the data volume, and then write the data to be processed into the target file based on the configured storage method. It can realize that the persistent file storage location will not change and the number of files will not increase or decrease. It can support data to be processed of different capacities and configure different storage methods according to different data volumes, thereby improving storage speed, and is small and lightweight. Attached Figure Description

[0047] Figure 1 This is a flowchart of one embodiment of the data processing method described in this invention;

[0048] Figure 2 This is a flowchart of a method for reading key-value data according to an embodiment of the present invention;

[0049] Figure 3 This is a block diagram of one embodiment of the data processing system described in this invention. Detailed Implementation

[0050] The advantages of the present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments.

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0052] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0053] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0054] In the description of this invention, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are performed, but are only used to facilitate the description of this invention and to distinguish each step, and therefore should not be construed as a limitation of this invention.

[0055] The data processing method and system of this application can be applied to fields such as automotive, finance, healthcare, and education. The data processing method can obtain the data volume of the received data to be processed, configure a corresponding storage method based on the data volume, and then write the data to be processed into a target file based on the configured storage method. This enables persistent file storage location to remain unchanged and the number of files to remain unchanged. It can support data of different capacities and configure different storage methods according to different data volumes, thereby improving storage speed, and is compact and lightweight.

[0056] This invention addresses the shortcomings of existing key-value storage systems, such as large size and low read / write efficiency, by providing a data processing method that is compact, lightweight, and offers fast read / write speeds. (See also...) Figure 1 This is a flowchart illustrating a data processing method according to a preferred embodiment of the present invention. As can be seen from the figure, the data processing method provided in this embodiment includes the following steps:

[0057] S1. Receive the data to be processed and obtain the data volume of the data to be processed;

[0058] S2. Configure the corresponding storage method according to the amount of data to be processed, and write the data to be processed into the target file based on the configured storage method.

[0059] In this embodiment, the target file is composed of several "pages". This is an example and not a limitation. The size of a page is 4096 bytes (4K). A 4K page will be referred to as a "physical page" in the following text.

[0060] Specifically, step S2 may include:

[0061] When the amount of data to be processed is less than or equal to the first threshold, the data to be processed is analyzed to obtain first management data, second management data, index data and key-value data, and the first management data, second management data, index data and key-value data are written into the target file, where the target file is a single physical page;

[0062] The first threshold is the amount of data stored in a single physical page (e.g., 4096 bytes, 4K). The following example uses a single physical page of 4K for illustration.

[0063] It should be noted that the first management data records the address information of the second management data and the address information of the index data; the second management data records the address information associated with the key-value pairs, as well as the corresponding redundant data addresses; and the index data records the address information of the key mapping.

[0064] Furthermore, step S2 may also include:

[0065] When the amount of data to be processed is greater than the first threshold and less than or equal to the second threshold, the data to be processed is analyzed to obtain the first management data, the second management data, the index data, and the key-value data.

[0066] The first management data, the second management data, and the index data are mixed and written into a single physical page;

[0067] Identify whether the value in the key-value data is greater than a first threshold;

[0068] If not, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file. Each first data block corresponds to an address information. The target file includes the physical page and the first data block.

[0069] If so, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file; the value is paginated to generate a second data block, the value is written to the second data block, and the second data block is written to the target file. Each second data block corresponds to an address information. The target file includes the physical page, the first data block, and the second data block.

[0070] Furthermore, step S2 may also include:

[0071] When the amount of data to be processed is greater than the second threshold, the data to be processed is analyzed to obtain first management data, second management data, index data and key-value data. The first management data is written into the first management block, the second management data is written into the second management block, the second management block corresponds to an address information, and the index data is written into the index block, the index block corresponds to an address information.

[0072] Identify whether the value in the key-value data is greater than a first threshold;

[0073] If not, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file, the target file including the first data block, the first management block, the second management block and the index block;

[0074] If so, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file; the value is paginated to generate a second data block, the value is written to the second data block, and the second data block is written to the target file, wherein the target file includes the first data block, the second data block, the first management block, the second management block, and the index block.

[0075] The first management block (Group management block): This block records information from several second management blocks (Boss management blocks). A 4K Group management block can manage a minimum of (4096-24-8-24-48) / 4 = 998 Boss pages, and a maximum of (4096-24-8-48) / 4 = 1004 Boss pages. Exceeding 1004 pages will trigger a Group management block expansion event.

[0076] The second management block is a free list of the internal management index block, the first data block, and the second data block. The maximum number is (4096-24-8-64-64) / 4 = 984, and the minimum number is 978.

[0077] Index block: Records the hash index of the key to the specific key storage page location.

[0078] The first data block stores several key-value pairs (key value, small value value, hash code, etc.). When the total length of the key and value exceeds a certain number, the key or value data will be transferred to the second data block (i.e., the value page) for storage.

[0079] The second data block stores value data; very large data sets may span multiple pages.

[0080] In actual data storage, initially, persistent storage files occupy only one "physical page" on the disk (default size 4K, which can be modified through configuration, but read / write efficiency is highest only when it matches the size of disk blocks and cached pages in the operating system's memory management). At this point, all the "data" (such as Group data, Boots data, Hash data, and KV storage data) of the "KV repository" in this embodiment is aggregated on this single physical page, called a "mixed page." Conversely, if a physical page is bound to only one single data type, it is called an "exclusive page." When all data types share a single physical page, the physical storage space can be described as "lightweight storage."

[0081] As key-value (KV) data pairs increase, when a single physical page cannot store all the data, the disk physical pages will expand. At this point, the key-value storage page will be preferentially separated from the mixed pages and become an exclusive page, binding to the newly added physical page. During this process, if a key-value data pair contains a large amount of value data (exceeding the default 4KB), a separate second data block (i.e., the value data page) will be added and bound to the new physical page. Before the second management block (Boss management page) is exclusively used, the physical storage space can be described as "medium-sized storage."

[0082] When the number of KV or Value data pages continues to increase, and the capacity of the Boss management data within the "hybrid page" becomes insufficient, a new physical page will be created to store the second management block (Boss management page). This means the Boss management page becomes an independent "exclusive page" from the "hybrid page." Simultaneously, a new physical page will be created to store the index block (Hash management page), meaning the "Hash management page" also becomes an independent "exclusive page" from the "hybrid page." When the KV storage page, Boss management page, and Hash management page all become independent "exclusive pages" from the "hybrid page," the original first data block (Group management page) within the "hybrid page" also successfully becomes an "exclusive page," but it remains bound to the first physical page. At this point, the physical storage space is between "medium-sized storage" and "heavy-duty storage."

[0083] As the KV data pairs continue to expand, new physical pages will be created to bind new KV storage pages, new value data pages, and new Boss management pages. The "Group management page" will manage all "Boss management pages" until the storage limit of the KV repository is triggered. At this point, the physical storage space corresponds to "heavy storage" in the diagram above.

[0084] A key-value pair contains key data, value data, and related storage structure. When a key-value pair exceeds 64 bytes, the value data is moved to a value data page for storage. When the capacity of a value data page needs to be expanded, a new physical page is created to bind the value data page. All key-value storage pages and value data pages are managed by the Boots management page.

[0085] In a preferred embodiment, the procedure before performing step S2 may further include:

[0086] A. Provide a redundant storage method;

[0087] S2. Configure the corresponding storage method according to the amount of data to be processed, and write the data to be processed into the target file based on the configured storage method and the redundant storage method.

[0088] As an example rather than a limitation, redundant storage can adopt an M / N redundancy configuration.

[0089] In a preferred embodiment, it may further include:

[0090] S3. Receive read command;

[0091] S4. Query the first management data in the target file according to the read instruction, and obtain the address information of the second management data and the address information of the index data from the first management data;

[0092] S5. Read the corresponding key-value data according to the address information of the second management data and the address information of the index data.

[0093] Specifically, see Figure 2 As shown, step S5 may include the following steps:

[0094] S51. Obtain the address information of the key mapping of the index data record;

[0095] S52. Based on the address information to which the key belongs, query the key-value pair information in the second management data that corresponds to the address information;

[0096] S53. Query the data block corresponding to the key based on the key-value pair information;

[0097] S54. Read the value data from the data block and verify the value data;

[0098] S55. If the verification passes, output the value data;

[0099] S56. If the verification fails, the redundant data associated with the value data is verified based on the redundant data address. If the redundant data passes the verification, the redundant data replaces the value data that failed the verification, and the redundant data is output.

[0100] In this embodiment, the data processing method can obtain the data volume of the received data to be processed, configure the corresponding storage method according to the data volume, and then write the data to be processed into the target file based on the configured storage method. This can achieve persistent file storage location that does not change and file number that does not increase or decrease. It can support data to be processed of different capacities and configure different storage methods according to different data volumes, thereby improving storage speed and making it small and lightweight.

[0101] The advantages of this embodiment are as follows: it provides application development with the ability to read and write KV data through static and dynamic linking; the code size is small, making it easy to integrate into embedded applications; in addition to basic data types, the Value class also supports custom data types; through M / N redundancy configuration, automatic recovery of persistent files after corruption can be achieved, ensuring data security; data security is ensured through encryption and permission management of KV storage files; by adding a caching layer, the read and write of KV library files is optimized, improving the read and write efficiency of KV data pairs; it supports single file sizes from 4K to 2G, and the optimized read and write algorithm ensures that the read time for large-capacity KV libraries is not significantly different from that for small-capacity KV libraries.

[0102] See Figure 3 As shown, this application also provides a data processing system 1, including: a receiving unit 11 and a writing unit 12.

[0103] Receiving unit 11 is used to receive data to be processed and to obtain the data volume of the data to be processed;

[0104] The writing unit 12 is used to configure a corresponding storage method according to the amount of data to be processed, and write the data to be processed into a target file based on the configured storage method.

[0105] In this embodiment, the target file is composed of several "pages". This is an example and not a limitation. The size of a page is 4096 bytes (4K). A 4K page will be referred to as a "physical page" in the following text.

[0106] Specifically, the writing unit 12 may include: when the amount of data to be processed is less than or equal to a first threshold, analyzing the data to be processed through the writing unit 12 to obtain first management data, second management data, index data and key value data, and writing the first management data, second management data, index data and key value data into the target file, wherein the target file is a single physical page;

[0107] It should be noted that the first management data records the address information of the second management data and the address information of the index data; the second management data records the address information associated with the key-value pairs, as well as the corresponding redundant data addresses; and the index data records the address information of the key mapping.

[0108] The further writing unit 12 may include: when the amount of data to be processed is greater than a first threshold and less than or equal to a second threshold, analyzing the data to be processed to obtain first management data, second management data, index data and key value data;

[0109] The first management data, the second management data, and the index data are mixed and written into a single physical page;

[0110] Identify whether the value in the key-value data is greater than a first threshold;

[0111] If not, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file. Each first data block corresponds to an address information. The target file includes the physical page and the first data block.

[0112] If so, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file; the value is paginated to generate a second data block, the value is written to the second data block, and the second data block is written to the target file. Each second data block corresponds to an address information. The target file includes the physical page, the first data block, and the second data block.

[0113] Furthermore, the writing unit 12 may include:

[0114] When the amount of data to be processed is greater than the second threshold, the data to be processed is analyzed to obtain first management data, second management data, index data and key-value data. The first management data is written into the first management block, the second management data is written into the second management block, the second management block corresponds to an address information, and the index data is written into the index block, the index block corresponds to an address information.

[0115] Identify whether the value in the key-value data is greater than a first threshold;

[0116] If not, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file, the target file including the first data block, the first management block, the second management block and the index block;

[0117] If so, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file; the value is paginated to generate a second data block, the value is written to the second data block, and the second data block is written to the target file, wherein the target file includes the first data block, the second data block, the first management block, the second management block, and the index block.

[0118] In a preferred embodiment, the data processing system 1 may include: providing a redundant storage method;

[0119] The writing unit 12 configures a corresponding storage method according to the amount of data to be processed, and writes the data to be processed into the target file based on the configured storage method and the redundant storage method.

[0120] As an example rather than a limitation, redundant storage can adopt an M / N redundancy configuration.

[0121] In a preferred embodiment, the receiving unit 11 can also be used to receive a read command;

[0122] The data processing system 1 may also include a query unit 13 and a reading unit 14.

[0123] The query unit 13 is used to query the first management data in the target file according to the read instruction, and obtain the address information of the second management data and the address information of the index data from the first management data;

[0124] The reading unit 14 is used to read the corresponding key-value data according to the address information of the second management data and the address information of the index data.

[0125] Specifically, the reading unit 14 may include: obtaining the address information of the key mapping of the index data record; querying the key-value pair information corresponding to the address information in the second management data based on the address information to which the key belongs; querying the data block corresponding to the key according to the key-value pair information; reading the value data in the data block and verifying the value data; if the verification passes, outputting the value data; if the verification fails, verifying the redundant data associated with the value data based on the redundant data address; if the redundant data passes the verification, replacing the value data that failed the verification with the redundant data and outputting the redundant data.

[0126] In this embodiment, the writing unit 12 in the data processing system 1 can obtain the data volume of the received data to be processed, configure the corresponding storage method according to the data volume, and then write the data to be processed into the target file based on the configured storage method. This can achieve persistent file storage location that does not change and file quantity that does not increase or decrease. It can support data to be processed of different capacities and configure different storage methods according to different data volumes, thereby improving storage speed and making it small and lightweight.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Receive the data to be processed and obtain the data volume of the data to be processed; Configure the corresponding storage method according to the amount of data to be processed, and write the data to be processed into the target file based on the configured storage method; Configure a corresponding storage method based on the amount of data to be processed, and write the data to be processed into a target file based on the configured storage method, including: When the amount of data to be processed is greater than the first threshold and less than or equal to the second threshold, the data to be processed is analyzed to obtain the first management data, the second management data, the index data, and the key-value data. The first management data, the second management data, and the index data are mixed and written into a single physical page; Identify whether the value in the key-value data is greater than a first threshold; If not, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file. Each first data block corresponds to an address information. The target file includes the physical page and the first data block. If so, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file; the value is paginated to generate a second data block, the value is written to the second data block, and the second data block is written to the target file. Each second data block corresponds to an address information. The target file includes the physical page, the first data block, and the second data block.

2. The data processing method according to claim 1, characterized in that, Configure a corresponding storage method based on the amount of data to be processed, and write the data to be processed into a target file based on the configured storage method, including: When the amount of data to be processed is less than or equal to the first threshold, the data to be processed is analyzed to obtain first management data, second management data, index data and key-value data, and the first management data, second management data, index data and key-value data are written into the target file, where the target file is a single physical page; The first threshold is the amount of data stored in a single physical page.

3. The data processing method according to claim 1, characterized in that, Configure a corresponding storage method based on the amount of data to be processed, and write the data to be processed into a target file based on the configured storage method, including: When the amount of data to be processed is greater than the second threshold, the data to be processed is analyzed to obtain first management data, second management data, index data and key-value data. The first management data is written into the first management block, the second management data is written into the second management block, the second management block corresponds to an address information, and the index data is written into the index block, the index block corresponds to an address information. Identify whether the value in the key-value data is greater than a first threshold; If not, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file, the target file including the first data block, the first management block, the second management block and the index block; If so, the key-value data is paginated to generate at least one first data block, and all the first data blocks are written to the target file; the value is paginated to generate a second data block, the value is written to the second data block, and the second data block is written to the target file, wherein the target file includes the first data block, the second data block, the first management block, the second management block, and the index block.

4. The data processing method according to claim 1, characterized in that, Before configuring the appropriate storage method based on the data volume of the data to be processed, and writing the data to be processed into the target file based on the configured storage method, the method further includes: Provide a redundant storage method; Configure the corresponding storage method according to the amount of data to be processed, and write the data to be processed into the target file based on the configured storage method and the redundant storage method.

5. The data processing method according to any one of claims 1-4, characterized in that, The first management data record contains the address information of the second management data and the address information of the index data; The second management data record includes the address information associated with the key-value pairs, as well as the corresponding redundant data addresses; The index data records the address information of the key mapping.

6. The data processing method according to claim 5, characterized in that, Also includes: Receive read command; According to the read instruction, query the first management data in the target file, and obtain the address information of the second management data and the address information of the index data from the first management data; Based on the address information of the second management data and the address information of the index data, the corresponding key-value data is read.

7. The data processing method according to claim 6, characterized in that, The step of reading the corresponding key-value data based on the address information of the second management data and the address information of the index data includes: Obtain the address information of the key mapping of the index data record; Based on the address information to which the key belongs, query the key-value pair information in the second management data that corresponds to the address information; Query the data block corresponding to the key-value pair information; Read the value data from the data block and verify the value data; If the verification passes, output the value data; If the verification fails, the redundant data associated with the value data is verified based on the redundant data address. If the redundant data passes the verification, the redundant data replaces the value data that failed the verification, and the redundant data is output.

8. A data processing system, characterized in that, The data processing method described in any one of claims 1-7 includes: The receiving unit is used to receive the data to be processed and to obtain the data volume of the data to be processed. The writing unit is used to configure a corresponding storage method according to the amount of data to be processed, and write the data to be processed into a target file based on the configured storage method.

9. The data processing system according to claim 8, characterized in that: The receiving unit is also used to receive read instructions; Also includes: The query unit is configured to query the first management data in the target file according to the read instruction, and obtain the address information of the second management data and the address information of the index data from the first management data; The reading unit is used to read the corresponding key-value data based on the address information of the second management data and the address information of the index data.

Images