Data writing method, device and server

Abstract

The present disclosure relates to a data writing method, device and server, the data writing method comprising: obtaining identification data of content to be written, determining the first hash code of the identification data of the content to be written in N first-level sets, N being a positive integer; reading data stored in a target set and obtaining a first reading result, the target set being the first-level set corresponding to the first hash code in the N first-level sets; and writing the identification data of the content to be written into the set associated with the first hash code according to the first reading result. This can reduce the amount of data read and improve the writing efficiency of the identification data of the content to be written.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data writing method, device and server. Background Art

[0002] In the scenario of retrieving similar articles from a database of hundreds of billions of articles, real-time, efficient data updates and fast retrieval are issues that analysts often care about. Since HBase (distributed database, Hadoop DateBase) is suitable for storing massive data, and collections of similar articles can be quickly retrieved based on row keys, the industry often uses HBase as a storage engine.

[0003] In the related technology, the data is generally read and written in a row with multiple columns. For example, the identification data of similar articles is stored as a data unit in different columns of the same row, and logically one row is a set. However, as the number of data units written increases, the number of data units in the set also increases. When the set needs to be retrieved and read, the Hbase engine layer will do a lot of data unit comparison work, which will affect the data processing efficiency. Summary of the invention

[0004] The present disclosure provides a data writing method, device and server to at least solve the problem of poor processing efficiency in the data reading and writing scheme of one row and multiple columns in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a data writing method is provided, comprising:

[0006] Obtain identification data of the content to be written, and determine a first hash code of the identification data of the content to be written in N first-level sets, where N is a positive integer;

[0007] Reading data stored in a target set and obtaining a first reading result, wherein the target set is a first-level set corresponding to the first hash code in the N first-level sets;

[0008] According to the first reading result, the identification data of the content to be written is written into a set associated with the first hash code.

[0009] Optionally, the step of writing the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result includes:

[0010] When the first reading result indicates that a first condition is satisfied, the identification data of the content to be written is written into the target set; wherein the first condition is that the storage amount of data in the target set is greater than zero and less than a preset storage amount.

[0011] Optionally, the step of writing the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result includes:

[0012] In the case where the first reading result indicates that a second condition is satisfied, splitting the target set into N secondary sets, and determining a second hash code of the identification data of the content to be written in the N secondary sets; wherein the second condition is that the storage amount of data in the target set is greater than or equal to a preset storage amount;

[0013] The identification data of the content to be written is written into a secondary set corresponding to the second hash code in the N secondary sets.

[0014] Optionally, the step of writing the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result includes:

[0015] When the first reading result indicates that a third condition is satisfied, obtaining metadata of the distributed database; wherein the third condition is that the storage amount of data in the target set is zero;

[0016] Based on the meta information, the identification data of the content to be written is written into a set associated with the first hash code.

[0017] Optionally, the step of writing the identification data of the content to be written into a set associated with the first hash code based on the meta information includes:

[0018] In a case where the meta information indicates that the target set has not been split, the identification data of the to-be-written content is written into the target set.

[0019] Optionally, the step of writing the identification data of the content to be written into a set associated with the first hash code based on the meta information includes:

[0020] In a case where the meta information indicates that the target set is split into N secondary sets, determining a third hash code of the identification data of the to-be-written content in the N secondary sets;

[0021] Reading content of a secondary set corresponding to the third hash code in the N secondary sets to obtain a second reading result;

[0022] According to the second reading result, the identification data of the content to be written is written into the set associated with the third hash code.

[0023] Optionally, the step of splitting the target set into N secondary sets includes:

[0024] Determine N column sets in the target row where the N primary sets are located, where the N column sets are N column sets in the target row except for the column sets corresponding to the N primary sets;

[0025] The target set is split into N secondary sets, and a mapping relationship between the N secondary sets and the N column sets is established.

[0026] According to a second aspect of an embodiment of the present disclosure, there is provided a data writing device, including:

[0027] A determination module is configured to obtain identification data of the content to be written, and determine a first hash code of the identification data of the content to be written in N first-level sets, where N is a positive integer;

[0028] a reading module configured to read data stored in a target set and obtain a first reading result, wherein the target set is a first-level set corresponding to the first hash code in the N first-level sets;

[0029] The writing module is configured to write the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result.

[0030] Optionally, the writing module is configured to write the identification data of the content to be written into the target set when the first reading result indicates that a first condition is satisfied; wherein the first condition is that the storage amount of data in the target set is greater than zero and less than a preset storage amount.

[0031] Optionally, the writing module includes:

[0032] a determining unit configured to execute, when the first reading result indicates that a second condition is satisfied, splitting the target set into N secondary sets and determining a second hash code of the identification data of the content to be written in the N secondary sets; wherein the second condition is that a storage amount of data in the target set is greater than or equal to a preset storage amount;

[0033] The first writing unit is configured to write the identification data of the to-be-written content into a secondary set corresponding to the second hash code in the N secondary sets.

[0034] Optionally, the writing module includes:

[0035] an acquisition unit, configured to acquire the meta information of the distributed database when the first reading result indicates that a third condition is satisfied; wherein the third condition is that the storage amount of the data in the target set is zero;

[0036] The second writing unit is configured to write the identification data of the to-be-written content into the set associated with the first hash code based on the meta information.

[0037] Optionally, the second writing unit is configured to write the identification data of the to-be-written content into the target set when the meta information indicates that the target set has not been split.

[0038] Optionally, the second writing unit includes:

[0039] A determination subunit is configured to determine, when the meta information indicates that the target set is split into N secondary sets, a third hash code of the identification data of the content to be written in the N secondary sets;

[0040] a reading subunit, configured to read the content of the secondary set corresponding to the third hash code in the N secondary sets to obtain a second reading result;

[0041] The writing subunit is configured to write the identification data of the to-be-written content into the set associated with the third hash code according to the second reading result.

[0042] Optionally, the determining unit includes:

[0043] a determining subunit, configured to determine N column sets in the target row where the N primary sets are located, the N column sets being N column sets in the target row except the column sets corresponding to the N primary sets;

[0044] The establishing subunit is configured to execute splitting of the target set into N secondary sets, and establish a mapping relationship between the N secondary sets and the N column sets.

[0045] According to a third aspect of an embodiment of the present disclosure, a server is provided, including:

[0046] processor;

[0047] a memory for storing instructions executable by the processor;

[0048] The processor is configured to execute the instructions to implement the data writing method described in any one of the first aspects.

[0049] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, when instructions in the storage medium are written by the data writing method described in any one of the first aspects.

[0050] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising: executable instructions, which, when executed on a computer, enable the computer to execute the data writing method described in any one of the first aspects.

[0051] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0052] By splitting the identification data set into N first-level sets, the identification data set is a set of identification data for storing the content to be written in the distributed database, and by determining the first hash code of the identification data of the content to be written in the N first-level sets, the target set corresponding to the first hash code in the N first-level sets is determined, that is, the target set to be written with the identification data of the content to be written is determined, so as to read the content stored in the target set and realize duplicate checking and reading of the identification data of the content to be written. Compared with directly performing duplicate checking and reading of the identification data of the content to be written in the identification data set, the amount of data read can be effectively reduced and the writing efficiency of the identification data of the content to be written can be improved.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0055] Figure 1 The figure is a flow chart showing a method for writing data according to an exemplary embodiment.

[0056] Figure 2 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0057] Figure 3 is a block diagram of a data writing device according to an exemplary embodiment.

[0058] Figure 4 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0060] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0061] Figure 1 is a flow chart of a data writing method according to an exemplary embodiment. Figure 1 ,This data writing method can be applied to a server, which can be used as a distributed database, Figure 1 The method shown includes the following steps:

[0062] Step 101: Obtain identification data of content to be written, and determine a first hash code of the identification data of the content to be written in N first-level sets.

[0063] In this step, the content to be written may be an article, a picture or the like, and the identification data may be an identification or mark of the article or the picture.

[0064] For example, in a usage scenario of retrieving similar articles from hundreds of billions of articles, the content to be written may be the article to be written, and the identification data may be an identification or mark of the article to be written.

[0065] Among them, the first-level set can be obtained by splitting the identification data set, and the identification data set is a set of identification data used to store the content to be written in the distributed database. In one embodiment, the identification data set can be split based on the bucketing technology to obtain the first-level set. For example, the identification data set can be split according to the number of N buckets to obtain N first-level sets.

[0066] The first hash code is used to determine the set corresponding to the identification data of the content to be written in N first-level sets; for example, if the value of the identification data of the content to be written is M, the first hash code of the identification data of the content to be written can be determined as the remainder of M / N. That is, the first hash code of the identification data of the content to be written in N first-level sets can be determined by taking the modulus.

[0067] For example, based on the bucketing technology, set A is split into four first-level sets, or set B is split into five first-level sets. After taking the modulus, the remainders of set A include 0, 1, 2, 3; the remainders of set B include 0, 1, 2, 3, 4; where the remainders are the hash codes corresponding to each first-level set, that is, the four first-level sets based on set A have hash codes of 0, 1, 2, 3, and the five first-level sets based on set B have hash codes of 0, 1, 2, 3, and 4.

[0068] For example, the identification data of the content to be written is numbered 8 in the N first-level sets, that is, the content to be written is the eighth content written in the N first-level sets. By taking the modulus, it can be determined that the remainder of the identification data of the content to be written in set A is 0, that is, the first hash code of the identification data of the content to be written in the four first-level sets in set A is 0; correspondingly, it can be determined that the remainder of the identification data of the content to be written in set B is 3, that is, the first hash code of the identification data of the content to be written in the five first-level sets in set B is 3.

[0069] Among them, when the four first-level sets included in set A are respectively the first-level set, the second-level set, the third-level set and the fourth-level set, and the hash code corresponding to the first-level set is set to 0, the hash code corresponding to the second-level set is set to 1, the hash code corresponding to the third-level set is set to 2, and the hash code corresponding to the fourth-level set is set to 3, it can be determined that the set corresponding to the identification data of the content to be written is the first-level set with a hash code of 0.

[0070] Step 102: read the data stored in the target set and obtain a first reading result.

[0071] In this step, the target set is a first-level set corresponding to the first hash code among the N first-level sets, that is, it can be a set to which identification data of the content to be written is to be written.

[0072] Among them, the data stored in the target set can be read to read the storage amount of the data in the target set, and can also read whether the target set stores identification data identical to the identification data of the content to be written, so as to avoid duplicate writing; that is, the first reading result at least includes the storage amount of the data in the target set and duplicate checking information, and the duplicate checking information is used to indicate whether the target set stores identification data identical to the identification data of the content to be written. Moreover, by reading the duplicate checking information, the problem of duplicate writing of the identification data of the content to be written can be avoided.

[0073] Step 103: Write the identification data of the content to be written into the set associated with the first hash code according to the first reading result.

[0074] In this step, the set corresponding to the identification data of the content to be written can be determined according to the first reading result, and written into the set corresponding to the identification data of the content to be written, thereby realizing the writing of the identification data.

[0075] For example, when the first reading result indicates that the storage amount of data in the target set is greater than zero and less than the preset storage amount, and the target set does not store identification data identical to the identification data of the content to be written, that is, it is the first time to write the identification data of the content to be written, then the identification data of the content to be written can be written into the target set, thereby realizing the writing of the identification data of the content to be written.

[0076] In the present disclosure, the identification data set is split into N first-level sets, and the first hash code of the identification data of the content to be written in the N first-level sets is determined to determine the target set corresponding to the first hash code, that is, the target set to be written with the identification data of the content to be written is determined, so as to read the content stored in the target set, and realize duplicate reading of the identification data of the content to be written. Compared with directly performing duplicate reading of the identification data of the content to be written in the identification data set, the amount of data reading can be effectively reduced, and the writing efficiency of the identification data of the content to be written can be improved.

[0077] Optionally, the step of writing the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result includes:

[0078] When the first reading result indicates that a first condition is satisfied, the identification data of the content to be written is written into the target set; wherein the first condition is that the storage amount of data in the target set is greater than zero and less than a preset storage amount.

[0079] In the present embodiment, when the first reading result indicates that the storage amount of data in the target set is greater than zero and less than the preset storage amount, it means that data can still be written into the target set; and the first reading result also indicates that no identification data identical to the identification data of the content to be written is stored in the target set, that is, it is the first time that the identification data of the content to be written is written, then the identification data of the content to be written can be written into the target set, thereby realizing the writing of the identification data of the content to be written.

[0080] Optionally, the step of writing the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result includes:

[0081] In the case where the first reading result indicates that a second condition is satisfied, splitting the target set into N secondary sets, and determining a second hash code of the identification data of the content to be written in the N secondary sets; wherein the second condition is that the storage amount of data in the target set is greater than or equal to a preset storage amount;

[0082] The identification data of the content to be written is written into a secondary set corresponding to the second hash code in the N secondary sets.

[0083] In this embodiment, when the first reading result indicates that the stored data in the target set is full, the target set is split into N secondary sets to increase the data storage capacity of the N secondary sets associated with the target set, thereby achieving dynamic expansion capability of data storage.

[0084] Among them, when the first reading result indicates that the storage capacity of data in the target set is greater than or equal to the preset storage capacity, the data stored in the target set is full, that is, no new data can be stored in the target set; and the first reading result also indicates that the target set does not store identification data identical to the identification data of the content to be written, that is, the first writing of the identification data of the content to be written; then the target set can be split into N secondary sets, and the second hash code of the identification data of the content to be written in the N secondary sets is determined, so as to write the identification data of the content to be written into the secondary set corresponding to the second hash code in the N secondary sets, thereby realizing the writing of the identification data of the content to be written.

[0085] For example, the data storage limit of the target set is 1MB, that is, 1024KB. When the size of the identification data of each content to be written is larger than 50KB, the preset storage capacity of the target set can be set to 1000KB. That is, when the storage capacity of the data in the target set is greater than or equal to 1000KB, it means that new data can no longer be stored in the target set.

[0086] Among them, in the process of splitting the target set into N secondary sets, a mapping relationship between the target set and the N secondary sets can be established, and the data storage capacity of each secondary set can also be set to 1MB, which is equivalent to mapping the target set with a storage upper limit of 1MB into N secondary sets with a storage upper limit of 1MB, thereby realizing the dynamic expansion capability of data storage.

[0087] Among them, in the process of determining the second hash code of the identification data of the content to be written in N secondary sets, the number of data already stored in the target set, the write number of each data, and the number of the identification data of the content to be written can be determined first; then based on the number of each data, the hash code of each data in the N secondary sets is determined, and it is written into the corresponding secondary set.

[0088] For example, if the number of data stored in the target set is 100, then the identification data of the content to be written can be determined to be numbered 101. When N is 5, the 100 data stored in the target set can be written into 5 secondary sets respectively, and the second hash code of the identification data of the content to be written in the 5 secondary sets can be determined, so as to write the identification data of the content to be written into the secondary set corresponding to the second hash code in the N secondary sets, thereby realizing the writing of the identification data of the content to be written.

[0089] Among them, the five secondary sets include the first secondary set, the second secondary set, the third secondary set, the fourth secondary set and the fifth secondary set, and the hash code corresponding to the first secondary set is set to 0, the hash code corresponding to the second secondary set is set to 1, the hash code corresponding to the third secondary set is set to 2, the hash code corresponding to the fourth secondary set is set to 3, and the hash code corresponding to the fifth secondary set is set to 4; it can be determined that the second hash code corresponding to the identification data of the content to be written numbered 101 is 0, that is, it can be determined that the set corresponding to the identification data of the content to be written is the first secondary set, that is, the first secondary set is the set to be written of the identification data of the content to be written, thereby realizing the determination of the set to be written of the identification data of the content to be written.

[0090] In addition, it should be noted that the N first-level sets can be the N column sets in the target row, that is, the N first-level sets can be different column sets in the target row, and the N column sets are the N column sets in the target row except the column sets corresponding to the N first-level sets; and by splitting the target set into N second-level sets, and establishing a mapping relationship between the N second-level sets and the N column sets, the target set is split into N second-level sets, that is, the splitting of the target set is realized, and the dynamic expansion capability of data storage is realized.

[0091] Among them, the N first-level sets and the N second-level sets are all column sets in the target row, that is, the N first-level sets and the N second-level sets are different columns of the same row. Through such a setting, the data processing efficiency of the distributed database can be improved, thereby improving the reading and writing speed of data.

[0092] Optionally, the step of writing the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result includes:

[0093] When the first reading result indicates that a third condition is satisfied, obtaining metadata of the distributed database; wherein the third condition is that the storage amount of data in the target set is zero;

[0094] Based on the meta information, the identification data of the content to be written is written into a set associated with the first hash code.

[0095] In this implementation, there are two situations in which the storage amount of data in the target set is zero. One is that no data has been stored in the target set, and the other is that the target set has been split, that is, the content of the target set is empty. In this case, the identification data of the content to be written can be written into the set associated with the first hash code by obtaining meta-information, such as the write record of the data.

[0096] In which, when the meta information indicates that the target set has not been split, the identification data of the content to be written can be directly written into the target set, thereby realizing the writing of the identification data of the content to be written.

[0097] Among them, when the meta information indicates that the target set is split into N secondary sets, the third hash code of the description data of the content to be written in the N secondary sets is determined; then the content of the secondary set corresponding to the third hash code in the N secondary sets is read to obtain a second reading result; and then, based on the second reading result, the identification data of the content to be written is written into the set associated with the third hash code, thereby realizing the writing of the identification data of the content to be written.

[0098] The second reading result at least includes the storage amount of data in the secondary set corresponding to the third hash code in the N secondary sets and duplicate checking information, and the duplicate checking information is used to indicate whether the secondary set corresponding to the third hash code in the N secondary sets stores identification data identical to the identification data of the content to be written. Moreover, by reading the duplicate checking information, the problem of duplicate writing of the identification data of the content to be written can be avoided.

[0099] In addition, when the identification data of the content to be written is written into the set associated with the third hash code based on the second reading result, the writing of the identification data of the content to be written can be realized based on the indication of the second reading result. For example, when the second reading result indicates that the storage amount of data in the secondary set corresponding to the third hash code in the N secondary sets is greater than zero and less than the preset storage amount, the identification data of the content to be written is written into the secondary set corresponding to the third hash code in the N secondary sets; when the second reading result indicates that the storage amount of data in the secondary set corresponding to the third hash code in the N secondary sets is greater than or equal to the preset storage amount, the secondary set corresponding to the third hash code in the N secondary sets is split to obtain N tertiary sets, and the fourth hash code of the identification data of the content to be written in the N tertiary sets is determined, and the identification data of the content to be written is written into the tertiary set corresponding to the fourth hash code in the N tertiary sets, so as to realize the writing of the identification data of the content to be written.

[0100] The data writing method provided by the embodiment of the present disclosure obtains the identification data of the content to be written, determines the first hash code of the identification data of the content to be written in N first-level sets, N is a positive integer; reads the data stored in the target set and obtains a first reading result, the target set is the first-level set corresponding to the first hash code in the N first-level sets; according to the first reading result, writes the identification data of the content to be written into the set associated with the first hash code. In this way, the amount of data read can be reduced and the writing efficiency of the identification data of the content to be written can be improved.

[0101] Figure 2 is a flow chart of a data processing method according to an exemplary embodiment. Figure 2 , the data processing method can be applied to the reading and writing of article content, and specifically includes the following steps:

[0102] Step 201: The HBase coprocessor calls an insert request to insert identification data of the article to be written into the target collection.

[0103] Step 202: The HBase coprocessor receives the insert request and calculates the hash code of the identification data according to the length of 16 buckets.

[0104] Step 203: The HBase coprocessor uses the hash code as the column name to query whether the corresponding sub-collection already exists.

[0105] Step 204: If the corresponding subset exists and the amount of identification data stored in it is less than its storage upper limit, the identification data of the article to be inserted is written into the subset.

[0106] Step 205: If the corresponding subset exists and the amount of identification data stored in it reaches its storage upper limit, the subset is subjected to secondary hash processing.

[0107] The logic of the secondary hashing process is as follows:

[0108] a) Parse the subset and perform secondary hashing on each article identification data according to the length of 16 buckets to obtain a new hash code;

[0109] b) Use the new hash code as the column name and re-insert the smaller subset after the secondary hashing into HBase;

[0110] c) Write the secondary hash meta information into the Meta table, indicating that the subset has been decomposed into secondary hash buckets;

[0111] d) Delete the data in the column corresponding to the hash code, that is, clean up the old data in the bucket corresponding to the hash code.

[0112] Step 206: If the corresponding sub-collection does not exist and the Meta table is not marked as having undergone secondary hashing, the identification data of the article to be inserted is directly written into the sub-collection.

[0113] Step 207: If the corresponding subset does not exist and the Meta table has been marked as having undergone secondary hashing, then obtain a new hash code for the identification data of the article to be inserted, and continue with step 203.

[0114] During the data query process, the HBase coprocessor reads the data in the entire target collection; then reads the sub-collections in all the columns of a row to achieve data reading.

[0115] In this way, in the scenario of storing similar articles, hashing can avoid the problem of poor reading and writing efficiency caused by the dynamic multi-column solution in one row. At the same time, the dynamic expansion capability of data can be guaranteed through secondary hashing technology without consuming additional space.

[0116] Figure 3 is a block diagram of a data writing device according to an exemplary embodiment. Figure 3 , the device 300 includes a determination module 301, a reading module 302 and a writing module 303, wherein:

[0117] The determination module 301 is configured to execute obtaining identification data of the content to be written, and determining a first hash code of the identification data of the content to be written in N first-level sets, where N is a positive integer;

[0118] A reading module 302 is configured to read data stored in a target set and obtain a first reading result, wherein the target set is a first-level set corresponding to the first hash code in the N first-level sets;

[0119] The writing module 303 is configured to write the identification data of the to-be-written content into the set associated with the first hash code according to the first reading result.

[0120] Optionally, the writing module 303 is configured to write the identification data of the content to be written into the target set when the first reading result indicates that a first condition is satisfied; wherein the first condition is that the storage amount of data in the target set is greater than zero and less than a preset storage amount.

[0121] Optionally, the writing module 303 includes:

[0122] a determining unit configured to execute, when the first reading result indicates that a second condition is satisfied, splitting the target set into N secondary sets and determining a second hash code of the identification data of the content to be written in the N secondary sets; wherein the second condition is that a storage amount of data in the target set is greater than or equal to a preset storage amount;

[0123] The first writing unit is configured to write the identification data of the to-be-written content into a secondary set corresponding to the second hash code in the N secondary sets.

[0124] Optionally, the writing module 303 includes:

[0125] an acquisition unit, configured to acquire the meta information of the distributed database when the first reading result indicates that a third condition is satisfied; wherein the third condition is that the storage amount of the data in the target set is zero;

[0126] The second writing unit is configured to write the identification data of the to-be-written content into the set associated with the first hash code based on the meta information.

[0127] Optionally, the second writing unit is configured to write the identification data of the to-be-written content into the target set when the meta information indicates that the target set has not been split.

[0128] Optionally, the second writing unit includes:

[0129] A determination subunit is configured to determine, when the meta information indicates that the target set is split into N secondary sets, a third hash code of the identification data of the content to be written in the N secondary sets;

[0130] a reading subunit, configured to read the content of the secondary set corresponding to the third hash code in the N secondary sets to obtain a second reading result;

[0131] The writing subunit is configured to write the identification data of the to-be-written content into the set associated with the third hash code according to the second reading result.

[0132] Optionally, the determining unit includes:

[0133] a determining subunit, configured to determine N column sets in the target row where the N primary sets are located, the N column sets being N column sets in the target row except the column sets corresponding to the N primary sets;

[0134] The establishing subunit is configured to execute splitting of the target set into N secondary sets, and establish a mapping relationship between the N secondary sets and the N column sets.

[0135] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0136] Figure 4 4 is a block diagram of a device 400 for writing data according to an exemplary embodiment. For example, the device 400 may be provided as a server. Figure 4 The apparatus 400 includes a processing component 422, which further includes one or more processors, and a memory resource represented by a memory 432 for storing instructions executable by the processing component 422, such as an application. The application stored in the memory 432 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 422 is configured to execute instructions to perform the above-mentioned method data writing method.

[0137] The device 400 may also include a power supply component 426 configured to perform power management of the device 400, a wired or wireless network interface 450 configured to connect the device 400 to a network, and an input / output (I / O) interface 458. The device 400 may operate based on an operating system stored in the memory 432, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0138] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0139] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data writing method, characterized in that: include: Obtain identification data of the content to be written, and determine a first hash code of the identification data of the content to be written in N first-level sets, where N is a positive integer; Reading data stored in a target set and obtaining a first reading result, wherein the target set is a first-level set corresponding to the first hash code in the N first-level sets; According to the first reading result, writing the identification data of the content to be written into the set associated with the first hash code; The step of writing the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result comprises: In the case where the first reading result indicates that a second condition is satisfied, splitting the target set into N secondary sets, and determining a second hash code of the identification data of the content to be written in the N secondary sets; wherein the second condition is that the storage amount of data in the target set is greater than or equal to a preset storage amount; The identification data of the content to be written is written into a secondary set corresponding to the second hash code in the N secondary sets.

2. The method according to claim 1, characterized in that The step of writing the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result comprises: When the first reading result indicates that a first condition is satisfied, the identification data of the content to be written is written into the target set; wherein the first condition is that the storage amount of data in the target set is greater than zero and less than a preset storage amount.

3. The method according to claim 1, characterized in that The step of writing the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result comprises: When the first reading result indicates that a third condition is satisfied, obtaining metadata of the distributed database; wherein the third condition is that the storage amount of data in the target set is zero; Based on the meta information, the identification data of the content to be written is written into a set associated with the first hash code.

4. The method according to claim 3, characterized in that The step of writing the identification data of the content to be written into the set associated with the first hash code based on the meta information comprises: In a case where the meta information indicates that the target set has not been split, the identification data of the to-be-written content is written into the target set.

5. The method according to claim 3, characterized in that: The step of writing the identification data of the content to be written into the set associated with the first hash code based on the meta information comprises: In a case where the meta information indicates that the target set is split into N secondary sets, determining a third hash code of the identification data of the to-be-written content in the N secondary sets; Reading content of a secondary set corresponding to the third hash code in the N secondary sets to obtain a second reading result; According to the second reading result, the identification data of the content to be written is written into the set associated with the third hash code.

6. The method according to claim 1, characterized in that The step of splitting the target set into N secondary sets includes: Determine N column sets in the target row where the N primary sets are located, where the N column sets are N column sets in the target row except for the column sets corresponding to the N primary sets; The target set is split into N secondary sets, and a mapping relationship between the N secondary sets and the N column sets is established.

7. A data writing device, characterized in that: include: A determination module is configured to obtain identification data of the content to be written, and determine a first hash code of the identification data of the content to be written in N first-level sets, where N is a positive integer; a reading module configured to read data stored in a target set and obtain a first reading result, wherein the target set is a first-level set corresponding to the first hash code in the N first-level sets; a writing module, configured to write the identification data of the to-be-written content into a set associated with the first hash code according to the first reading result; The writing module comprises: a determining unit configured to execute, when the first reading result indicates that a second condition is satisfied, splitting the target set into N secondary sets and determining a second hash code of the identification data of the content to be written in the N secondary sets; wherein the second condition is that a storage amount of data in the target set is greater than or equal to a preset storage amount; The first writing unit is configured to write the identification data of the to-be-written content into a secondary set corresponding to the second hash code in the N secondary sets.

8. The device according to claim 7, characterized in that The writing module is configured to write the identification data of the content to be written into the target set when the first reading result indicates that a first condition is satisfied; wherein the first condition is that the storage amount of data in the target set is greater than zero and less than a preset storage amount.

9. The device according to claim 7, characterized in that The writing module comprises: an acquisition unit, configured to acquire the meta information of the distributed database when the first reading result indicates that a third condition is satisfied; wherein the third condition is that the storage amount of the data in the target set is zero; The second writing unit is configured to write the identification data of the to-be-written content into the set associated with the first hash code based on the meta information.

10. The device according to claim 9, characterized in that The second writing unit is configured to write the identification data of the to-be-written content into the target set when the meta information indicates that the target set has not been split.

11. The device according to claim 9, characterized in that The second writing unit comprises: A determination subunit is configured to determine, when the meta information indicates that the target set is split into N secondary sets, a third hash code of the identification data of the content to be written in the N secondary sets; a reading subunit, configured to read the content of the secondary set corresponding to the third hash code in the N secondary sets to obtain a second reading result; The writing subunit is configured to write the identification data of the to-be-written content into the set associated with the third hash code according to the second reading result.

12. The device according to claim 7, characterized in that The determining unit comprises: a determining subunit, configured to determine N column sets in the target row where the N primary sets are located, the N column sets being N column sets in the target row except the column sets corresponding to the N primary sets; The establishing subunit is configured to execute splitting of the target set into N secondary sets, and establish a mapping relationship between the N secondary sets and the N column sets.

13. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data writing method according to any one of claims 1 to 6.

14. A storage medium, when instructions in the storage medium are executed by a processor of a server, enables the server to execute the data writing method according to any one of claims 1 to 6.

15. A computer program product comprising: An executable instruction, when the executable instruction is executed on a computer, enables the computer to execute the data writing method according to any one of claims 1 to 6.

Images