Additional data writing method and device, storage medium and electronic equipment

Abstract

The invention relates to the technical field of data processing, in particular to an additional data writing method, an additional data writing device, a storage medium and electronic equipment. The additional data writing method comprises the steps that a first value in an sst file in an lsm storage engine is separated and stored into a blob file in advance, and a to-be-written space of a preset size is additionally arranged in the blob file to obtain a second value; storing a blob position index corresponding to the blob file in the first value so as to update the first value; in response to a writing request of value additional data, querying whether a target key corresponding to the value additional data exists in the lsm or not; and when the target key exists, writing the value additional data into the to-be-written space in the blob file corresponding to the target key. According to the additional data writing method provided by the invention, the updating efficiency of the long-sequence value additional data can be remarkably improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and in particular to an additional data writing method, an additional data writing device, a storage medium and an electronic device. Background Art

[0002] The storage engine is an online system that provides data storage services, allowing users to write and query data, and usually responds within milliseconds. When appending and updating data to a long sequence value in the LSM storage engine, the existing technology first reads the current value from the storage engine, appends new elements to the end of the value, and then writes the updated value back to the storage engine.

[0003] Although this method can complete the append update, it is very inefficient. First, for long sequence values, the number of elements in the sequence can reach tens of thousands or even more, and the number of bytes of the value can reach hundreds of KB or even more. Assuming that the current value length is 100KB and the length of the sequence to be added is 3KB, it is necessary to first read 100KB of original data and then write 103KB of new data. Although this process is simple, it involves a large amount of disk data reading and writing. A more serious problem is that due to the underlying design principle of LSM, although data is written very quickly, data merging (compaction) is required from time to time to clear old data, which will further increase the amount of data read and written to the disk. For high-performance scenarios such as the KV storage engine, frequent reading and writing of large value data will cause serious performance problems and affect the service capabilities of the storage engine.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a method for writing additional data, an apparatus for writing additional data, a storage medium and an electronic device, aiming to solve the problem of low update efficiency caused by frequent reading and writing of value additional data.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of an embodiment of the present disclosure, a method for writing additional data is provided, comprising: preliminarily separating and storing a first value in an SST file in an LSM storage engine into a blob file, and adding a preset size of a to-be-written space in the blob file to obtain a second value;

[0008] storing a blob position index corresponding to the blob file in the first value to update the first value;

[0009] In response to a write request for value additional data, querying in the lsm whether there is a target key corresponding to the value additional data;

[0010] When the target key exists, the value additional data is written into the to-be-written space in the blob file corresponding to the target key.

[0011] According to some embodiments of the present disclosure, based on the aforementioned solution, when the target key does not exist, the method further includes:

[0012] Create a key in the lsm file, write the value in the sst file and append data to obtain the first value;

[0013] The first value is stored separately in a blob file, and a space to be written that matches the sequence length of the additional data of the value is added to the blob file to obtain a second value.

[0014] According to some embodiments of the present disclosure, based on the above solution, before writing the value additional data into the to-be-written space, the method further includes:

[0015] Performing space verification and validity verification on the space to be written in the blob file;

[0016] When the space check and the validity check pass, writing the value additional data into the to-be-written space; and

[0017] When the space check and / or the validity check fails, the current value is read, and data is appended based on the value to update the current value.

[0018] According to some embodiments of the present disclosure, based on the above solution, performing space verification on the space to be written in the blob file includes:

[0019] Obtain the sequence length of the value additional data, and obtain the sequence length of the space to be written;

[0020] Check whether the sequence length of the space to be written meets the condition of the sequence length for storing the value additional data.

[0021] According to some embodiments of the present disclosure, based on the above solution, performing validity verification on the space to be written in the blob file includes:

[0022] Obtaining invalid value data of the LSM; wherein the invalid value data is generated after the LSM performs invalid data merging;

[0023] Check whether the blob file exists in the invalid value data.

[0024] According to some embodiments of the present disclosure, based on the aforementioned scheme, the first value includes version information, used space size, and blob location index; the second value includes value sequence data and space to be written.

[0025] According to some embodiments of the present disclosure, based on the above solution, before writing the value additional data into the to-be-written space, the method further includes:

[0026] Determine the first value according to the target key;

[0027] Determine that version information of the value additional data is higher than version information in the first value.

[0028] According to a second aspect of an embodiment of the present disclosure, there is provided an additional data writing device, comprising:

[0029] A storage module, used for pre-storing the first value in the sst file in the lsm storage engine into a blob file separately, and adding a preset size of a to-be-written space in the blob file to obtain a second value;

[0030] an updating module, configured to store a blob position index corresponding to the blob file in the first value to update the first value;

[0031] A query module, configured to query in the LSM whether there is a target key corresponding to the value additional data in response to a write request for the value additional data;

[0032] A writing module is used to write the value additional data into the to-be-written space in the blob file corresponding to the target key when the target key exists.

[0033] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for writing additional data as in the above embodiment is implemented.

[0034] According to the fourth aspect of the embodiments of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the additional data writing method as in the above-mentioned embodiments.

[0035] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0036] In the technical solutions provided by some embodiments of the present disclosure, the storage format is designed, and based on the value data kv separation mechanism, a portion of disk space is reserved for each value in the kv separated blob file in advance, and the sequence is directly written to the tail of the value in the blob file when it is appended and updated. In this way, when the value is appended and updated, the value data can be updated in situ and written to the reserved space to be written, which can save a lot of unnecessary disk reading and writing, improve the update efficiency, and thus significantly improve the service performance of the storage engine in this scenario.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0039] Figure 1 A schematic diagram schematically illustrates a flow chart of a method for writing additional data in an exemplary embodiment of the present disclosure;

[0040] Figure 2 A schematic diagram schematically shows a KV separation storage in an exemplary embodiment of the present disclosure;

[0041] Figure 3 A data flow chart for updating value and appending data in an exemplary embodiment of the present disclosure is schematically shown;

[0042] Figure 4 A schematic diagram schematically shows the composition of an additional data writing device in an exemplary embodiment of the present disclosure;

[0043] Figure 5 A schematic diagram schematically illustrates a computer-readable storage medium in an exemplary embodiment of the present disclosure;

[0044] Figure 6 A schematic diagram schematically shows the structure of a computer system of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0046] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0047] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0048] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0049] A storage engine is an online system that provides data storage services and allows users to write and query data, usually responding within milliseconds. A kv storage engine is a storage engine with a key-value storage format, where key is a keyword and value is a value. Since the storage space of a single kv storage engine is limited, data needs to be stored in multiple kv storage engines of the same type to expand storage capacity and service capabilities. This is a distributed kv storage engine.

[0050] LSM tree, or Log-Structured Merge Tree, is a specially designed solution for storing key-value data on disk. It significantly improves write performance by sacrificing some read performance. Many open source storage engines use LSM tree, such as RocksDB. The storage engine uses this solution, namely LSM storage engine.

[0051] A long sequence value refers to an ordered list of elements. A long sequence value is a value in the kv storage engine that is a long sequence. The number of elements in the sequence can reach tens of thousands or even more, and the length of the value can reach hundreds of KB or even longer.

[0052] Append update means adding one or more new elements to the end of an existing sequence, and updating the sequence content by increasing the length of the sequence. For example, if a sequence records the daily behavior data of an object, append update means adding all the behaviors of the new day (also a sequence) to the end of the original sequence. In this way, all the behavior data of the object in history can be found every time a query is made, and all elements in the result sequence queried at any time are sorted by time.

[0053] When appending updated data to a long sequence value in the LSM storage engine, the existing technology first reads the current value from the storage engine, appends a new element to the end of the value, and then writes the updated value back to the storage engine. Specifically, it can be divided into the following three steps:

[0054] ① Read value: Query the current value corresponding to the specified key from the storage engine.

[0055] ②Update value: append new elements (one or more) to the end of the read value sequence.

[0056] ③Write back value: Write the updated value back to the storage engine with a new key-value, overwriting the original value.

[0057] Although the existing technology can complete the additional update, the efficiency is very low.

[0058] First, for long sequence values, the number of sequence elements can reach tens of thousands or even more, and the number of bytes of the value can reach hundreds of KB or even more. Assuming that the current value length is 100KB and the length of the sequence to be added is 3KB, then you need to read 100KB of original data first, and then write 103KB of new data. Although this process is simple, it involves a large amount of disk data reading and writing.

[0059] A more serious problem is that, due to the underlying design principle of LSM, although data is written very quickly, data consolidation (compaction) is required from time to time to clear old data, which further increases the amount of data read and written to the disk. For high-performance scenarios such as the KV storage engine, frequent reading and writing of large value data will cause serious performance problems and affect the service capabilities of the storage engine.

[0060] Therefore, in view of the defects of the prior art, the present invention provides a method for writing additional data, which is different from the prior art. On the basis of adopting the kv separation mechanism, value data is updated in situ, which can save a large amount of unnecessary disk reading and writing, improve the update efficiency, and thus significantly improve the service performance of the storage engine in this scenario.

[0061] The implementation details of the technical solution of the embodiment of the present disclosure are elaborated in detail below.

[0062] Figure 1 The following is a schematic diagram showing a flow chart of a method for writing additional data in an exemplary embodiment of the present disclosure. Figure 1 As shown, the additional data writing method includes steps S101 to S103:

[0063] Step S101, pre-separating and storing the first value in the sst file in the lsm storage engine into a blob file, and adding a preset size of a to-be-written space in the blob file to obtain a second value;

[0064] Step S102, storing the blob position index corresponding to the blob file in the first value to update the first value;

[0065] Step S103, in response to a write request for value additional data, querying in the lsm whether there is a target key corresponding to the value additional data;

[0066] Step S104, when the target key exists, writing the value additional data into the to-be-written space in the blob file corresponding to the target key.

[0067] In the technical solutions provided by some embodiments of the present disclosure, the storage format is designed, and based on the value data kv separation mechanism, a portion of disk space is reserved for each value in the kv separated blob file in advance, and the sequence is directly written to the tail of the value in the blob file when it is appended and updated. In this way, when the value is appended and updated, the value data can be updated in situ and written to the reserved space to be written, which can save a lot of unnecessary disk reading and writing, improve the update efficiency, and thus significantly improve the service performance of the storage engine in this scenario.

[0068] Below, each step of the method for writing additional data in this exemplary implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0069] The additional data writing method disclosed in the present invention is based on KV separation, which is an optimization solution for the LSM storage engine. The principle of the KV separation solution is first introduced below.

[0070] LSM merges invalid data through the compaction mechanism and ensures that data at each level is in order. The compaction process will cause write amplification problems. For long values, the write amplification problem is more serious because the values ​​are frequently read and written. To address the write amplification problem of long values, the industry already has a KV separation solution for SSD storage, which stores the value separately in a blob file, and LSM only stores the value's location index on the disk.

[0071] KV separation also requires a GC mechanism for invalid data recovery. The GC method is to update the amount of data value in each blob file that is effectively referenced during each compaction. If the effective data ratio of a blob file is lower than a certain threshold, the effective data is rewritten to a new file and the original file is deleted.

[0072] For long values, KV separation will improve performance to varying degrees in both random data writing and batch data import scenarios, but at the cost of more disk space. Considering the current low cost of SSD disks, the KV separation solution is cost-effective. In addition, the value length threshold for KV separation to be effective is measured to be between 0.5KB and 0.7KB, so values ​​longer than 1KB can generally be stored separately.

[0073] The above is the reason for the KV separation mechanism. Based on the KV separation mechanism, we will now introduce how to efficiently implement the appending and updating of long sequence values ​​by updating value data in situ.

[0074] In step S101, the first value in the sst file in the lsm storage engine is separated and stored in a blob file in advance, and a preset size of space to be written is added to the blob file to obtain the second value.

[0075] Specifically, the LSM storage engine is optimized for storage, that is, the value is stored separately in the blob file, and at the same time, a part of the disk space is reserved for each value in the blob file after the KV is separated, that is, a preset size of space to be written is added, so as to update the first value to obtain the second value.

[0076] In step S102, a blob position index corresponding to the blob file is stored in the first value to update the first value.

[0077] Similar to the existing LSM storage engine optimized storage, after the value is stored separately in the blob file, LSM only stores the location index of the value on the disk.

[0078] First, define the protobuf format of the sequence value as follows. The version field is used to ensure idempotence and avoid repeated appending. For example, the update time of the data can be obtained. The elements field is used to save the list of elements in the sequence. There is no restriction on the field information contained in the elements. It only needs to be consistent in the rules when encoding and decoding protobuf. The storage engine will not pay attention to the meaning of the elements themselves.

[0079] In one embodiment of the present disclosure, the first value includes version information, used space size, and blob location index; the second value includes value sequence data and space to be written. Figure 2 A schematic diagram of KV separation storage in an exemplary embodiment of the present disclosure is schematically shown.

[0080] refer to Figure 2 As shown:

[0081] The storage format of the LSM engine includes the key-value in the SST file and the key-value in the blob file, which correspond to the storage formats before and after KV separation.

[0082] The first field of the value in the sst file is the version field, whose meaning has been introduced above. Then comes the used size field, which is used to record the actual space occupied by the sequence data. Finally, there are two cases: if the value is not separated, the sequence data is directly stored; if the value is separated, the index data of the value in the blob file is stored.

[0083] The first part of the value in the blob file is the real sequence data, with a length of unsed size, and the latter part is a reserved unused space. When appending and updating, the content to be appended is directly written to the corresponding unused position in the blob file by pwrite.

[0084] In step S103, in response to a write request for value additional data, it is queried in the lsm whether there is a target key corresponding to the value additional data.

[0085] When you need to write a value to append data, you first need to find out whether the appended value is pre-stored, that is, query the key list.

[0086] In step S104, when the target key exists, the value additional data is written into the to-be-written space in the blob file corresponding to the target key.

[0087] Specifically, if the target key exists, it can be directly written into the previously reserved additional space to be written.

[0088] In one embodiment of the present disclosure, when the target key does not exist, the method further includes: creating a key in the lsm, writing the value appended data in the sst file to obtain a first value; storing the first value separately in a blob file, and adding a space to be written in the blob file that matches the sequence length of the value appended data to obtain a second value.

[0089] That is to say, the operation of updating the value in place in the blob file needs to meet some prerequisites, such as the key already exists in the LSM. If the key is not found, the value is not stored in the LSM storage engine in advance. At this time, the value can be stored in the LSM first, and the data can be directly appended according to the value to create a corresponding size of the waiting writing space for writing.

[0090] Among them, the most critical detail is that when the long vlaue in the sst file is separated into the blob file, the original kv separation scheme does not reserve space, while this application reserves a corresponding size of space to be written, such as 0.5 times the length of the value. This is also the key to the in-place update of the value.

[0091] It should be noted that it is also possible to configure a fixed value (determined according to the service scenario of the storage engine) instead of calculating based on the current value length. Of course, this reserved length value can also be dynamically adjusted in the service, and you only need to provide an interface or mechanism for dynamically modifying the configuration.

[0092] Reserving space in proportion to the length of the current value and reserving space by configuring a fixed length value according to the storage scenario are both feasible methods for appending and updating a long sequence value of the present invention.

[0093] In one embodiment of the present disclosure, before writing the value appended data into the space to be written, the method also includes: performing space verification and validity verification on the space to be written in the blob file; when the space verification and the validity verification pass, writing the value appended data into the space to be written; and when the space verification and / or the validity verification fail, reading the current value and updating the current value based on the value appended data.

[0094] It should be noted that the in-place update operation of the value in the blob file needs to meet some prerequisites, such as (1) the space reserved after the value in the blob is sufficient for the additional update; and (2) the blob file is not being GCed.

[0095] Specifically, the space reserved after the value in the blob is sufficient for the additional update condition, that is, it is necessary to perform a space check on the space to be written in the blob file, specifically including: obtaining the sequence length of the value additional data, and obtaining the sequence length of the space to be written; and checking whether the sequence length of the space to be written meets the condition of storing the sequence length of the value additional data.

[0096] If the blob file is not in the GC, it is necessary to perform validity verification on the space to be written in the blob file, including: obtaining invalid value data of the lsm; wherein the invalid value data is generated after the lsm performs invalid data merging; and verifying whether the blob file exists in the invalid value data.

[0097] When the space check and / or the validity check fails, the current value is read, and the current value is updated based on the value additional data, that is, the value additional data is updated according to the existing re-reading.

[0098] In one embodiment of the present disclosure, the precondition for performing the in-place update operation of the value in the blob file is that the version of the new element is higher than the version in the original sequence. Therefore, in one embodiment of the present disclosure, before writing the value appended data into the to-be-written space, the method further includes: determining the first value according to the target key; and determining that the version information of the value appended data is higher than the version information in the first value.

[0099] Figure 3 A data flow chart for updating value and appending data in an exemplary embodiment of the present disclosure is schematically shown. Figure 3 As shown, when an append request for value additional data update is received, the key is queried from LSM. If the key does not exist, the new value is directly written to LSM (expanded by 1.5 times the length after KV separation). If the key exists, it is determined whether the version meets the conditions. If the version does not meet the requirements that the new element's version is higher than the version in the original sequence, then the data writing fails. If the version meets the conditions, it is determined whether the value has been stored separately. If it has not been stored separately, the original value is still read out according to the previous method and the sequence append is executed; if it has been stored separately, it is determined whether the blog is sufficient and whether it is not in GC. If the blog is not sufficient or in GC, the original value is still read out and the sequence append is executed; and if the blog is sufficient and not in GC, then append can be directly executed in the blog; then the new index is written to LSM.

[0100] The above is the complete technical solution of the present disclosure. After KV separation and in-place value update, new elements can be efficiently appended to the long sequence value of the LSM storage engine. The focus is on two parts, namely ① the design of the storage format, including the key-value format in SST and the key-value format in Blob; ② the process of appending and updating sequence values, including the processing methods in various situations when implementing appending and updating.

[0101] Based on the above method, the beneficial effect brought about is a significant improvement in the efficiency of appending and updating data of long sequence values. Compared with the method of reading and then writing the complete value, the efficiency of the appending data writing method disclosed in the present invention can be improved by about 10 times. The main reason is that the present invention does not need to read and write a large number of long sequence values, but only needs to process a small part of the value with insufficient reserved space. In most cases, it only needs to directly write incremental appended data.

[0102] The following is an example of a real scenario. The kv storage engine stores the daily behavior records of all users. The value is the behavior sequence of a single user. The space is about 200KB, and the newly generated behavior sequence of the day is added and updated each time.

[0103] If the read-before-write method is adopted, the amount of data updated each time is all users with new behavior sequences, and the amount of data read and written is about 2TB. In order to ensure the service performance of the storage engine, it is necessary to control the speed of appending and updating data, and the total time takes about 10 hours.

[0104] By adopting the solution disclosed in this disclosure, after KV separation, the value is added and updated in situ. In the same scenario, the amount of data read and written is about 5GB. Also, under the premise of ensuring the service performance of the storage engine, the total time is 1 hour, that is, the speed of adding and updating data is increased by 10 times.

[0105] Table 1 Comparison of the effects of the prior art solution and the disclosed solution

[0106]

[0107]

[0108] Figure 4 The following is a schematic diagram showing the composition of an additional data writing device in an exemplary embodiment of the present disclosure. Figure 4 As shown, the additional data writing device 400 may include a storage module 401, an update module 402, a query module 403 and a writing module 404. Among them:

[0109] According to an exemplary embodiment of the present disclosure, the write module 403 is also used to create a key in the lsm when the target key does not exist, write the value appended data in the sst file to obtain a first value; store the first value separately in a blob file, and add a to-be-written space in the blob file that matches the sequence length of the value appended data to obtain a second value.

[0110] According to an exemplary embodiment of the present disclosure, the writing module 403 is also used to perform space verification and validity verification on the space to be written in the blob file before writing the value appended data into the space to be written; when the space verification and the validity verification pass, the value appended data is written into the space to be written; and when the space verification and / or the validity verification fails, read the current value and update the current value based on the value appended data.

[0111] According to an exemplary embodiment of the present disclosure, the writing module 403 is also used to obtain the sequence length of the value appended data and the sequence length of the space to be written; and to check whether the sequence length of the space to be written meets the condition for storing the sequence length of the value appended data.

[0112] According to an exemplary embodiment of the present disclosure, the writing module 403 is also used to obtain invalid value data of the LSM; wherein the invalid value data is generated after the LSM performs invalid data merging; and verify whether the blob file exists in the invalid value data.

[0113] According to an exemplary embodiment of the present disclosure, the first value includes version information, used space size, and blob location index; the second value includes value sequence data and space to be written.

[0114] According to an exemplary embodiment of the present disclosure, the writing module 403 is also used to determine the first value according to the target key before writing the value appended data into the to-be-written space; and determine that the version information of the value appended data is higher than the version information in the first value.

[0115] The specific details of each module in the above-mentioned additional data writing device 400 have been described in detail in the corresponding additional data writing method, so they are not repeated here.

[0116] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0117] In an exemplary embodiment of the present disclosure, a storage medium capable of implementing the above method is also provided. Figure 5 A schematic diagram schematically illustrates a computer-readable storage medium in an exemplary embodiment of the present disclosure, such as Figure 5 As shown, a program product 500 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a mobile phone. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0118] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. Figure 6 A schematic diagram schematically shows the structure of a computer system of an electronic device in an exemplary embodiment of the present disclosure.

[0119] It should be noted that Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0120] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0121] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0122] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 609, and / or installed from a removable medium 611. When the computer program is executed by a central processing unit (CPU) 601, various functions defined in the system of the present disclosure are executed.

[0123] It should be noted that the computer-readable medium shown in the embodiment of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0124] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0126] As another aspect, the present disclosure further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0127] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0128] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0129] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0130] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for writing additional data, characterized in that: include: Separate and store the first value in the sst file in the lsm storage engine into a blob file in advance, and add a preset size of space to be written in the blob file to obtain the second value; storing a blob position index corresponding to the blob file in the first value to update the first value; In response to a write request for value additional data, querying in the lsm whether there is a target key corresponding to the value additional data; When the target key exists, the value additional data is written into the to-be-written space in the blob file corresponding to the target key.

2. The method for writing additional data according to claim 1, characterized in that: When the target key does not exist, the method further includes: Create a key in the lsm file, write the value in the sst file and append data to obtain the first value; The first value is stored separately in a blob file, and a space to be written that matches the sequence length of the additional data of the value is added to the blob file to obtain a second value.

3. The method for writing additional data according to claim 1, characterized in that: Before writing the value additional data into the to-be-written space, the method further includes: Performing space verification and validity verification on the space to be written in the blob file; When the space check and the validity check pass, writing the value additional data into the to-be-written space; and When the space check and / or the validity check fails, the current value is read, and data is appended based on the value to update the current value.

4. The method for writing additional data according to claim 3, characterized in that: Performing space verification on the space to be written in the blob file includes: Obtain the sequence length of the value additional data, and obtain the sequence length of the space to be written; Check whether the sequence length of the space to be written meets the condition of the sequence length for storing the value additional data.

5. The method for writing additional data according to claim 3, characterized in that: Performing a validity check on the space to be written in the blob file includes: Obtaining invalid value data of the LSM; wherein the invalid value data is generated after the LSM performs invalid data merging; Check whether the blob file exists in the invalid value data.

6. The method for writing additional data according to claim 1, characterized in that: The first value includes version information, used space size, and blob location index; the second value includes value sequence data and space to be written.

7. The method for writing additional data according to claim 1, characterized in that: Before writing the value additional data into the to-be-written space, the method further includes: Determine the first value according to the target key; Determine that version information of the value additional data is higher than version information in the first value.

8. An additional data writing device, characterized in that: include: A storage module, used for pre-storing the first value in the sst file in the lsm storage engine into a blob file separately, and adding a preset size of a to-be-written space in the blob file to obtain a second value; an updating module, configured to store a blob position index corresponding to the blob file in the first value to update the first value; A query module, configured to query in the LSM whether there is a target key corresponding to the value additional data in response to a write request for the value additional data; A writing module is used to write the value additional data into the to-be-written space in the blob file corresponding to the target key when the target key exists.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for writing additional data according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the additional data writing method as described in any one of claims 1 to 7.

Images