Data writing method, system, medium and computer device based on heap writing

Abstract

The application provides a data writing method, system, medium and computer device based on stack writing, which comprises generating an index file and a content file of to-be-written data; the index file is used to record index information of the to-be-written data; the content file is used to record other field information of the to-be-written data except the index information; and the index file and the content file of to-be-written data are stacked after the index file and the content file of previously-written data. The existing technology has the defect that when the data of a certain root node is wrong or crashes, all the data of the nodes under the root node will be wrong, which is a great trouble for effective data storage. The application stores data in a stack writing stacking mode, and splits files when the data reaches a certain amount, so that even if a power failure occurs suddenly, only the data being processed will be affected, and a large amount of data will not be affected, thereby realizing an efficient and safe data writing scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data writing, in particular to a data writing method and system based on heap writing, a medium and a computer device. BACKGROUND

[0002] With the advent of the big data era, the processing of massive data presents new challenges, such as the data insertion problem caused by huge data volume. In the existing commonly used data insertion method, each time a row of data is inserted, the data is reordered according to the id sequence, that is, each time a row of data is written, the entire file is erased and rewritten, which greatly reduces the service life of the hard disk. SUMMARY

[0003] In view of the above-mentioned shortcomings of the prior art, the present application aims to provide a data writing method and system based on heap writing, a medium and a computer device, to solve the technical problem of serious reduction of the service life of the hard disk caused by the data writing process in the prior art.

[0004] To achieve the above-mentioned purposes and other related purposes, the first aspect of the present application provides a data writing method based on heap writing, comprising: generating an index file and a content file of to-be-written data; the index file is used to record the index information of the to-be-written data; the content file is used to record other field information of the to-be-written data except the index information; and the index file and the content file of the data written later are superimposed after the index file and the content file of the data written earlier.

[0005] In some embodiments of the first aspect of the present application, the index information of the to-be-written data at least includes the following field information: id information, offset information, length information and check code information.

[0006] In some embodiments of the first aspect of the present application, the index file includes an ids file; and the content file includes a dat file.

[0007] In some embodiments of the first aspect of the present application, the method further comprises: if the cumulative storage capacity of the index file of the written data exceeds a first capacity threshold, or the cumulative storage capacity of the content file of the written data exceeds a second capacity threshold, splitting the storage file currently used to store the index file and the content file of the written data.

[0008] In some embodiments of the first aspect of the present application, the splitting of the storage file currently used to store the index file and the content file of the write data comprises: storing the id values of the write data in the memory, sorting the written data according to the id values, and selecting an id value in the middle of the sequence as a comparison object for subsequent id values of the to-be-written data, and storing the to-be-written data with id values less than the comparison object and the to-be-written data with id values greater than the comparison object in different files, respectively.

[0009] In some embodiments of the first aspect of the present application, before writing data, the id value of the to-be-written data is extracted and compared with the id values of the written data stored in the memory to eliminate duplicate data.

[0010] To achieve the above object and other related objects, the second aspect of the present application provides a data writing system based on heap writing, comprising: a file generation module for generating an index file and a content file of to-be-written data; the index file is used to record the index information of the to-be-written data; the content file is used to record other field information of the to-be-written data except the index information; a data writing module for superimposing the index file and the content file of the subsequent write data after the index file and the content file of the previous write data.

[0011] In some embodiments of the second aspect of the present application, the data writing system further comprises a file splitting module; the file splitting module is used to perform: if the cumulative storage capacity of the index file of the write data exceeds a first capacity threshold, or the cumulative storage capacity of the content file of the write data exceeds a second capacity threshold, the storage file currently used to store the index file and the content file of the write data is split.

[0012] To achieve the above object and other related objects, the third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the data writing method based on heap writing.

[0013] To achieve the above object and other related objects, the fourth aspect of the present application provides an electronic terminal, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to make the device execute the data writing method based on heap writing.

[0014] As described above, the data writing method, system, medium and computer device based on heap writing of the present application have the following beneficial effects: the prior art usually uses a tree structure to store data, for example, a B+ tree, which is usually used in databases and file systems of operating systems, and has the characteristics of keeping data stable and orderly, and has a relatively stable logarithmic time complexity for insertion and modification. The B+ tree elements are inserted from bottom to top. However, the disadvantages of storing data in a tree structure are also obvious, that is, when the data of a certain root node is wrong or crashes (for example, data crash caused by sudden power failure), all the data of the nodes under the root node will be problematic, which is a great trouble for effective data storage. Therefore, the present application uses a heap writing superposition method to store data, and performs file splitting when the data reaches a certain amount. During the splitting process, the id value is used to complete the quicksort, so even if a sudden power failure occurs, only the data currently being processed will be affected, and a large amount of data will not be affected, thereby realizing an efficient and safe data writing scheme. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A flowchart of a data writing method based on heap writing is shown as an embodiment of the present application.

[0016] Figure 2A A schematic diagram of ids files and dat files is shown as an embodiment of the present application.

[0017] Figure 2B A schematic diagram of a data writing process based on heap writing is shown as an embodiment of the present application.

[0018] Figure 3 A schematic diagram of file splitting is shown as an embodiment of the present application.

[0019] Figure 4 A structural diagram of a data writing system based on heap writing is shown as an embodiment of the present application.

[0020] Figure 5 A structural diagram of a computer device is shown as an embodiment of the present application. DETAILED DESCRIPTION

[0021] The embodiments of the present application will be described in detail below with specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure. The present application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0022] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, unless otherwise specifically indicated. Spatially relative terms, such as "upper," "lower," "left," "right," "below," "above," "bottom," "top," and the like, can be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be further understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientations depicted in the figures. For example, if a device described herein is turned over in use, a fixed surface can then be oriented upwards (beneath) that surface, rather than downward (above) it. The device can be oriented in any direction and terms such as "above" or "below" can be understood accordingly.

[0023] In the present application, unless specifically stated and limited otherwise, the terms "mounting", "connected", "connecting", "fixed", "fixedly", and the like, should be understood broadly, for example, can be fixedly connected, or detachably connected, or integrally connected; can be mechanically connected, or electrically connected; can be directly connected, or indirectly connected via an intermediate medium; can be internal communication between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0024] Further, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "first", "second", "third", "fourth" and the like in the description and in the claims, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in other sequences than described or otherwise illustrated herein. Furthermore, the terms "comprise", "comprising", "include", "including", and the like, mean the presence of stated features, operations, elements, components, items, and / or groups but do not preclude the presence or addition of one or more other features, operations, elements, components, items, and / or groups thereof. It is further understood that the use of relational terms, if any, such as "and / or", is intended to include the possibility of a combination of the named features, operations, elements, components, items, and / or groups thereof or the like. The terms "comprise", "comprising", "comprises" and the like can have the meaning ascribed to them in U.S. patent law and can mean "includes", "including" and the like; "comprising" has the meaning ascribed in U.S. patent law, i.e., it means "including, but not limited to", and the like; and "comprises" and "comprising", "includes" and "including" and the like have the meaning ascribed to them in U.S. patent law and are intended to be open-ended, meaning that they allow for the possibility that other features, operations, elements, components, items, and / or groups thereof can be added and still be within the scope of the application.

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the following embodiments and in conjunction with the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0026] As shown in Figure 1 , a flowchart of a heap-based data writing method in an embodiment of the present application is shown.

[0027] It should be noted that the above heap-based data writing method can be applied to a controller, such as an ARM (Advanced RISC Machines) controller, an FPGA (Field Programmable Gate Array) controller, a SoC (System on Chip) controller, a DSP (Digital Signal Processing) controller, or an MCU (Microcontroller Unit) controller, etc. It can also be applied to a computer device including a memory, a storage controller, one or more processing units (CPU), a peripheral interface, an RF circuit, an audio circuit, a speaker, a microphone, an input / output (I / O) subsystem, a display screen, other output or control devices, and an external port, etc. The computer device can be a desktop computer, a notebook computer, a tablet computer, a smart phone, a smart bracelet, a smart watch, a smart helmet, a smart television, etc. The heap-based data writing method can also be applied to a server, which can be arranged on one or more physical servers according to functions, loads, and other factors, or can be composed of a distributed or centralized server cluster.

[0028] In the present embodiment, the heap-based data writing method mainly includes steps S11 and S12, which will be explained in detail below.

[0029] Step S11: generating an index file and a content file of the data to be written; the index file is used to record index information of the data to be written; and the content file is used to record other field information of the data to be written except the index information.

[0030] In some examples, the index information of the data to be written includes, but is not limited to, the following field information: id information, offset information, length information, and checksum information, etc. The other field information except the index information is determined by the type defined by the user. If only the index is stored, there is no content file. If other fields are stored in addition to the index, these fields are stored in the content file.

[0031] For ease of understanding, the following Figure 2AThe description is as follows: the index file is an ids file, and the content file is a dat file; the index file records an id value, an offset, a length, and a check code of the data to be written, for example, the id value is 100, the offset is 0, and the check value is X1; the dat file records the data content of the data to be written, for example, x1, x2, … x 1000 Thus, the index file and the content file of the data to be written are generated.

[0032] In the example, each index file occupies a certain storage space. Taking the ids file as an example, the id information occupies 20 Bytes, the offset information occupies 3 Bytes, the length information occupies 2 Bytes, and the check value occupies 1 Bytes. Therefore, each index file occupies a total of 26 Bytes of storage space. It should be understood that the above example is provided for illustrative purposes and should not be understood as limiting.

[0033] Step S12: The index file and the content file of the data written later are superimposed after the index file and the content file of the data written earlier.

[0034] Specifically combined Figure 2B The description is as follows: the id value of the data 1 to be written is 100, the offset is 0, the length is 1000, and the check value is X1. The data is written into the permanent storage file. The id value of the data 2 to be written later is 101, the offset is 1000, the length is 1001, and the check value is X2. The data 2 is written into the permanent storage file after the data 1. The id value of the data 3 to be written later is 99, the offset is 2001, the length is 1002, and the check value is X3. The data 3 is written into the permanent storage file after the data 2. In this way, the data written later is superimposed after the data written earlier in a heap writing manner.

[0035] It should be noted that the data writing process in the embodiment discards the method of reordering and inserting according to the id value in the prior art. Taking the data 3 as an example, since the id value of the data 3 is 99, if the prior art scheme is adopted, the data 3 should be inserted before the data 1 according to the id order. In this way, the data in the file is always sorted according to the id. However, this technical scheme causes the data to be reordered once every time the data is inserted, and the entire file needs to be erased and written, which is very harmful to the service life of the hard disk. Therefore, the present application adopts the heap writing superimposition manner, that is, although the id value of the data 3 is 99, the data 3 is actually written after the data 2 in a heap writing manner. Only the order of writing the data is used to superimpose, and reordering is not performed. This avoids the damage to the hard disk caused by erasing and writing the entire file every time the data is written.

[0036] In some examples, the heap-based data writing method further comprises: if the cumulative storage capacity of the index file of the writing data exceeds a first capacity threshold, or the cumulative storage capacity of the content file of the writing data exceeds a second capacity threshold, splitting the storage file currently used to store the index file and the content file of the writing data. For example, as the writing data is continuously accumulated, the cumulative storage capacity of the ids file and the dat file continuously increases, and when the cumulative storage capacity of the ids file exceeds the first capacity threshold of 8M or the cumulative storage capacity of the dat file exceeds the second capacity threshold of 256M, the storage file currently used to store the index file and the content file of the writing data needs to be split. It should be noted that the embodiment does not limit the size of the first capacity threshold and the second capacity threshold, and the above example is only used for explanation and is not the only limitation of the threshold.

[0037] In some examples, the process of splitting the storage file currently used to store the index file and the content file of the writing data comprises: storing the id values of each writing data in the memory, sorting the written data according to the id values, selecting an id value located in the middle of the sequence as a comparison object for the id values of the subsequent writing data, and storing the writing data with id values less than the comparison object and the writing data with id values greater than the comparison object in different files respectively. It should be noted that if there are two id values located in the middle of the sequence, one can be arbitrarily selected as the comparison object.

[0038] Further, before writing data, the id values of the writing data are extracted and compared with the id values of each written data stored in the memory to ensure that the writing data is not repeated.

[0039] When the cumulative storage capacity of the ids file exceeds the first capacity threshold of 8M or the cumulative storage capacity of the dat file exceeds the second capacity threshold of 256M, assuming that there are 100 pieces of data stored at this time, the id values of these data are stored in the memory and sorted by id, that is, each piece of data originally stacked together by the heap writing method is quickly sorted according to its id value, and the middle id value is taken as the comparison object. The data with an id value less than the middle id value (including the ids file and the dat file) is stored in the first file (for example Figure 3 ids1 file and dat1 file), and the data with an id value greater than the middle id value (including the ids file and the dat file, each ids file and dat file forms a combined block) is stored in the second file (for example Figure 3 ids2 file and dat2 file), thereby completing data sorting while splitting the file.

[0040] It is worth mentioning that the prior art usually adopts a tree structure to store data, for example, a B+ tree, which is usually used in databases and file systems of operating systems, and is characterized by being able to keep data stable and ordered, and has a relatively stable logarithmic time complexity for insertion and modification. The B+ tree elements are inserted from bottom to top. However, the disadvantage of storing data in a tree structure is also obvious, that is, when the data of a certain root node is wrong or crashes (for example, data crash caused by sudden power failure), all the data of the nodes under the root node will be affected, which is a great trouble for effective data storage. In view of this, the present application adopts a stack storage mode based on heap writing, and performs file splitting when the data reaches a certain amount, and the id value is used to complete the quick sorting in the splitting process, so that even if a sudden power failure occurs, only the data currently being processed will be affected, and a large amount of data will not be affected, thereby realizing an efficient and safe data writing scheme.

[0041] As shown in Figure 4 The structure diagram of a data writing system based on heap writing in an embodiment of the present application is shown. The data writing system based on heap writing 400 comprises a file generation module 401 and a data writing module 402.

[0042] The file generation module 401 is configured to generate an index file and a content file of to-be-written data. The index file is used to record index information of the to-be-written data, and the content file is used to record other field information of the to-be-written data except the index information.

[0043] Specifically, the index information of the to-be-written data includes but is not limited to the following field information: id information, offset information, length information, and checksum information. The other field information except the index information is determined by the type defined by the user. If only the index is stored, there is no content file. If there are other fields in addition to the index, these fields are stored in the content file.

[0044] The data writing module 402 is configured to stack the index file and the content file of the to-be-written data after the index file and the content file of the previously-written data.

[0045] In some examples, the data writing system based on heap writing further comprises a file splitting module 403, which is configured to perform: if the cumulative storage capacity of the index file of the to-be-written data exceeds a first capacity threshold, or the cumulative storage capacity of the content file of the to-be-written data exceeds a second capacity threshold, splitting the storage file currently used to store the index file and the content file of the to-be-written data.

[0046] For example, as the cumulative storage capacity of the ids file and the dat file increases with the accumulation of the write data, when the cumulative storage capacity of the ids file exceeds a first capacity threshold of 8M or the cumulative storage capacity of the dat file exceeds a second capacity threshold of 256M, the storage file currently used to store the index file and the content file of the write data needs to be split. It should be noted that the embodiment does not limit the size of the first capacity threshold and the second capacity threshold, and the above example is only used for explanation and is not the only limit of the threshold.

[0047] In some examples, the process of splitting the storage file currently used to store the index file and the content file of the write data includes: storing the id values of the write data in the memory, sorting the written data according to the id values, selecting an id value in the middle of the sequence as a comparison object for subsequent id values of the to-be-written data, and storing the to-be-written data with an id value less than the comparison object and the to-be-written data with an id value greater than the comparison object in different files respectively. It should be noted that if there are two id values in the middle of the sequence, one can be arbitrarily selected as the comparison object.

[0048] Further, before writing data, the id value of the to-be-written data is extracted and compared with the id values of each written data stored in the memory to ensure that the data is not repeatedly written.

[0049] When the cumulative storage capacity of the ids file exceeds the first capacity threshold of 8M or the cumulative storage capacity of the dat file exceeds the second capacity threshold of 256M, assuming that there are 100 pieces of data stored at this time, the id values of these data are stored in the memory and sorted by id, that is, each piece of data originally stacked together by heap writing is quickly sorted according to its id value, and the middle id value is taken as the comparison object. The data with an id value less than the middle id value (including the ids file and the dat file) is stored in the first file (for example Figure 3 ids1 file and dat1 file), and the data with an id value greater than the middle id value (including the ids file and the dat file, each ids file and dat file forms a combined block) is stored in the second file (for example Figure 3 ids2 file and dat2 file in

[0050] It should be understood that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the data writing module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its function can be called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0051] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0052] like Figure 5 The diagram illustrates the structure of a computer device according to one embodiment of the present invention. The computer device provided in this embodiment includes: a processor 51, a memory 52, and a communicator 53; the memory 52 is connected to the processor 51 and the communicator 53 via a system bus and communicates with them; the memory 52 is used to store computer programs; the communicator 53 is used to communicate with other devices; and the processor 51 is used to run the computer program, enabling the electronic terminal to execute the various steps of the above-described data writing method based on stack writing.

[0053] The system bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as a client, a read-write library and a read-only library). The memory can include a Random Access Memory (RAM), and can also include a non-volatile memory, such as at least one disk memory.

[0054] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0055] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the data writing method based on the heap writing.

[0056] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by a computer program related hardware. The aforementioned computer program can be stored in a computer readable storage medium. The program is executed to perform the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes a ROM, a RAM, a magnetic disk or an optical disk and various storage medium capable of storing program codes.

[0057] In the embodiments provided in the present application, the computer readable and writable storage medium can include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a magnetic disk storage device or other magnetic storage device, a flash memory, a U disk, a mobile hard disk, or any other medium capable of storing desired program code in the form of instructions or data structures and capable of being accessed by a computer. In addition, any connection can be appropriately referred to as a computer readable medium. For example, if instructions are sent from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave is included in the definition of the medium. However, it should be understood that the computer readable and writable storage medium and the data storage medium do not include connections, carriers, signals or other transitory media, but are intended for non-transitory, tangible storage media. As used in the application, magnetic disks and optical disks include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks and Blu-ray discs, in which magnetic disks typically magnetically copy data, and optical disks optically copy data with a laser.

[0058] In summary, the present application provides a data writing method, system, medium and computer device based on heap writing. The prior art usually uses a tree structure to store data, such as a B+ tree, which is usually used in databases and file systems of operating systems, and has the characteristics of keeping data stable and ordered, and has a relatively stable logarithmic time complexity for insertion and modification. The B+ tree elements are inserted from bottom to top. However, the disadvantages of storing data in a tree structure are also obvious, that is, when the data of a certain root node is wrong or crashes (such as data crash caused by sudden power failure), all the data of the nodes under the root node will be problematic, which is a great trouble for effective data storage. Therefore, the present application uses a heap writing superposition method to store data, and performs file splitting when the data reaches a certain amount. During the splitting process, the id value is used to complete the quicksort, so even if a sudden power failure occurs, only the data being processed will be affected, and a large amount of data will not be affected, thereby realizing an efficient and safe data writing scheme. Therefore, the present application effectively overcomes the various disadvantages of the prior art and has high industrial utilization value.

[0059] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not intended to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical idea disclosed in the present application should be covered by the claims of the present application.

Claims

1. A data writing method based on heap writing, characterized in that, include: Generate an index file and a content file for the data to be written; the index file is used to record the index information of the data to be written. The content file is used to record field information of the data to be written, excluding index information; The index files and content files for which data is written later will be superimposed on the index files and content files for which data is written earlier. If the cumulative storage capacity of the index file containing the written data exceeds a first capacity threshold, or the cumulative storage capacity of the content file containing the written data exceeds a second capacity threshold, the storage file currently used to store the index file and content file containing the written data will be split.

2. The data writing method based on heap writing according to claim 1, characterized in that, The index information of the data to be written includes at least the following fields: id information, offset information, length information, and checksum information.

3. The data writing method based on heap writing according to claim 1, characterized in that, The index file includes an ids file; the content file includes a dat file.

4. The data writing method based on heap writing according to claim 1, characterized in that, The process of splitting the storage file currently used to store the index file and content file for the written data includes: storing the ID value of each written data in memory; sorting the written data according to the ID value; selecting the ID value in the middle of the sequence as the comparison object for comparison with the ID value of subsequent data to be written; and storing the data to be written with an ID value less than the comparison object and the data to be written with an ID value greater than the comparison object into different files respectively.

5. The data writing method based on heap writing according to claim 4, characterized in that, Before writing data, the ID value of the data to be written is extracted and compared with the ID values ​​of each piece of data that has already been written and stored in the memory to remove duplicate data.

6. A data writing system based on heap writing, characterized in that, include: The file generation module is used to generate an index file and a content file for the data to be written; the index file is used to record the index information of the data to be written. The content file is used to record field information of the data to be written, excluding index information; The data writing module is used to overlay the index files and content files on which data is written later onto the index files and content files on which data is written earlier. If the cumulative storage capacity of the index file containing the written data exceeds a first capacity threshold, or the cumulative storage capacity of the content file containing the written data exceeds a second capacity threshold, the storage file currently used to store the index file and content file containing the written data will be split.

7. The data writing system based on heap writing according to claim 6, characterized in that, The data writing system also includes a file splitting module; the file splitting module is used to perform the following: if the cumulative storage capacity of the index file for writing data exceeds a first capacity threshold, or the cumulative storage capacity of the content file for writing data exceeds a second capacity threshold, split the storage file currently used to store the index file and content file for writing data.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data writing method based on heap writing as described in any one of claims 1 to 5.

9. A computer device, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory to cause the device to perform the data writing method based on heap writing as described in any one of claims 1 to 5.

Images