A data storage method, device, apparatus and medium

Abstract

The application discloses a data storage method and device, equipment and medium, and relates to the field of data storage. The method comprises the following steps: dividing a disk into a metadata segment, a cache data segment and a real data segment according to a preset space division method; the metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment; when initial data is issued, the issued data is written into the real data segment, and the real metadata corresponding to the initial issued data in the real metadata segment is pointed to the real data segment; when data is rewritten, the rewritten data is written into the cache data segment, the corresponding shadow metadata is pointed to the original data segment corresponding to the rewritten data, and then the real metadata segment is pointed to the corresponding data segment in the cache data segment corresponding to the rewritten data and the data segment not rewritten in the original data segment. The application can optimize the double-write and multi-write problems of a single machine storage engine and improve the performance of the single machine storage engine.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data storage, in particular to a data storage method, device, equipment and medium. BACKGROUND

[0002] With the rapid development of computer technology, hardware devices have evolved from traditional magnetic tapes, to physical hard disks, to the current SSD (Solid State Disk or Solid State Drive), nvme (Non-Volatile Memory Express), pmem and DRAM (Dynamic Random Access Memory). The performance of hardware is getting better and better, and the performance of storage is no longer limited to the old reduction of disk addressing to accelerate the performance of storage engines. Single machine storage engines have evolved from bare disks in the past, to file systems in the future, to the current kv storage (Key-Value) engine, etc. In order to ensure data security and stability, data is generally first written into a log system, which is considered successful, and then gradually flushed into storage. However, such storage engines face double writing, i.e., data is written into the log system for the first time, and then written into the real data segment for the second time, while metadata needs to be updated, etc., resulting in double writing or even triple writing when the real IO (Input / Output) is landed. This further leads to a decline in the overall performance of the storage engine, which cannot fully utilize the performance of the basic storage device.

[0003] As can be seen from the above, in the data storage process, how to avoid the situation that the overall performance of the storage engine declines due to the possible double writing or triple writing in the storage, and the storage engine cannot fully utilize the performance of the basic storage device is a problem to be solved in the field. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a data storage method, device, equipment and medium, which can reduce data double writing problems while improving the read / write performance of large blocks of data, so that the single machine storage engine can reach the performance bottleneck of the disk and better utilize the performance of the new type of disk. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a data storage method, comprising:

[0006] dividing the disk into a metadata segment, a cache data segment and a real data segment according to a preset space division method; wherein the metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment;

[0007] When the initial data is issued, the initial issued data is written into the real data section, and the real metadata corresponding to the initial issued data in the real metadata section is pointed to the real data section;

[0008] When the data is overwritten, the overwritten data is written into the cache data section, the corresponding shadow metadata is pointed to the original data section corresponding to the overwritten data, and then the real metadata section is pointed to the corresponding data section in the cache data section corresponding to the overwritten data and the data section not overwritten in the original data section.

[0009] Optionally, the disk is divided into the metadata section, the cache data section and the real data section according to the preset space division method, comprising:

[0010] The space capacity of the metadata section and the cache data section is determined according to the preset space division method;

[0011] The space capacity of the disk except the space capacity of the metadata section and the cache data section is taken as the space capacity of the real data section.

[0012] Optionally, when the data is overwritten, the overwritten data is written into the cache data section, comprising:

[0013] According to a preset data judgment rule, the current overwritten data is determined to be large block data or small block data;

[0014] If the current overwritten data is large block data, the target fragment data in the shadow data section is written into the fragment management data section, the corresponding shadow metadata is pointed to the overwritten data, and then the overwritten read-write data is sequentially written into the shadow data section;

[0015] If the current overwritten data is small block data, the overwritten data is written into the fragment management data section in the cache data section.

[0016] Optionally, the overwritten data is written into the cache data section, comprising:

[0017] According to the data size of the overwritten data, a target data section is applied for the cache data section;

[0018] The overwritten data is written into the target data section.

[0019] Optionally, the overwritten read-write data is sequentially written into the shadow data section, comprising:

[0020] According to the data size of the overwritten data, a target shadow data section is applied for the shadow data section, and the overwritten data is written into the target shadow data section;

[0021] Correspondingly, the writing of the copy data into the fragment management data segment in the cache data segment comprises:

[0022] Based on the data size of the copy data, a target small block data segment is applied for from the fragment management data segment, and the copy data is written into the target small block data segment.

[0023] Optionally, the writing of the target fragment data in the shadow data segment into the fragment management data segment comprises:

[0024] Determining a current minimum backup amount;

[0025] Intelligently evaluating data segments in the cache data segment by using a preset intelligent evaluation method to determine target fragment data from the cache data segment;

[0026] Based on the minimum backup amount, the target fragment data in the shadow data segment is written into the fragment management data segment.

[0027] Optionally, the data storage method further comprises:

[0028] Determining invalid data segments in the cache data segment by using a preset resource adjustment recovery method, and recovering the invalid data segments;

[0029] Determining invalid metadata segments corresponding to the invalid data segments, and updating metadata recorded in the invalid metadata segments to be empty.

[0030] In a second aspect, the application discloses a data storage device, comprising:

[0031] A space division module is configured to divide a disk into metadata segments, cache data segments and real data segments according to a preset space division method; wherein the metadata segments are divided into real metadata segments and shadow metadata segments, and the cache data segments are divided into shadow data segments and fragment management data segments;

[0032] A first data delivery module is configured to write first-delivered data into the real data segments and point real metadata corresponding to the first-delivered data in the real metadata segments to the real data segments when the first data is delivered;

[0033] A data copy module is configured to write copy data into the cache data segments and point corresponding shadow metadata to original data segments corresponding to the copy data when data is copied, and then point real metadata segments to corresponding data segments in the cache data segments corresponding to the copy data and data segments in the original data segments that are not copied.

[0034] In a third aspect, the application discloses an electronic device, comprising:

[0035] a memory for storing the computer program;

[0036] a processor for executing the computer program to implement the data storage method.

[0037] In a fourth aspect, the present application discloses a computer storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the steps of the data storage method disclosed above.

[0038] In the present application, the disk is first divided into a metadata segment, a cache data segment and a real data segment according to a preset space division method; wherein the metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment. When the initial data is issued, the initial issued data is written into the real data segment, and the real metadata corresponding to the initial issued data in the real metadata segment is pointed to the real data segment. When the data is rewritten, the rewritten data is written into the cache data segment, and the corresponding shadow metadata is pointed to the original data segment corresponding to the rewritten data, and then the real metadata segment is pointed to the corresponding data segment in the cache data segment corresponding to the rewritten data and the data segment in the original data segment which is not rewritten. In this way, the present embodiment can realize shadow management of data by using the shadow data segment and the shadow metadata segment, ensure that data writing is performed only once IO issuing and once metadata updating issuing, and further optimize the single machine storage engine double-write and multiple-write problem, improve the performance of the single machine storage engine, maximize the performance of the hardware device, make the distributed storage performance not be in the bottom single machine storage engine, maximize the use of hardware performance, and make the distributed storage be able to provide optimal performance data. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0040] Figure 1 A data storage method flowchart is provided for the present application;

[0041] Figure 2 A specific data storage method flowchart is provided for the present application;

[0042] Figure 3 A data storage device structure schematic diagram is provided for the present application;

[0043] Figure 4 An electronic device structure diagram is provided in the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0045] In the prior art, the double-write or triple-write may occur in the distributed storage, which leads to the decline of the overall performance of the storage engine, and the storage engine cannot exert the performance of the basic storage device. In the present application, the double-write problem of data can be reduced while the read-write performance of large data is improved, so that the single machine storage engine can reach the performance bottleneck of the disk, and the performance of the new type of disk can be better exerted.

[0046] The embodiment of the present application discloses a data storage method, referring to Figure 1 The method comprises the following steps:

[0047] Step S11: the disk is divided into a metadata segment, a cache data segment and a real data segment according to a preset space division method; wherein the metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment.

[0048] In the embodiment, the logical division of the bare disk space of the disk is first performed, and the disk is divided into a metadata segment, a cache data segment and a real data segment. The metadata segment is used for storing real metadata and shadow metadata, the cache data segment is used for storing shadow data and fragment management data, and the real data segment is used for storing storage data.

[0049] In the embodiment, the logical division can achieve the isolation of the disk data segment, and the three divided data segments are located in different disk address spaces, which can ensure the continuity of the space and also ensure the fast addressing on the traditional physical disk, thereby increasing the performance of the storage engine. On the new Nand, IO merging can be performed when multiple data updates, so as to achieve the highest performance of the Nand.

[0050] In the embodiment, only the separate space index is needed during initialization, so as to achieve the management index operation of the disk.

[0051] It should be noted that the metadata segment in the embodiment is used to store real metadata and shadow metadata, and the old data and the new data can be managed through metadata management and shadow metadata management. In a specific implementation, the attributes of the shadow metadata and the real metadata are dynamically modified to achieve mutual backup, the metadata backup is achieved through metadata index, the historical metadata information is always retained, the data history protection is achieved, and the phenomenon of metadata loss caused by power failure and the like is avoided.

[0052] In the embodiment, the disk is divided into the metadata segment, the cache data segment and the real data segment according to the preset space division method, which can include: determining the space capacity of the metadata segment and the cache data segment according to the preset space division method; and taking the space capacity of the disk except the space capacity of the metadata segment and the cache data segment as the space capacity of the real data segment.

[0053] In the embodiment, the shadow data segment can realize full mapping of the real data segment, and when large data rewriting is performed, the data can be directly written into the shadow data segment in sequence.

[0054] In a specific implementation, when the disk is divided, the disk space can be divided by function, sufficient metadata space is reserved to store metadata and shadow metadata, a part of cache data space is reserved, the cache data space is divided into a shadow data segment and a fragment management data segment, and the remaining space is a real data segment.

[0055] Step S12: When initial data is issued, the initial issued data is written into the real data segment, and the real metadata corresponding to the initial issued data in the real metadata segment is pointed to the real data segment.

[0056] In the embodiment, the metadata segment can realize indexing of data, and when initial data is issued, the initial issued data is written into the real data segment, and the real metadata corresponding to the initial issued data is indexed to the real data segment. In a specific implementation, when initial data is issued, the data can be divided and issued according to the data segment corresponding to the metadata.

[0057] Step S13: When data rewriting is performed, the rewritten data is written into the cache data segment, the corresponding shadow metadata is pointed to the original data segment corresponding to the rewritten data, and then the real metadata segment is pointed to the corresponding data segment in the cache data segment corresponding to the rewritten data and the data segment in the original data segment which is not rewritten.

[0058] In the embodiment, when data is overwritten, the shadow metadata can be directed to the old data segment by the metadata management method proposed in the application, so as to avoid data loss caused by the fact that data or metadata is not saved to disk after power failure. Meanwhile, the method also plans the data block, writes the overwritten content into the cache data segment, and simultaneously points the new metadata space to the shadow block of the overwriting and the data block which is not overwritten.

[0059] In the embodiment, the data saved by the shadow metadata is old data, and data access and refreshing only occur in the real metadata, so that the real data orientation is not affected when the shadow metadata and shadow data are operated, so that only one IO is issued when writing, and one metadata update is issued at the same time. The embodiment also achieves single data writing through the shadow metadata and shadow data segment, and ensures data backup when data is issued abnormally, so as to avoid data loss caused by power failure or other modes.

[0060] In the embodiment, the data storage method can further include: determining the invalid data segment in the cache data segment by using a preset resource adjustment recovery method, and recovering the invalid data segment; determining the invalid metadata segment corresponding to the invalid data segment, and updating the metadata recorded in the invalid metadata segment to empty. It can be understood that the cache data segment in the embodiment is a dynamic recoverable data space. When the system performs data access and refreshing, the background can also determine the invalid data segment from the cache data segment, and the background can also determine the invalid metadata segment corresponding to the invalid data segment, recover and arrange the content of the shadow metadata, and avoid waste of disk space. In the embodiment, the history data is retained through the shadow data segment, and the dynamic data recovery makes enough space in the cache data segment to cache new overwritten data.

[0061] In the embodiment, the initial data IO is directly written into the real data segment, and the metadata information is updated; when the subsequent overwriting occurs, the data is written into the cache data segment, the real metadata is updated to new data, the initial metadata is updated to shadow metadata, and the next IO is issued or the data of the cache data segment is recovered, so as to realize single writing of the data segment and single change of the metadata information.

[0062] It should be pointed out that in the embodiment, the data storage method can further include: when the shadow data segment of the cache data space needs to be recycled after being used to a certain extent, intelligently evaluating the data to be recycled, i.e., recycling the data segment containing a small amount of data in the shadow data segment, storing the old data in the fragment management data segment in the cache data segment and updating the shadow metadata, then writing the valid data segment in the shadow metadata into the real data segment, and then recycling the shadow data segment and the data in the fragment management data segment in the cache data segment, so as to ensure that the data segment has two valid data segments as backup, and the shadow data and the shadow metadata can be recycled after the new data segment is updated, thereby achieving data space recycling and avoiding data space waste.

[0063] In the embodiment, the disk is first divided into a metadata segment, a cache data segment and a real data segment according to a preset space division method; the metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment. When initial data is issued, the initial issued data is written into the real data segment, and the real metadata corresponding to the initial issued data in the real metadata segment is pointed to the real data segment. When data is rewritten, the rewritten data is written into the cache data segment, the corresponding shadow metadata is pointed to the original data segment corresponding to the rewritten data, and then the real metadata segment is pointed to the corresponding data segment in the cache data segment corresponding to the rewritten data and the data segment in the original data segment that is not rewritten. In this way, the embodiment can realize shadow management of data by using the shadow data segment and the shadow metadata segment, and ensure that data writing is performed only once. The intelligent recycling method proposed in the embodiment optimizes the single machine storage engine double-write and multiple-write problem, improves the performance of the single machine storage engine, maximizes the performance of the hardware device, maximizes the use of hardware performance, and makes the distributed storage provide optimal performance data.

[0064] Figure 2 A specific data storage method flowchart is provided for the embodiment of the application. Referring to FIG. 21, Figure 2 The method includes the following steps:

[0065] Step S21: Dividing the disk into a metadata segment, a cache data segment and a real data segment according to a preset space division method; the metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment.

[0066] The more specific process of step S21 can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0067] Step S22: When the initial data is issued, the initial issued data is written into the real data segment, and the real metadata corresponding to the initial issued data in the real metadata segment is pointed to the real data segment.

[0068] The more specific process of step S22 can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0069] Step S23: When the data is overwritten, it is determined according to a preset data judgment rule whether the current overwriting data is large block data or small block data; if the current overwriting data is large block data, the target fragment data in the shadow data segment is written into the fragment management data segment, the corresponding shadow metadata is pointed to the overwriting data, and then the overwriting read-write data is sequentially written into the shadow data segment; if the current overwriting data is small block data, the overwriting data is written into the fragment management data segment in the cache data segment.

[0070] In this embodiment, the writing of the overwriting data into the cache data segment can include: applying a target data segment to the cache data segment based on the data size of the overwriting data; and writing the overwriting data into the target data segment.

[0071] In this embodiment, the sequentially writing of the overwriting read-write data into the shadow data segment can include: applying a target shadow data segment to the shadow data segment based on the data size of the overwriting data, and writing the overwriting data into the target shadow data segment; correspondingly, the writing of the overwriting data into the fragment management data segment in the cache data segment includes: applying a target small block data segment to the fragment management data segment based on the data size of the overwriting data, and writing the overwriting data into the target small block data segment. It can be understood that, in this embodiment, before overwriting data, a data segment space needs to be applied to the corresponding data segment for storing the overwriting data. Specifically, when the overwriting data is small block data, a target small block data segment is applied to the fragment management data segment in the cache data segment, and when the overwriting data is large block data, a target shadow data segment is applied to the shadow data segment in the cache data segment. After the data segment is applied, the overwriting data can be written into the corresponding data segment. In a specific implementation, a target data segment is applied to the corresponding data segment every time data is overwritten, so as to complete the writing of the overwriting data.

[0072] In the embodiment, the writing of the target fragment data in the shadow data section into the fragment management data section can include: determining a current minimum backup amount; intelligently evaluating data sections in the cache data section by using a preset intelligent evaluation method to determine target fragment data from the cache data section; and writing the target fragment data in the shadow data section into the fragment management data section based on the minimum backup amount. In the embodiment, for large IO, the large sequential read and write can be realized with the minimum data backup amount according to the content evaluation of the shadow data, thereby improving the performance of the entire single machine storage engine and thereby better exerting the disk performance of the physical hardware.

[0073] In the embodiment, the data block is backed up with the minimum data amount during the large block IO read and write. Since the data in the cache data section and the data section is frequently modified when the physical disk frequently performs small block IO, when the large block IO is issued, the effective data is distributed in the shadow data section of the cache data and the real data section. According to the traditional IO read and write, the large block IO needs to be split and then issued to different data sections, thereby causing the large block IO to be split into small blocks, the sequential IO can be changed into random IO, and the performance of the disk cannot be well exerted. Therefore, the IO of the physical disk is dynamically adjusted, the data block distribution space is intelligently evaluated, the minimum data section backup data section or shadow data section is performed, the rewriting content is written into the fragment management data space in the cache data section, the content of the shadow metadata is then changed, the entire data section is backed up, the large block data is sequentially written into the space of the minimum backup data section, the metadata is then updated, the large block sequential read and write are achieved, and the small block data section is recycled after the writing is completed, thereby achieving the small amount of read and write of the data and achieving the continuity of the large block data section. The embodiment can also perform the rewriting backup through the shadow data section and backup the data block with the minimum data amount during the large block IO read and write through the small block IO section, thereby achieving the large disk sequential read and write and thereby exerting the large block IO bandwidth of the physical disk.

[0074] In the embodiment, when data is overwritten, in the case of large block data read and write, the position of the data object can be dynamically adjusted. In some specific embodiments, the newer data in the shadow data block can be written into the cache data segment of the fragment management data segment for management, and the shadow metadata is modified to point to the new data block. At this time, the content in the shadow data block can be recycled, so as to write large blocks of data into the shadow data block, thereby achieving sequential writing of large blocks of data and optimal performance of the disk. In the embodiment, after the large block data is written into the shadow data block, the content update of the shadow data block is completed. After the content update of the shadow block is completed, the resource recycling of the original data block is performed, and the small block data written in the cache data block is recycled to avoid wasting space. At the same time, the shadow metadata is updated to be empty, that is, only the shadow block is used for data IO. When data rewriting occurs again, a data block is re-applied, data is written into the data block, the shadow metadata is updated, and single writing of the disk and metadata update are achieved.

[0075] In some specific implementation processes, data slicing operations can also be performed when data is issued. Specifically, the size of the disk and the estimated metadata space ratio can be coordinated to use large slices for large disks and small slices for small disks, thereby reducing the use of metadata space and ensuring the unified management of data.

[0076] In the embodiment, the disk is divided into a metadata segment, a cache data segment and a real data segment according to a preset space division method; the metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment. When initial data is issued, the initial issued data is written into the real data segment, and real metadata corresponding to the initial issued data in the real metadata segment is pointed to the real data segment. When data is rewritten, it is determined according to a preset data judgment rule that the current rewritten data is bulk data or small block data; if the current rewritten data is bulk data, target fragment data in the shadow data segment is written into the fragment management data segment, corresponding shadow metadata is pointed to the rewritten data, and then the rewritten read-write data is sequentially written into the shadow data segment; if the current rewritten data is small block data, the rewritten data is written into the fragment management data segment in the cache data segment. In the embodiment, old data and new data can be managed through metadata management and shadow metadata management, data loss after power failure is avoided, data history protection is realized, and data consistency after power failure is ensured. In the embodiment, the fragment management space is used to realize fragmented management backup when bulk read-write is performed, so that the rewritten data is directly written into the shadow data segment, single writing is realized, and the copied data segment can be written into the fragment management interval when bulk read-write is intelligently reduced, partial small writing is realized, most data is sequentially written in bulk, and the sequential read-write performance of bulk data is improved. In the embodiment, the shadow data and data of bulk can be evaluated, the minimum read-write content is put into the small block shadow data segment, and the shadow metadata is updated to ensure the integrity of the data segment.

[0077] Referring to Figure 3 The embodiment of the application discloses a data storage device, which can specifically include:

[0078] The space division module 11 is configured to divide the disk into a metadata segment, a cache data segment and a real data segment according to a preset space division method; the metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment.

[0079] The initial data issuing module 12 is configured to write initial issued data into the real data segment when initial data is issued, and point real metadata corresponding to the initial issued data in the real metadata segment to the real data segment.

[0080] The data rewriting module 13 is configured to write rewritten data into the cache data segment when data is rewritten, point corresponding shadow metadata to the original data segment corresponding to the rewritten data, and then point the real metadata segment to the corresponding data segment in the cache data segment corresponding to the rewritten data and the data segment in the original data segment which is not rewritten.

[0081] In the present application, the disk is first divided into a metadata segment, a cache data segment, and a real data segment according to a preset space division method; the metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment. When initial data is issued, the initial issued data is written into the real data segment, and the real metadata corresponding to the initial issued data in the real metadata segment is pointed to the real data segment. When data is rewritten, the rewritten data is written into the cache data segment, and the corresponding shadow metadata is pointed to the original data segment corresponding to the rewritten data, and then the real metadata segment is pointed to the corresponding data segment in the cache data segment corresponding to the rewritten data and the data segment in the original data segment that is not rewritten. In this way, the present embodiment can realize shadow management of data by using the shadow data segment and the shadow metadata segment, ensure that data writing is performed only once IO issuance and once metadata update issuance, thereby optimizing the single machine storage engine double-write and multiple-write problem, improving the performance of the single machine storage engine, maximizing the performance of the hardware device, and making the distributed storage performance no longer at the bottom of the single machine storage engine, maximizing the use of hardware performance, and making the distributed storage provide optimal performance data.

[0082] In some specific embodiments, the space division module 11 comprises:

[0083] A first capacity determination unit is configured to determine the space capacity of the metadata segment and the cache data segment according to a preset space division method.

[0084] A second capacity determination unit is configured to determine the space capacity of the real data segment as the space capacity of the disk except the space capacity of the metadata segment and the cache data segment.

[0085] In some specific embodiments, the data rewriting module 13 can specifically comprise:

[0086] A data determination unit is configured to determine whether the current rewritten data is large block data or small block data according to a preset data determination rule.

[0087] A large block rewriting unit is configured to, if the current rewritten data is large block data, write the target fragment data in the shadow data segment into the fragment management data segment, point the corresponding shadow metadata to the rewritten data, and then write the rewritten read-write data into the shadow data segment in sequence.

[0088] A small block rewriting unit is configured to, if the current rewritten data is small block data, write the rewritten data into the fragment management data segment in the cache data segment.

[0089] In some embodiments, the data rewriting module 13 can specifically include:

[0090] a data segment application unit configured to apply a target data segment to the cache data segment based on the data size of the rewriting data;

[0091] a rewriting data writing unit configured to write the rewriting data into the target data segment.

[0092] In some embodiments, the large-block rewriting unit can include:

[0093] a shadow data segment application unit configured to apply a target shadow data segment to the shadow data segment based on the data size of the rewriting data, and write the rewriting data into the target shadow data segment;

[0094] Correspondingly, the small-block rewriting unit can include:

[0095] a small-block data segment application unit configured to apply a target small-block data segment to the fragment management data segment based on the data size of the rewriting data, and write the rewriting data into the target small-block data segment.

[0096] In some embodiments, the large-block rewriting unit can include:

[0097] a backup amount determination unit configured to determine a current minimum backup amount;

[0098] an intelligent evaluation unit configured to intelligently evaluate the data segments in the cache data segment using a preset intelligent evaluation method, to determine a target fragment data from the cache data segment;

[0099] a fragment data writing unit configured to write the target fragment data in the shadow data segment into the fragment management data segment based on the minimum backup amount.

[0100] In some embodiments, the data storage device further includes:

[0101] a data segment recycling unit configured to determine invalid data segments in the cache data segment using a preset resource adjustment recycling method, and recycle the invalid data segments;

[0102] a metadata recycling unit configured to determine invalid metadata segments corresponding to the invalid data segments, and update the metadata recorded in the invalid metadata segments to be empty.

[0103] Further, the embodiments of the application also disclose an electronic device, Figure 4 is the structure diagram of the electronic device 20 according to the exemplary embodiments, and the content in the figure cannot be considered as any limitation on the use range of the application.

[0104] Figure 4 A structural schematic diagram of an electronic device 20 is provided in the embodiment. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a display screen 24, an input / output interface 25, a communication interface 26 and a communication bus 27. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the data storage method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the embodiment can be specifically an electronic computer.

[0105] In the embodiment, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 26 is capable of creating a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 26 can be any communication protocol applicable to the technical solution of the present application, which is not specifically limited here; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not specifically limited here.

[0106] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222 and virtual machine data 223, etc., and the virtual machine data 223 can include various data. The storage mode can be temporary storage or permanent storage.

[0107] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. The computer program 222 can further include computer programs capable of completing other specific work in addition to the computer programs capable of completing the data storage method executed by the electronic device 20 disclosed in any of the foregoing embodiments.

[0108] Further, the present application further discloses a computer readable storage medium, which includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a magnetic disk or an optical disk or any other form of storage medium known in the technical field. The computer program is executed by the processor to implement the foregoing disclosed data storage method. The specific steps of the method can refer to the corresponding contents disclosed in the foregoing embodiments, which will not be repeated here.

[0109] The various embodiments described in this specification are presented by way of example, and each embodiment is not necessarily composed of all features described with respect to other embodiments. Each embodiment described in this specification can be implemented in software, firmware, hardware, or any combination thereof. The various embodiments described in this specification can be implemented in any combination of the following technologies: software defined radio, code division multiple access (CDMA) networks, time division multiple access (TDMA) networks, frequency division multiple access (FDMA) networks, orthogonal frequency division multiplexing (OFDM) networks, orthogonal frequency division multiple access (OFDMA) networks, global system for mobile communications (GSM) networks, 3rd Generation (3G) networks, 4th Generation (4G) networks, and draft 5th Generation (5G) networks, long term evolution (LTE) networks, and the like.

[0110] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC).

[0111] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and the terms "include", "comprise", and changes thereof, are intended to encompass non-exclusive inclusion, such that processes, methods, articles, or apparatuses that include a set of elements not expressly listed or including additional elements are not excluded. In the absence of more limitations, the element defined by the phrase "including a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0112] The data storage method, device, equipment and storage medium provided by the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed, and the above description should not be understood as a limitation on the present application.

Claims

1. A data storage method, characterized by, The method comprises the following steps: According to the preset space division method, the disk is divided into metadata segments, cache data segments and real data segments; Among them, the metadata segment is divided into real metadata segment and shadow metadata segment, and the cache data segment is divided into shadow data segment and fragment management data segment; When the initial data is issued, the initial issued data is written into the real data segment, and the real metadata corresponding to the initial issued data in the real metadata segment is pointed to the real data segment; When data is overwritten, the overwritten data is written into the cache data segment, and the corresponding shadow metadata is pointed to the original data segment corresponding to the overwritten data, and then the real metadata segment is pointed to the corresponding data segment in the cache data segment corresponding to the overwritten data and the data segment in the original data segment which is not overwritten; Among them, when the data is overwritten, the overwritten data is written into the cache data segment, which comprises: judging whether the current overwritten data is large block data or small block data according to the preset data judgment rule; if the current overwritten data is large block data, the target fragment data in the shadow data segment is written into the fragment management data segment, and the corresponding shadow metadata is pointed to the overwritten data, and then the overwritten data is written into the shadow data segment according to the read-write sequence; if the current overwritten data is small block data, the overwritten data is written into the fragment management data segment in the cache data segment; The target fragment data in the shadow data segment is written into the fragment management data segment, which comprises: determining the current minimum backup amount; intelligently evaluating the data segments in the current cache data segment by using the preset intelligent evaluation method to determine the target fragment data from the cache data segment; based on the minimum backup amount, the target fragment data in the shadow data segment is written into the fragment management data segment.

2. The data storage method of claim 1, wherein, According to the preset space division method, the disk is divided into metadata segments, cache data segments and real data segments, which comprises: According to the preset space division method, the space capacity of the metadata segment and the cache data segment is determined; The space capacity of the disk except the space capacity of the metadata segment and the cache data segment is taken as the space capacity of the real data segment.

3. The data storage method of claim 1, wherein, The overwritten data is written into the cache data segment, which comprises: Based on the data size of the overwritten data, a target data segment is applied for the cache data segment; The overwritten data is written into the target data segment.

4. The data storage method of claim 3, wherein, The overwritten data is written into the shadow data segment according to the read-write sequence, which comprises: Based on the data size of the overwritten data, a target shadow data segment is applied for the shadow data segment, and the overwritten data is written into the target shadow data segment; Correspondingly, the overwritten data is written into the fragment management data segment in the cache data segment, which comprises: Based on the data size of the overwritten data, a target small block data segment is applied for the fragment management data segment, and the overwritten data is written into the target small block data segment.

5. The data storage method according to any one of claims 1 to 4, characterized in that, Further comprising: The invalid data segment in the cache data segment is determined by using the preset resource adjustment recovery method, and the invalid data segment is recovered; The invalid metadata segment corresponding to the invalid data segment is determined, and the metadata recorded in the invalid metadata segment is updated to be empty.

6. A data storage device, characterized by Comprise: The space division module is configured to divide the disk into a metadata segment, a cache data segment, and a real data segment according to a preset space division method. The metadata segment is divided into a real metadata segment and a shadow metadata segment, and the cache data segment is divided into a shadow data segment and a fragment management data segment. The initial data delivery module is configured to, when initial data is delivered, write the initial delivered data into the real data segment, and point the real metadata corresponding to the initial delivered data in the real metadata segment to the real data segment. The data rewriting module is configured to, when data is rewritten, write the rewritten data into the cache data segment, point the corresponding shadow metadata to the original data segment corresponding to the rewritten data, and then point the real metadata segment to the corresponding data segment in the cache data segment corresponding to the rewritten data and the data segment in the original data segment that is not rewritten. The data rewriting module is configured to determine whether the current rewritten data is large data or small data according to a preset data judgment rule. If the current rewritten data is large data, the target fragment data in the shadow data segment is written into the fragment management data segment, the corresponding shadow metadata is pointed to the rewritten data, and then the rewritten data is written into the shadow data segment according to the read-write order. If the current rewritten data is small data, the rewritten data is written into the fragment management data segment in the cache data segment. The data rewriting module is configured to determine the current minimum backup amount, intelligently evaluate the data segments in the cache data segment according to a preset intelligent evaluation method, and determine the target fragment data from the cache data segment. The target fragment data in the shadow data segment is written into the fragment management data segment based on the minimum backup amount.

7. An electronic device, comprising: The data storage method comprises a processor and a memory. When the processor executes a computer program stored in the memory, the data storage method of any one of claims 1 to 5 is realized.

8. A computer-readable storage medium, characterized in that, The computer program is stored in the memory. When the processor executes the computer program, the data storage method of any one of claims 1 to 5 is realized.

Images