Data storage method, device, equipment and medium

Abstract

The present application discloses a data storage method, apparatus, device, and medium, which relates to the field of computer technology and includes: obtaining data to be stored and filtering out target data to be stored from all the data to be stored; determining a target data segmentation method corresponding to the target data to be stored, and segmenting the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored; generating a virtual disk space based on the target sub-data to be stored and the remaining storage space information of the storage disk obtained in advance, and storing the target sub-data to be stored on the storage disk based on the virtual disk space. Through the above technical solution of the present application, it is possible to improve read and write performance, ensure data continuity, and reduce the amount of metadata in large file scenarios.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage method, device, equipment and medium. Background Art

[0002] BlueStore is the storage engine for Ceph (a distributed storage system). Designed to replace traditional FileStore (file storage), BlueStore serves as a next-generation, high-performance object storage backend. BlueStore supports both large and small file scenarios, but is not optimized for specific scenarios. For large file scenarios, such as the 2+1 erasure scenario, a 4MB object is split into two 2MB data files. When writing, each 2MB file must be split and then written in multiple batches. This creates data discontinuity, impacting read and write performance, and increasing the amount of metadata.

[0003] As can be seen from the above, in the process of data storage, how to improve read and write performance, ensure data continuity and reduce the amount of metadata in large file scenarios are problems to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data storage method, apparatus, device and medium that can improve read and write performance, ensure data continuity and reduce metadata volume in large file scenarios. The specific solution is as follows:

[0005] In a first aspect, the present application discloses a data storage method, comprising:

[0006] Acquire data to be stored, and filter out target data to be stored from all the data to be stored;

[0007] Determining a target data segmentation method corresponding to the target data to be stored, and segmenting the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored;

[0008] A virtual disk space is generated based on the target sub-data to be stored and pre-acquired remaining storage space information of the storage disk, and the target sub-data to be stored is stored in the storage disk based on the virtual disk space.

[0009] Optionally, obtaining the data to be stored includes:

[0010] Establishing a basic data management unit from the Blue Store storage engine and establishing a connection relationship between the Blue Store storage engine and the basic data management unit;

[0011] The Blue Store storage engine is used to obtain the data to be stored, and the data to be stored is sent to the basic data management unit.

[0012] Optionally, the filtering out target data to be stored from all the data to be stored includes:

[0013] Determining the occupied storage capacity information corresponding to the data to be stored in the basic data management unit;

[0014] A preset threshold is obtained, and it is determined whether the occupied storage capacity information is greater than the preset threshold; if the occupied storage capacity information is greater than the preset threshold, the data to be stored is determined as the target data to be stored.

[0015] Optionally, determining a target data segmentation method corresponding to the target data to be stored, and segmenting the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored includes:

[0016] Determining a target data segmentation method corresponding to the target data to be stored based on the occupied storage capacity information;

[0017] The target data to be stored is segmented based on erasure correction technology and according to the target data segmentation method to obtain target sub-data to be stored, and the target sub-data to be stored is saved to a Blob that establishes a mapping relationship with the basic data management unit.

[0018] Optionally, the segmenting the target data to be stored based on the erasure technology and according to the target data segmentation method to obtain target sub-data to be stored includes:

[0019] Determining the storage capacity occupied by the Blob;

[0020] The target data to be stored is divided based on the erasure correction technology and according to the target data division method to obtain the target sub-data to be stored with the same storage capacity as that occupied by the Blob.

[0021] Optionally, before generating the virtual disk space based on the target sub-data to be stored and the pre-acquired remaining storage space information of the storage disk, the method further includes:

[0022] Obtaining remaining storage space information of the storage disk;

[0023] Target storage space information is filtered out from all the remaining storage space information based on the target sub-data to be stored.

[0024] Optionally, generating a virtual disk space based on the target sub-data to be stored and pre-acquired remaining storage space information of the storage disk includes:

[0025] Establish a mapping connection relationship between Blob and blank virtual disk space;

[0026] A virtual disk space is generated based on the target sub-data to be stored, target storage space information, and the blank virtual disk space.

[0027] In a second aspect, the present application discloses a data storage device, comprising:

[0028] A data acquisition module is used to acquire data to be stored and filter out target data to be stored from all the data to be stored;

[0029] a data segmentation module, configured to determine a target data segmentation method corresponding to the target data to be stored, and segment the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored;

[0030] The data storage module is configured to generate a virtual disk space based on the target sub-data to be stored and pre-acquired remaining storage space information of the storage disk, and store the target sub-data to be stored in the storage disk based on the virtual disk space.

[0031] In a third aspect, the present application discloses an electronic device, comprising:

[0032] Memory, used to store computer programs;

[0033] The processor is used to execute the computer program to implement the aforementioned data storage method.

[0034] In a fourth aspect, the present application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed data storage method are implemented.

[0035] It can be seen that the present application provides a data storage method, including obtaining data to be stored, and screening out target data to be stored from all the data to be stored; determining a target data segmentation method corresponding to the target data to be stored, and segmenting the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored; generating a virtual disk space based on the target sub-data to be stored and the remaining storage space information of the storage disk obtained in advance, and storing the target sub-data to be stored on the storage disk based on the virtual disk space. The present application segments the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored, and then generates a virtual disk space based on the target sub-data to be stored and the remaining storage space information of the storage disk, thereby improving read and write performance, ensuring data continuity, and reducing the amount of metadata in large file scenarios. The key point of this application is to establish a one-to-one metadata mapping relationship between lextent, blob and pextent. The one-to-one correspondence between lextent, blob and pextent is achieved by controlling the size of the split data block and the size of the blob. For example, in the scenario of 2+1 erasure of a large file, the object size is 4M. After erasure, the object becomes two 2M data. The size of the blob and the split data block are set to 2M respectively. At this time, the data block is no longer split and is directly written down in 2M size. That is, a lextent is created corresponding to a 2M blob, and this blob corresponds to a 2M pextent, achieving a one-to-one correspondence. The size of the split data block and the size of the blob are set to the same size. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0037] Figure 1 A flow chart of a data storage method disclosed in this application;

[0038] Figure 2 A specific architecture diagram of a data storage method disclosed in this application;

[0039] Figure 3 A flow chart of a data storage method disclosed in this application;

[0040] Figure 4 This is a specific architecture diagram of an existing data storage method disclosed in this application;

[0041] Figure 5 This is a structural schematic diagram of a data storage device disclosed in this application;

[0042] Figure 6 This is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] BlueStore is the storage engine of Ceph (distributed storage system). BlueStore was created to replace the traditional FileStore (file storage) as a new generation of high-performance object storage backend. BlueStore supports both large and small file scenarios and is not optimized for specific scenarios. For large file scenarios, such as the 2+1 erasure scenario, a 4M object will be split into two 2M data. When writing, each 2M data needs to be split first, and then written in several times. This will cause data discontinuity, affect read and write performance, and increase the amount of metadata. As can be seen from the above, in the process of data storage, how to improve read and write performance, make data continuous, and reduce the amount of metadata in large file scenarios is a problem to be solved in this field. In the existing technology, BlueStore supports both large and small file scenarios, but is not optimized for specific scenarios. For large file scenarios, such as the 2+1 erasure scenario, a 4M object will be split into two 2M data. When writing, each 2M data needs to be split first, and then written several times. This will cause data discontinuity, affect read and write performance, and increase the amount of metadata. The reason for the above situation is the metadata mapping structure of BlueStore. The metadata mapping structure of BlueStore is as follows: Figure 4As shown, the correspondence between blob and lextent, as well as blob and pextent, may be that one blob may correspond to multiple lextents and multiple pextents, which causes data discontinuity, affects read and write performance, and has the problem of a large amount of metadata. However, the key point of this application is to establish a one-to-one metadata mapping relationship where one lextent corresponds to one blob and one pextent, and to achieve a one-to-one correspondence between lextent, blob, and pextent by controlling the size of the split data block and the size of the blob. For example, in the scenario of large file erasure 2+1, the object size is 4M. After erasure, the object becomes two 2M data. The size of the blob and the split data block is set to 2M respectively. At this time, the data block is no longer split and is directly written down in 2M size, that is, a lextent is created corresponding to a 2M blob, and this blob corresponds to a 2M pextent, achieving a one-to-one correspondence, and the size of the split data block and the size of the blob are set to the same size.

[0045] See also Figure 1 As shown, an embodiment of the present invention discloses a data storage method, which may specifically include:

[0046] Step S11: Acquire data to be stored, and filter out target data to be stored from all the data to be stored.

[0047] In this embodiment, a basic data management unit is established from the Blue Store storage engine, and a connection relationship is established between the Blue Store storage engine and the basic data management unit. Then, the Blue Store storage engine is used to obtain the data to be stored, and the data to be stored is sent to the basic data management unit. Then, the occupied storage capacity information corresponding to the data to be stored in the basic data management unit is determined, a preset threshold is obtained, and it is determined whether the occupied storage capacity information is greater than the preset threshold. If the occupied storage capacity information is greater than the preset threshold, the data to be stored is determined as the target data to be stored.

[0048] It is understandable that this application runs on the Linux platform and optimizes the read and write performance, data continuity and metadata volume in large file scenarios by changing the mapping relationship of BlueStore metadata. The specific architecture of BlueStore is as follows Figure 2As shown, a one-to-one connection relationship is established between Onode (structure) and lextent (basic data management unit) in BlueStore, and then the data to be stored is obtained, and the data to be stored is sent to the lextent, and the occupied storage capacity information corresponding to the data to be stored in the lextent is determined, and then a preset threshold is obtained, and it is determined whether the occupied storage capacity information is greater than the preset threshold (that is, it is determined whether the data to be stored is a large file). If the occupied storage capacity information is greater than the preset threshold, the data to be stored is determined as the target data to be stored.

[0049] Step S12: determining a target data segmentation method corresponding to the target data to be stored, and segmenting the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored.

[0050] In this embodiment, a target data segmentation method corresponding to the target data to be stored is determined based on the occupied storage capacity information, and then the target data to be stored is segmented based on the erasure technology and in accordance with the target data segmentation method to obtain target sub-data to be stored, and the target sub-data to be stored is saved to a Blob (Binary Large Object, representing a large object of binary type) that establishes a mapping relationship with the basic data management unit.

[0051] In this embodiment, the storage capacity occupied by the Blob is determined, and then the target data to be stored is divided based on the erasure technology and according to the target data division method to obtain the target sub-data to be stored with the same storage capacity as the Blob.

[0052] In other words, a one-to-one mapping relationship between a blob and a lextent is established. Based on the occupied storage capacity information, a corresponding target data segmentation method is determined. Then, based on erasure technology and the target data segmentation method, the target data to be stored is segmented to obtain target sub-data to be stored. For example, a 4MB file is first determined with a corresponding target data segmentation method. Then, an erasure 2+1 segmentation method is employed. After erasure, the data is converted into two 2MB target sub-data to be stored. The size of the blob and the segmented data block is set to 2MB respectively. At this point, the data block is no longer segmented and is written directly down in 2MB size.

[0053] Step S13: generating a virtual disk space based on the target sub-data to be stored and the pre-acquired remaining storage space information of the storage disk, and storing the target sub-data to be stored in the storage disk based on the virtual disk space.

[0054] In this embodiment, when it is detected that there is target sub-data to be stored, a blank virtual disk space is created and mapped to the blob. Then, based on the target sub-data to be stored, the target storage space information, and the blank virtual disk space, a virtual disk space is generated. The target sub-data to be stored is then stored on the disk (i.e., the storage disk) based on the virtual disk space. Ultimately, a lextent is created that corresponds to a 2M blob, and this blob corresponds to a 2M pextent (i.e., the virtual disk space), achieving a one-to-one correspondence.

[0055] The key point of this application is to establish a one-to-one metadata mapping relationship between lextent, blob and pextent, and to achieve a one-to-one correspondence between lextent, blob and pextent by controlling the size of the split data block and the size of the blob. For example, in the scenario of large file erasure 2+1, the object size is 4M. After erasure, the object becomes two 2M data. The size of the blob and the split data block are set to 2M respectively. At this time, the data block is no longer split and is directly written down at a size of 2M, that is, a lextent is created corresponding to a 2M blob, and this blob corresponds to a 2M pextent, achieving a one-to-one correspondence. The size of the split data block and the size of the blob are set to the same size. In large file scenarios, the above-mentioned BlueStore optimization strategy can enhance data continuity, improve read and write performance, increase the size of the written data block, and reduce the amount of metadata of the object.

[0056] In this embodiment, the data to be stored is obtained, and the target data to be stored is filtered out from all the data to be stored; a target data segmentation method corresponding to the target data to be stored is determined, and the target data to be stored is segmented according to the target data segmentation method to obtain target sub-data to be stored; a virtual disk space is generated based on the target sub-data to be stored and the remaining storage space information of the storage disk obtained in advance, and the target sub-data to be stored is stored to the storage disk based on the virtual disk space. The present application segments the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored, and then generates a virtual disk space based on the target sub-data to be stored and the remaining storage space information of the storage disk, thereby improving read and write performance, ensuring data continuity, and reducing the amount of metadata in large file scenarios.

[0057] The present invention runs on the Linux platform, and optimizes the read and write performance, data continuity and metadata amount in large file scenarios by changing the mapping relationship of BlueStore metadata, which has great advantages in improving read and write performance, data continuity and reducing metadata amount. BlueStore is the storage engine of Ceph (distributed storage system). The birth of BlueStore is to replace the traditional FileStore (file storage) as a new generation of high-performance object storage backend. BlueStore supports both large and small file scenarios, and is not optimized for specific scenarios. For large file scenarios, such as the 2+1 erasure scenario, a 4M object will be divided into two 2M data. When writing, each 2M data needs to be divided first, and then written several times. This will cause data discontinuity, affect read and write performance, and cause the amount of metadata to increase. As can be seen from the above, in the process of data storage, how to improve read and write performance, make data continuity and reduce the amount of metadata in large file scenarios is a problem to be solved in this field. The key point of this application is to establish a one-to-one metadata mapping relationship between lextent, blob and pextent, and to achieve a one-to-one correspondence between lextent, blob and pextent by controlling the size of the split data block and the size of the blob. For example, in the scenario of large file erasure 2+1, the object size is 4M. After erasure, the object becomes two 2M data. The size of the blob and the split data block are set to 2M respectively. At this time, the data block is no longer split and is directly written down at a size of 2M, that is, a lextent is created corresponding to a 2M blob, and this blob corresponds to a 2M pextent, achieving a one-to-one correspondence. The size of the split data block and the size of the blob are set to the same size. In large file scenarios, the above-mentioned BlueStore optimization strategy can enhance data continuity, improve read and write performance, increase the size of the written data block, and reduce the amount of metadata of the object.

[0058] See also Figure 3 As shown, an embodiment of the present invention discloses a data storage method, which may specifically include:

[0059] Step S21: Acquire data to be stored, and filter out target data to be stored from all the data to be stored.

[0060] Step S22: determining a target data segmentation method corresponding to the target data to be stored, and segmenting the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored.

[0061] Step S23: obtaining the remaining storage space information of the storage disk, and then filtering out target storage space information from all the remaining storage space information based on the target sub-data to be stored.

[0062] Step S24: generating a virtual disk space based on the target sub-data to be stored and the pre-acquired remaining storage space information of the storage disk, and storing the target sub-data to be stored in the storage disk based on the virtual disk space.

[0063] For more detailed processing procedures in steps S21, S22, S23, and S24, please refer to the aforementioned disclosed embodiments, which will not be described again here.

[0064] The present invention runs on the Linux platform, and optimizes the read and write performance, data continuity and metadata amount in large file scenarios by changing the mapping relationship of BlueStore metadata, which has great advantages in improving read and write performance, data continuity and reducing metadata amount. BlueStore is the storage engine of Ceph (distributed storage system). The birth of BlueStore is to replace the traditional FileStore (file storage) as a new generation of high-performance object storage backend. BlueStore supports both large and small file scenarios, and is not optimized for specific scenarios. For large file scenarios, such as the 2+1 erasure scenario, a 4M object will be divided into two 2M data. When writing, each 2M data needs to be divided first, and then written several times. This will cause data discontinuity, affect read and write performance, and cause the amount of metadata to increase. As can be seen from the above, in the process of data storage, how to improve read and write performance, make data continuity and reduce the amount of metadata in large file scenarios is a problem to be solved in this field. In the existing technology, BlueStore supports both large and small file scenarios, but is not optimized for specific scenarios. For large file scenarios, such as the 2+1 erasure scenario, a 4M object will be split into two 2M data. When writing, each 2M data needs to be split first, and then written several times. This will cause data discontinuity, affect read and write performance, and increase the amount of metadata. The reason for the above situation is the metadata mapping structure of BlueStore. The metadata mapping structure of BlueStore is as follows: Figure 4As shown, the correspondence between blob and lextent, as well as blob and pextent, may be that one blob may correspond to multiple lextents and multiple pextents, which causes data discontinuity, affects read and write performance, and has the problem of a large amount of metadata. However, the key point of this application is to establish a one-to-one metadata mapping relationship where one lextent corresponds to one blob and one pextent, and to achieve a one-to-one correspondence between lextent, blob, and pextent by controlling the size of the split data block and the size of the blob. For example, in the scenario of large file erasure 2+1, the object size is 4M. After erasure, the object becomes two 2M data. The size of the blob and the split data block is set to 2M respectively. At this time, the data block is no longer split and is directly written down in 2M size, that is, a lextent is created corresponding to a 2M blob, and this blob corresponds to a 2M pextent, achieving a one-to-one correspondence, and the size of the split data block and the size of the blob are set to the same size.

[0065] In this embodiment, the data to be stored is obtained, and the target data to be stored is filtered out from all the data to be stored; the target data segmentation method corresponding to the target data to be stored is determined, and the target data to be stored is segmented according to the target data segmentation method to obtain target sub-data to be stored; a virtual disk space is generated based on the target sub-data to be stored and the remaining storage space information of the storage disk obtained in advance, and the target sub-data to be stored is stored to the storage disk based on the virtual disk space. The present application segments the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored, and then generates a virtual disk space based on the target sub-data to be stored and the remaining storage space information of the storage disk, thereby improving read and write performance, ensuring data continuity and reducing the amount of metadata in large file scenarios. The present invention runs on the Linux platform, and optimizes the read and write performance, data continuity and metadata amount in large file scenarios by changing the mapping relationship of the BlueStore metadata, which has great advantages in improving read and write performance, data continuity and reducing the amount of metadata. In large file scenarios, the above-mentioned BlueStore optimization strategy can enhance data continuity, improve read and write performance, increase the size of written data blocks, and reduce the amount of object metadata.

[0066] See also Figure 5 As shown, an embodiment of the present invention discloses a data storage device, which may specifically include:

[0067] The data acquisition module 11 is used to acquire data to be stored and select target data to be stored from all the data to be stored;

[0068] The data segmentation module 12 is configured to determine a target data segmentation method corresponding to the target data to be stored, and segment the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored;

[0069] The data storage module 13 is configured to generate a virtual disk space based on the target sub-data to be stored and pre-acquired remaining storage space information of the storage disk, and store the target sub-data to be stored in the storage disk based on the virtual disk space.

[0070] In this embodiment, the data to be stored is obtained, and the target data to be stored is filtered out from all the data to be stored; a target data segmentation method corresponding to the target data to be stored is determined, and the target data to be stored is segmented according to the target data segmentation method to obtain target sub-data to be stored; a virtual disk space is generated based on the target sub-data to be stored and the remaining storage space information of the storage disk obtained in advance, and the target sub-data to be stored is stored to the storage disk based on the virtual disk space. The present application segments the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored, and then generates a virtual disk space based on the target sub-data to be stored and the remaining storage space information of the storage disk, thereby improving read and write performance, ensuring data continuity, and reducing the amount of metadata in large file scenarios.

[0071] The present invention runs on the Linux platform, and optimizes the read and write performance, data continuity and metadata amount in large file scenarios by changing the mapping relationship of BlueStore metadata, which has great advantages in improving read and write performance, data continuity and reducing metadata amount. BlueStore is the storage engine of Ceph (distributed storage system). The birth of BlueStore is to replace the traditional FileStore (file storage) as a new generation of high-performance object storage backend. BlueStore supports both large and small file scenarios, and is not optimized for specific scenarios. For large file scenarios, such as the 2+1 erasure scenario, a 4M object will be divided into two 2M data. When writing, each 2M data needs to be divided first, and then written several times. This will cause data discontinuity, affect read and write performance, and cause the amount of metadata to increase. As can be seen from the above, in the process of data storage, how to improve read and write performance, make data continuity and reduce the amount of metadata in large file scenarios is a problem to be solved in this field. In the existing technology, BlueStore supports both large and small file scenarios, but is not optimized for specific scenarios. For large file scenarios, such as the 2+1 erasure scenario, a 4M object will be split into two 2M data. When writing, each 2M data needs to be split first, and then written several times. This will cause data discontinuity, affect read and write performance, and increase the amount of metadata. The reason for the above situation is the metadata mapping structure of BlueStore. The metadata mapping structure of BlueStore is as follows: Figure 4As shown, the correspondence between blob and lextent, as well as blob and pextent, may be that one blob may correspond to multiple lextents and multiple pextents, which causes data discontinuity, affects read and write performance, and has the problem of a large amount of metadata. The key point of this application is to establish a one-to-one metadata mapping relationship where one lextent corresponds to one blob and one pextent, and to achieve a one-to-one correspondence between lextent, blob, and pextent by controlling the size of the split data block and the size of the blob. For example, in the scenario of large file erasure 2+1, the object size is 4M. After erasure, the object becomes two 2M data. The size of the blob and the split data block is set to 2M respectively. At this time, the data block is no longer split and is directly written down in 2M size, that is, a lextent is created corresponding to a 2M blob, and this blob corresponds to a 2M pextent, achieving a one-to-one correspondence. The size of the split data block and the size of the blob are set to the same size. This invention, running on the Linux platform, optimizes read / write performance, data continuity, and metadata volume in large file scenarios by modifying the BlueStore metadata mapping relationship. This significantly improves read / write performance, data continuity, and metadata volume. In large file scenarios, the aforementioned BlueStore optimization strategy enhances data continuity, improves read / write performance, increases write data block size, and reduces the amount of object metadata.

[0072] In some specific embodiments, the data acquisition module 11 may specifically include:

[0073] A unit establishment module is used to establish a basic data management unit from the Blue Store storage engine and establish a connection relationship between the Blue Store storage engine and the basic data management unit;

[0074] The data sending module is used to obtain the data to be stored by using the Blue Store storage engine and send the data to be stored to the basic data management unit.

[0075] In some specific embodiments, the data acquisition module 11 may specifically include:

[0076] a capacity information determination module, configured to determine the occupied storage capacity information corresponding to the data to be stored in the basic data management unit;

[0077] The judgment module is used to obtain a preset threshold and judge whether the occupied storage capacity information is greater than the preset threshold. If the occupied storage capacity information is greater than the preset threshold, the data to be stored is determined as the target data to be stored.

[0078] In some specific embodiments, the data segmentation module 12 may specifically include:

[0079] a data segmentation method determination module, configured to determine a target data segmentation method corresponding to the target data to be stored based on the occupied storage capacity information;

[0080] The data segmentation module is used to segment the target data to be stored based on the erasure technology and according to the target data segmentation method to obtain target sub-data to be stored, and save the target sub-data to be stored to a Blob that establishes a mapping relationship with the basic data management unit.

[0081] In some specific embodiments, the data segmentation module 12 may specifically include:

[0082] A storage capacity determination module, configured to determine the storage capacity occupied by the Blob;

[0083] The target sub-data to be stored determining module is used to split the target data to be stored based on the erasure technology and according to the target data splitting method to obtain the target sub-data to be stored with the same storage capacity as that occupied by the Blob.

[0084] In some specific embodiments, the data storage module 13 may specifically include:

[0085] A remaining storage space information acquisition module is used to obtain the remaining storage space information of the storage disk;

[0086] The target storage space information determining module is configured to filter out target storage space information from all the remaining storage space information based on the target sub-data to be stored.

[0087] In some specific embodiments, the data storage module 13 may specifically include:

[0088] Establish a Blob module for establishing a mapping connection relationship between Blob and blank virtual disk space;

[0089] The virtual disk space generating module is configured to generate a virtual disk space based on the target sub-data to be stored, target storage space information and the blank virtual disk space.

[0090] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the data storage method performed by the electronic device disclosed in any of the aforementioned embodiments.

[0091] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0092] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0093] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, so as to enable the processor 21 to calculate and process the data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. In addition to including computer programs capable of performing the data storage method performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the data storage device from external devices, as well as data collected by its own input and output interface 25.

[0094] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0095] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the data storage method disclosed in any of the aforementioned embodiments are implemented.

[0096] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0097] The above is a detailed introduction to a data storage method, device, equipment and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A data storage method, characterized in that: include: Acquire data to be stored, and filter out target data to be stored from all the data to be stored; Determining a target data segmentation method corresponding to the target data to be stored, and segmenting the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored; generating a virtual disk space based on the target sub-data to be stored and pre-acquired remaining storage space information of the storage disk, and storing the target sub-data to be stored in the storage disk based on the virtual disk space; The determining of a target data segmentation method corresponding to the target data to be stored, and segmenting the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored, includes: Determining a target data segmentation method corresponding to the target data to be stored based on the occupied storage capacity information; Based on the erasure correction technology and in accordance with the target data segmentation method, the target data to be stored is segmented to obtain target sub-data to be stored, and the target sub-data to be stored is saved to a Blob that establishes a mapping relationship with the basic data management unit; The target data to be stored is segmented based on the erasure correction technology and according to the target data segmentation method to obtain target sub-data to be stored, including: Determining the storage capacity occupied by the Blob; The target data to be stored is divided based on the erasure correction technology and according to the target data division method to obtain the target sub-data to be stored with the same storage capacity as that occupied by the Blob.

2. The data storage method according to claim 1, wherein: The obtaining of the data to be stored includes: Establishing a basic data management unit from the Blue Store storage engine and establishing a connection relationship between the Blue Store storage engine and the basic data management unit; The Blue Store storage engine is used to obtain the data to be stored, and the data to be stored is sent to the basic data management unit.

3. The data storage method according to claim 2, wherein: The step of filtering out target data to be stored from all the data to be stored includes: Determining the occupied storage capacity information corresponding to the data to be stored in the basic data management unit; A preset threshold is obtained, and it is determined whether the occupied storage capacity information is greater than the preset threshold; if the occupied storage capacity information is greater than the preset threshold, the data to be stored is determined as the target data to be stored.

4. The data storage method according to any one of claims 1 to 3, characterized in that: Before generating the virtual disk space based on the target sub-data to be stored and the pre-acquired remaining storage space information of the storage disk, the method further includes: Obtaining remaining storage space information of the storage disk; Target storage space information is filtered out from all the remaining storage space information based on the target sub-data to be stored.

5. The data storage method according to claim 4, characterized in that: The generating of the virtual disk space based on the target sub-data to be stored and the pre-acquired remaining storage space information of the storage disk includes: Establish a mapping connection relationship between Blob and blank virtual disk space; A virtual disk space is generated based on the target sub-data to be stored, target storage space information, and the blank virtual disk space.

6. A data storage device, characterized in that include: A data acquisition module is used to acquire data to be stored and filter out target data to be stored from all the data to be stored; a data segmentation module, configured to determine a target data segmentation method corresponding to the target data to be stored, and segment the target data to be stored according to the target data segmentation method to obtain target sub-data to be stored; a data storage module, configured to generate a virtual disk space based on the target sub-data to be stored and pre-acquired remaining storage space information of the storage disk, and store the target sub-data to be stored in the storage disk based on the virtual disk space; The data segmentation module is specifically used to: Determining a target data segmentation method corresponding to the target data to be stored based on the occupied storage capacity information; Based on the erasure correction technology and in accordance with the target data segmentation method, the target data to be stored is segmented to obtain target sub-data to be stored, and the target sub-data to be stored is saved to a Blob that establishes a mapping relationship with the basic data management unit; The data segmentation module includes: A storage capacity determination module, configured to determine the storage capacity occupied by the Blob; The target sub-data to be stored determining module is used to split the target data to be stored based on the erasure technology and according to the target data splitting method to obtain the target sub-data to be stored with the same storage capacity as that occupied by the Blob.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the data storage method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that Used to store computer programs; wherein, when the computer program is executed by a processor, the data storage method according to any one of claims 1 to 5 is implemented.

Images