Data storage method and device, data reading method and device and electronic equipment

Abstract

The embodiment of the invention provides a data storage method and device, a data reading method and device and electronic equipment, and the method comprises the steps: employing a preset hash function to calculate a hash value based on a resource locator in response to a storage request for a target file; the hash value is composed of N bit segments; the target file is obtained based on the resource locator; calculating a target storage space of the ith hierarchy under the storage space of the (i-1) th hierarchy based on the ith bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the (i-1) th level of storage space is the total storage space; and storing the target file in the target storage space of the Nth level. By using the scheme to store and query mass data, data query and return can be quickly carried out, the response speed of storage service can be improved, and the service quality is improved. And meta information corresponding to mass data does not need to be stored in a memory, so that precious memory resources are saved.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a data storage method, a data reading method, a device and an electronic device. Background Art

[0002] At present, electronic devices often use a more traditional addressing method when storing and reading files. That is, part of the file information is used as the key, the file storage address is used as the value, and the key-value information is loaded into the memory as the file's meta (meta information) for file addressing. However, when there are many files to be stored, a large amount of meta information will be generated. This meta information will occupy a large amount of memory resources, and even the large amount of meta information may cause the memory to not be able to load all the cached file's meta information. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a data storage method, a data reading method, a device and an electronic device, which solves the problem that the original metadata cannot be directly loaded into the memory for addressing due to the large amount of data, and solves the problem by using a pre-set rule and algorithm for addressing. The specific technical solution is as follows:

[0004] In a first aspect of the present application, a data storage method is provided, the method comprising:

[0005] In response to a storage request for a target file, a hash value is calculated based on a resource locator using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; wherein the target file is obtained based on the resource locator;

[0006] The target storage space of the i-th level under the i-1-th level storage space is calculated based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space;

[0007] Optionally, the step of calculating the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment includes:

[0008] Determine a first value of a target base corresponding to the i-th bit segment;

[0009] Determine the number of storage spaces at level i;

[0010] A modulo operation is performed on the number of storage spaces at the i-th level based on the first value, and a storage space with a sequence number as a result of the modulo operation is determined as a target storage space at the i-th level.

[0011] Optionally, each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs;

[0012] The address of the target storage space at the Nth level is: the sum of the address offsets corresponding to the target storage spaces at the N levels.

[0013] In a second aspect of the present application, a data reading method is provided, the method comprising:

[0014] Obtaining a read request for a target file, wherein the read request carries a resource locator of the target file;

[0015] Based on the resource locator, a hash value is calculated using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1;

[0016] The target storage space of the i-th level under the i-1-th level storage space is calculated based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space;

[0017] The target file is read from the target storage space at the Nth level.

[0018] Optionally, the step of calculating the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment includes:

[0019] Determine a first value of a target base corresponding to the i-th bit segment;

[0020] Determine the number of storage spaces at level i;

[0021] A modulo operation is performed on the number of storage spaces at the i-th level based on the first value, and a storage space with a sequence number as a result of the modulo operation is determined as a target storage space at the i-th level.

[0022] Optional,

[0023] Each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs;

[0024] The address of the target storage space at the Nth level is: the sum of the address offsets corresponding to the target storage spaces at the N levels.

[0025] Optionally, the method further includes:

[0026] The resource locator corresponding to the storage file is read from the target storage space of the Nth level, and the accuracy is verified based on the read resource locator and the resource locator carried in the read request. If the verification passes, the storage file is determined to be the target file.

[0027] In a third aspect of the present application, a data storage device is further provided, the device comprising:

[0028] A first calculation module, configured to calculate a hash value based on a resource locator and in response to a storage request for a target file, using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; wherein the target file is obtained based on the resource locator;

[0029] The second calculation module is used to calculate the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space;

[0030] The storage module is used to store the target file in the target storage space of the Nth level.

[0031] Optionally, the second computing module is specifically configured to:

[0032] Determine a first value of a target base corresponding to the i-th bit segment;

[0033] Determine the number of storage spaces at level i;

[0034] A modulo operation is performed on the number of storage spaces at the i-th level based on the first value, and a storage space with a sequence number as a result of the modulo operation is determined as a target storage space at the i-th level.

[0035] Optionally, each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs;

[0036] The address of the target storage space at the Nth level is: the sum of the address offsets corresponding to the target storage spaces at the N levels.

[0037] In a fourth aspect of the present application, a data reading device is further provided, the device comprising:

[0038] An acquisition module, used for acquiring a read request for a target file, wherein the read request carries a resource locator of the target file;

[0039] A third calculation module, configured to calculate a hash value based on the resource locator using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1;

[0040] A fourth calculation module is used to calculate the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space;

[0041] The reading module is used to read the target file from the target storage space at the Nth level.

[0042] Optionally, the fourth computing module is specifically configured to:

[0043] Determine a first value of a target base corresponding to the i-th bit segment;

[0044] Determine the number of storage spaces at level i;

[0045] A modulo operation is performed on the number of storage spaces at the i-th level based on the first value, and a storage space with a sequence number as a result of the modulo operation is determined as a target storage space at the i-th level.

[0046] Optionally, each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs;

[0047] The address of the target storage space at the Nth level is: the sum of the address offsets corresponding to the target storage spaces at the N levels.

[0048] Optionally, the device further comprises:

[0049] The verification module is used to read the resource locator corresponding to the storage file from the target storage space of the Nth level, and perform accuracy verification based on the read resource locator and the resource locator carried in the read request. If the verification passes, it is determined that the storage file is the target file.

[0050] In a fifth aspect of the present application, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0051] Memory, used to store computer programs;

[0052] The processor is used to implement any of the above-mentioned method steps when executing the program stored in the memory.

[0053] In a sixth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above method steps is implemented.

[0054] In a seventh aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements any of the above method steps when executed by a processor.

[0055] The data storage method, data reading method, device and electronic device provided in the embodiments of the present application are applied, and in response to a storage request for a target file, a hash value is calculated based on a resource locator using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; wherein the target file is obtained based on the resource locator; the i-th level target storage space under the i-1 level storage space is calculated based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1 level storage space is the total storage space; and the target file is stored in the N-th level target storage space.

[0056] The data storage method provided in the embodiment of the present application is used to store massive data. Since the storage space is divided into levels in advance, the hash function is pre-set, and the segmentation of the hash value is specified, when the target file needs to be stored, it is only necessary to perform a hash operation, and then calculate the target storage space at different levels according to the various bit segments contained in the hash value, and finally locate the target storage space at the Nth level, and store the target file in the target storage space at the Nth level.

[0057] It can be seen that the problem that the original metadata could not be directly loaded into the memory for addressing due to the large amount of data is solved by a solution of addressing with a pre-set rule and algorithm. It is not necessary to store the metadata corresponding to the massive data in the memory, saving precious memory resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0059] Figure 1 A schematic diagram of a flow chart of a data storage method provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of the storage space division provided in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of a flow chart of a data reading method provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of a structure of a data storage device provided in an embodiment of the present application;

[0063] Figure 5 A schematic diagram of the structure of a data reading device provided in an embodiment of the present application;

[0064] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0066] The traditional file addressing method requires loading the key-value as the file's meta information into the memory. When the number of files to be stored is large, a large amount of meta information needs to be stored in the memory, which takes up a lot of memory resources. The amount of meta information is so large that the memory cannot load the meta information of all cached files at all.

[0067] In order to solve the above technical problems, the embodiments of the present application provide a data storage method, a data reading method, a device and an electronic device, which are applicable to electronic devices, storage systems or servers that need to store a large number of files, such as CDN (Content Delivery Network) servers.

[0068] It should be noted that any electronic device, storage system or server that has the need to store massive files, or any electronic device, storage system or server that wants to avoid meta information occupying more memory space, can adopt the data storage method and data reading method provided in the embodiments of the present application.

[0069] Since the CDN server needs to store a massive amount of small file data, the data storage method and data reading method provided in the embodiments of the present application are very suitable for the CDN server. In the embodiments below, the CDN server is used as the execution subject for introduction.

[0070] To facilitate understanding, we first briefly introduce CDN technology.

[0071] The essence of CDN technology is to cache the website content at the edge of the network. When a user accesses the website content, the scheduling system will route or guide the user's request to the cache node that is closest to the user's access network or has the best access effect. The cache node will provide content services to the user. Compared with directly accessing the source station, this method can reduce the number of routing times, increase download speed, shorten transmission time, and improve user experience.

[0072] The CDN storage server actually belongs to the above-mentioned cache node and needs to store data of various services.

[0073] The number of files cached by CDN storage servers is very large. Usually, one server can cache billions of data, while the average file size is only a few hundred bytes to a few hundred KB (kilobytes). Therefore, if the traditional file addressing method is used, billions of metadata information needs to be stored for billions of data to be stored. This information needs to be stored in memory, and memory resources are very precious and limited. In addition, the memory may not be able to store the metadata information of all cached files.

[0074] See also Figure 1 , is a flow chart of a data storage method provided in an embodiment of the present application, comprising the following steps:

[0075] S101: In response to a storage request for a target file, a hash value is calculated based on a resource locator using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; wherein the target file is obtained based on the resource locator.

[0076] In the embodiment of the present application, the target file to be stored is acquired from the network in advance based on the resource locator.

[0077] Among them, the resource locator, or URL, is a string used to identify and locate resources on the Internet. URL is a standardized format consisting of multiple components used to specify the location and access method of a resource. A complete URL usually contains the following parts: Protocol: The protocol part of the URL specifies the protocol used to access the resource. Such as HTTP (Hypertext Transfer Protocol), HTTPS (Hypertext Transfer Protocol Secure), FTP (File Transfer Protocol), etc. Host: The host part of the URL specifies the host name or IP address of the server where the resource is located. Port: The port part of the URL specifies the communication port number of the server where the resource is located. The default port number is related to the protocol. Path: The path part of the URL specifies the specific location of the resource on the server. Query String: The query string part of the URL is used to pass parameters to the server so that the server can dynamically generate a response based on the value of the parameter. Fragment: The fragment identifier part of the URL is used to identify a specific fragment or location in the resource.

[0078] After obtaining the target file based on the URL, the target file is stored in its own storage space for easy reading next time, so that when the target file needs to be obtained again later, it can be obtained from the storage space.

[0079] As an example, a user device initiates an access request to a CDN server, which carries the URL of a target file. The CDN server obtains the target file from the network based on the URL. When the CDN server determines that the target file is a frequently accessed file, it can store the target file in its own storage space for direct retrieval next time.

[0080] For the target file to be stored, a hash value is calculated using a preset hash function based on the resource locator corresponding to the target file, and then addressing is performed according to multiple bit segments contained in the hash value to determine the corresponding storage space.

[0081] A hash function is a function that maps input data to a hash value of a fixed size. By calculating the input data, a unique hash value is generated, which can be used to quickly find or verify the integrity of the data.

[0082] The present application embodiment does not limit the selection of the hash function. As an example, the SHA-1 function, the MD5 function, etc. may be used.

[0083] Since the length of the calculated hash value is fixed, the bit segment division of the hash value can be preset.

[0084] As an example, a 128-bit hash value is calculated based on the MD5 function, and the first 32 bits can be divided into a first bit segment, and the remaining 96 bits can be divided into a second bit segment.

[0085] S102: Calculate the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i=1, the i-1-th level storage space is the total storage space.

[0086] In the embodiment of the present application, the storage space used to store the target file is also pre-divided into hierarchical forms.

[0087] Specifically, when the storage system is first started, a large storage block may be allocated according to the disk configuration size as the total storage space, and then hierarchical regions may be divided based on the total storage space.

[0088] For example, if a disk is 850G, 800G of the 850G disk is used as a large storage block, namely the total storage space. The 800G total storage space is further divided into several areas, such as A, B, C, and D, each with 200G. This is the first level of division. Then each 200G storage space is further divided, for example, into M storage spaces, which is the second level of division.

[0089] In the embodiment of the present application, the number of bit segments divided by the hash value is consistent with the number of levels into which the total storage space is divided. For example, if the hash value is divided into 2 bit segments, that is, N=2, then the number of levels into which the total storage space is divided is also 2.

[0090] Thus, a single level of target storage space can be addressed based on each bit segment.

[0091] Specifically, firstly, the target storage space of the first level under the storage space of the 0th level is calculated based on the 1st bit segment, wherein the storage space of the 0th level is the total storage space.

[0092] Continuing with the above example, if a 128-bit hash value is calculated based on the MD5 function, the first 32 bits are divided into the first bit segment, and the remaining 96 bits are divided into the second bit segment. Then the target storage space of the first level under the total storage space is calculated based on the 32 bits contained in the first bit segment. The total storage space is 800G of storage space, and the first level of storage space contains 4 200G storage spaces (i.e., A, B, C, and D storage spaces). Then, the operation is performed based on the first bit segment, and the target storage space of the first level is determined based on the operation result.

[0093] If the target storage space of the first level is determined to be storage space B, the target storage space of the second level is calculated based on the 96 bits contained in the second bit segment. The storage space of the second level includes M storage spaces, each of which has a size of 200G / M. The calculation is performed based on the second bit, and the target storage space of the second level can be determined based on the calculation result.

[0094] It should be noted that, using the data storage method provided in the embodiment of the present application, the target files are ultimately stored in the Nth level storage space, and each storage space is only allowed to store one target file. The size of the Nth level storage space can be set according to demand.

[0095] In one embodiment of the present application, the step of calculating the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment may specifically include:

[0096] Determine a first numerical value of the target base corresponding to the i-th bit segment; determine the number of storage spaces at the i-th level; perform a modulo operation on the number of storage spaces at the i-th level based on the first numerical value, and determine the storage space with the modulo operation result as the serial number as the target storage space at the i-th level.

[0097] Specifically, a modulo operation may be used to determine the target storage space of the i-th level corresponding to the i-th bit segment. First, the i-th bit segment is converted into a target system to obtain a first value. For example, the target system is decimal. Then, a modulo operation is performed in combination with the number of storage spaces of the i-th level.

[0098] Continuing with the above example, use Sum(first 32 bits of MD5)%4 to calculate the serial number of the target storage space at the first level. The first 32 bits of MD5 are the first bit segment of the hash value calculated using the MD5 function, Sum means converting the bit segment to decimal, and %4 means taking the modulo 4.

[0099] It can be seen that in the embodiment of the present application, when determining the serial number of the target storage space of the i-th level, the numerical value corresponding to the i-th bit is modulo the number of storage spaces of the i-th level, which can efficiently determine the corresponding target storage space, and the modulo operation can ensure the even distribution of the storage space to a certain extent.

[0100] S103: Store the target file in the target storage space at the Nth level.

[0101] In the embodiment of the present application, since each storage space of each level is pre-divided, the address of each storage space is also determined for the i-th level. After N operations, the address of the target storage space of the N-th level is finally determined as the target address of the target file. Then the target file is stored in the target storage space of the N-th level.

[0102] Using the data storage method provided in the embodiment of the present application, in response to a storage request for a target file, a hash value is calculated based on a resource locator using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; wherein the target file is obtained based on the resource locator; the i-th level target storage space under the i-1 level storage space is calculated based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1 level storage space is the total storage space; and the target file is stored in the N-th level target storage space.

[0103] The data storage method provided in the embodiment of the present application is used to store massive data. Since the storage space is divided into levels in advance, the hash function is pre-set, and the segmentation of the hash value is specified, when the target file needs to be stored, it is only necessary to perform a hash operation, and then calculate the target storage space at different levels according to the various bit segments contained in the hash value, and finally locate the target storage space at the Nth level, and store the target file in the target storage space at the Nth level.

[0104] It can be seen that the problem that the original metadata could not be directly loaded into the memory for addressing due to the large amount of data is solved by a solution of addressing with a pre-set rule and algorithm. It is not necessary to store the metadata corresponding to the massive data in the memory, saving precious memory resources.

[0105] In one embodiment of the present application, each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs; the address of the target storage space of the N-th level is: the sum of the address offsets corresponding to the target storage spaces of the N levels.

[0106] Since each storage space of each level is pre-divided, the address of each storage space for the i-th level is also determined, that is, each storage space corresponds to an address offset.

[0107] For each level, each storage space corresponds to an address offset, which represents the offset of the address of the storage space relative to the starting address of the storage space of the previous level to which the storage space belongs. As an example, see Figure 2 , is a schematic diagram of the storage space division provided in the embodiment of the present application, such as Figure 2 As shown, the first-level storage space includes A, B, C and D, and each storage space corresponds to an address offset, namely offsetA, offsetB, offsetC and offsetD. The second-level storage space includes bucket1, bucket2, bucket3...bucketM, and each second-level storage space corresponds to an address offset, namely offset1, offset2, offset3...offsetM.

[0108] Accordingly, when determining the address of the target storage space of the Nth level, the starting address of the total storage space is added with the address offset corresponding to the target storage space of the first level, and then the address offset corresponding to the target storage space of the second level is added... and then the address offset corresponding to the target storage space of the Nth level is added to obtain the address of the target storage space of the Nth level.

[0109] It can be seen that in the embodiment of the present application, since each storage space at each level is pre-divided, its offset relative to the starting address of the storage space at the previous level is fixed. Therefore, the addressing result of the target file can be obtained by superimposing the address offsets of the target storage spaces at each level, which can quickly and efficiently determine the address corresponding to the target file and improve the efficiency of storing the target file.

[0110] See also Figure 3 The embodiment of the present application also provides a data reading method, which corresponds to the data storage method provided in the embodiment of the present application. For data stored by the data storage method provided in the embodiment of the present application, when it is necessary to read the target data, the method can be used. The method specifically includes the following steps:

[0111] S301: Obtain a read request for a target file, where the read request carries a resource locator of the target file.

[0112] In one embodiment of the present application, a CDN storage server receives a request initiated by a user device, the request carries a resource locator of a target file, and the request is used to obtain the target file. The CDN server may regard the request as a read request, execute subsequent steps, determine a target storage space, and read the stored target file from the target storage space. If the target file is not stored in the target storage space, the target file is obtained from the network based on the resource locator.

[0113] S302: Based on the resource locator, a hash value is calculated using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1.

[0114] The present application embodiment does not limit the selection of the hash function. As an example, the SHA-1 function, the MD5 function, etc. may be used.

[0115] Since the length of the calculated hash value is fixed, the bit segment division of the hash value can be preset.

[0116] As an example, a 128-bit hash value is calculated based on the MD5 function, and the first 32 bits can be divided into a first bit segment, and the remaining 96 bits can be divided into a second bit segment.

[0117] S303: Calculate the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space.

[0118] In the embodiment of the present application, the storage space used to store the target file is also pre-divided into hierarchical forms.

[0119] Specifically, when the storage system is first started, a large storage block may be allocated according to the disk configuration size as the total storage space, and then hierarchical regions may be divided based on the total storage space.

[0120] For example, if a disk is 850G, 800G of the 850G disk is used as a large storage block, namely the total storage space. The 800G total storage space is further divided into several areas, such as A, B, C, and D, each with 200G. This is the first level of division. Then each 200G storage space is further divided, for example, into M storage spaces, which is the second level of division.

[0121] In the embodiment of the present application, the number of bit segments divided by the hash value is consistent with the number of levels into which the total storage space is divided. For example, if the hash value is divided into 2 bit segments, that is, N=2, then the number of levels into which the total storage space is divided is also 2.

[0122] Thus, a single level of target storage space can be addressed based on each bit segment.

[0123] Specifically, firstly, the target storage space of the first level under the storage space of the 0th level is calculated based on the 1st bit segment, wherein the storage space of the 0th level is the total storage space.

[0124] Continuing with the above example, if a 128-bit hash value is calculated based on the MD5 function, the first 32 bits are divided into the first bit segment, and the remaining 96 bits are divided into the second bit segment. Then the target storage space of the first level under the total storage space is calculated based on the 32 bits contained in the first bit segment. The total storage space is 800G of storage space, and the first level of storage space contains 4 200G storage spaces (i.e., A, B, C, and D storage spaces). Then, the operation is performed based on the first bit segment, and the target storage space of the first level is determined based on the operation result.

[0125] If the target storage space of the first level is determined to be storage space B, the target storage space of the second level is calculated based on the 96 bits contained in the second bit segment. The storage space of the second level includes M storage spaces, each of which has a size of 200G / M. The calculation is performed based on the second bit, and the target storage space of the second level can be determined based on the calculation result.

[0126] By using the data reading method provided in the embodiment of the present application, a storage space at the Nth level is finally determined as the target storage space.

[0127] In one embodiment of the present application, the step of calculating the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment may specifically include:

[0128] Determine a first numerical value of the target base corresponding to the i-th bit segment; determine the number of storage spaces at the i-th level; perform a modulo operation on the number of storage spaces at the i-th level based on the first numerical value, and determine the storage space with the modulo operation result as the serial number as the target storage space at the i-th level.

[0129] Specifically, a modulo operation may be used to determine the target storage space of the i-th level corresponding to the i-th bit segment. First, the i-th bit segment is converted into a target system to obtain a first value. For example, the target system is decimal. Then, a modulo operation is performed in combination with the number of storage spaces of the i-th level.

[0130] Continuing with the above example, use Sum(first 32 bits of MD5)%4 to calculate the serial number of the target storage space at the first level. The first 32 bits of MD5 are the first bit segment of the hash value calculated using the MD5 function, Sum means converting the bit segment to decimal, and %4 means taking the modulo 4.

[0131] It can be seen that in the embodiment of the present application, when determining the serial number of the target storage space of the i-th level, the numerical value corresponding to the i-th bit is modulo the number of storage spaces of the i-th level, which can efficiently determine the corresponding target storage space, and the modulo operation can ensure the even distribution of the storage space to a certain extent.

[0132] S304: Read the target file from the target storage space at the Nth level.

[0133] In the embodiment of the present application, since each storage space of each level is pre-divided, the address of each storage space of the i-th level is also determined. After N operations, the address of the target storage space of the N-th level is finally determined as the target address of the target file. Then the target file is read from the target storage space.

[0134] A data reading method provided in an embodiment of the present application is used to obtain a read request for a target file, wherein the read request carries a resource locator of the target file; based on the resource locator, a hash value is calculated using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; the target storage space of the i-th level under the i-1-th level storage space is calculated based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space; and the target file is read from the target storage space of the N-th level.

[0135] Using the data reading method provided by the embodiment of the present application, the storage space is pre-divided into layers, a hash function is set, the segmentation of the hash value is specified, and the target file is pre-addressed according to specific rules, and the target file is stored in the storage space of the Nth layer. Therefore, when the target file needs to be read, the target file is addressed according to the same rules, and only a hash operation is required, and then the target storage space at different layers is calculated according to each bit segment contained in the hash value, and finally the target storage space of the Nth layer is located, and the target file is read from the target storage space of the Nth layer.

[0136] It can be seen that the problem that the original metadata could not be directly loaded into the memory for addressing due to the large amount of data is solved by a solution of addressing with a pre-set rule and algorithm. It is not necessary to store the metadata corresponding to the massive data in the memory, saving precious memory resources.

[0137] When applied to CDN storage servers, it can quickly query and return data while storing and searching massive amounts of data, which can improve the response speed of CDN storage services and improve service quality.

[0138] In one embodiment of the present application, each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs; the address of the target storage space of the N-th level is: the sum of the address offsets corresponding to the target storage spaces of the N levels.

[0139] Since each storage space of each level is pre-divided, the address of each storage space for the i-th level is also determined, that is, each storage space corresponds to an address offset.

[0140] For each level, each storage space corresponds to an address offset, and the offset represents the offset of the address of the storage space relative to the starting address of the storage space of the previous level to which the storage space belongs.

[0141] Accordingly, when determining the address of the target storage space of the Nth level, the starting address of the total storage space is added with the address offset corresponding to the target storage space of the first level, and then the address offset corresponding to the target storage space of the second level is added... and then the address offset corresponding to the target storage space of the Nth level is added to obtain the address of the target storage space of the Nth level.

[0142] It can be seen that in the embodiment of the present application, since each storage space of each level is pre-divided, its offset relative to the starting address of the storage space of the previous level is fixed. Therefore, the addressing result of the target file can be obtained by superimposing the address offsets of the target storage spaces of each level, which can quickly and efficiently determine the address corresponding to the target file and improve the efficiency of reading the target file.

[0143] In one embodiment of the present application, after determining the target storage space of the Nth level, the target file is read, and the resource locator corresponding to the storage file is read therefrom, and the accuracy is verified based on the read resource locator and the resource locator carried in the read request. If the verification passes, the storage file is determined to be the target file.

[0144] Specifically, the hash values ​​calculated based on the hash function may overlap, that is, different inputs may correspond to the same hash value. If this happens, the wrong file will be read.

[0145] In order to avoid this, at the stage of storing the target file, after determining the target address, the resource locator of the target file can be stored in the storage space corresponding to the target address. Since the storage space used to store the target file in the embodiments of the present application is non-memory storage space, such as disk space, and the disk space is usually much larger than the memory space, the resource locator of the target file, or the correspondence between the resource locator and the target file, can be stored in the disk space.

[0146] Correspondingly, in the stage of reading the target file, after determining the target address, the resource locator is read from the storage space corresponding to the target address, and the accuracy is verified based on the read resource locator and the resource locator included in the read request. If the verification passes, the target file is read.

[0147] It can be seen that by storing the resource locator in the storage space, the correctness of the target file can be further verified, the risk of incorrect data reading due to overlapping hash values ​​can be avoided, and the security of the storage system can be improved.

[0148] Based on the same inventive concept as the data storage method, the present application embodiment also provides a data storage device, see Figure 4 , is a schematic diagram of a structure of a data storage device provided in an embodiment of the present application, comprising the following modules:

[0149] The first calculation module 401 is used to calculate a hash value based on a resource locator using a preset hash function in response to a storage request for a target file; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; wherein the target file is obtained based on the resource locator;

[0150] The second calculation module 402 is used to calculate the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space;

[0151] The storage module 403 is used to store the target file in the target storage space of the Nth level.

[0152] Since the storage space is divided into levels in advance, the hash function is pre-set, and the segmentation of the hash value is specified, when the target file needs to be stored, it is only necessary to perform a hash operation, and then calculate the target storage space at different levels according to the various bit segments contained in the hash value, and finally locate the target storage space at the Nth level, and store the target file in the target storage space at the Nth level.

[0153] It can be seen that the problem that the original metadata could not be directly loaded into the memory for addressing due to the large amount of data is solved by a solution of addressing with a pre-set rule and algorithm. It is not necessary to store the metadata corresponding to the massive data in the memory, saving precious memory resources.

[0154] In one embodiment of the present application, the second calculation module is specifically used to:

[0155] Determine a first value of a target base corresponding to the i-th bit segment;

[0156] Determine the number of storage spaces at level i;

[0157] A modulo operation is performed on the number of storage spaces at the i-th level based on the first value, and a storage space with a sequence number as a result of the modulo operation is determined as a target storage space at the i-th level.

[0158] It can be seen that in the embodiment of the present application, when determining the serial number of the target storage space of the i-th level, the numerical value corresponding to the first bit is modulo the number of storage spaces of the i-th level, which can efficiently determine the corresponding target storage space, and the modulo operation can ensure the even distribution of the storage space to a certain extent.

[0159] In one embodiment of the present application, each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs;

[0160] The address of the target storage space at the Nth level is: the sum of the address offsets corresponding to the target storage spaces at the N levels.

[0161] It can be seen that in the embodiment of the present application, since each storage space at each level is pre-divided, its offset relative to the starting address of the storage space at the previous level is fixed. Therefore, the addressing result of the target file can be obtained by superimposing the address offsets of the target storage spaces at each level, which can quickly and efficiently determine the address corresponding to the target file.

[0162] Based on the same inventive concept as the data reading method, the present application embodiment also provides a data reading device, see Figure 5 , is a structural schematic diagram of a data reading device provided in an embodiment of the present application, comprising the following modules:

[0163] An acquisition module 501 is used to acquire a read request for a target file, wherein the read request carries a resource locator of the target file;

[0164] A third calculation module 502 is used to calculate a hash value based on the resource locator using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1;

[0165] A fourth calculation module 503 is used to calculate the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space;

[0166] The reading module 504 is used to read the target file from the target storage space at the Nth level.

[0167] Using the data reading method provided by the embodiment of the present application, the storage space is pre-divided into layers, a hash function is set, the segmentation of the hash value is specified, and the target file is pre-addressed according to specific rules, and the target file is stored in the storage space of the Nth layer. Therefore, when the target file needs to be read, the target file is addressed according to the same rules, and only a hash operation is required, and then the target storage space at different layers is calculated according to each bit segment contained in the hash value, and finally the target storage space of the Nth layer is located, and the target file is read from the target storage space of the Nth layer.

[0168] It can be seen that the problem that the original metadata could not be directly loaded into the memory for addressing due to the large amount of data is solved by a solution of addressing with a pre-set rule and algorithm. It is not necessary to store the metadata corresponding to the massive data in the memory, saving precious memory resources.

[0169] When applied to CDN storage servers, it can quickly query and return data while storing and searching massive amounts of data, which can improve the response speed of CDN storage services and improve service quality.

[0170] In one embodiment of the present application, the fourth calculation module is specifically used to:

[0171] Determine a first value of a target base corresponding to the i-th bit segment;

[0172] Determine the number of storage spaces at level i;

[0173] A modulo operation is performed on the number of storage spaces at the i-th level based on the first value, and a storage space with a sequence number as a result of the modulo operation is determined as a target storage space at the i-th level.

[0174] It can be seen that in the embodiment of the present application, when determining the serial number of the target storage space of the i-th level, the numerical value corresponding to the i-th bit is modulo the number of storage spaces of the i-th level, which can efficiently determine the corresponding target storage space, and the modulo operation can ensure the even distribution of the storage space to a certain extent.

[0175] In one embodiment of the present application, each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs;

[0176] The address of the target storage space at the Nth level is: the sum of the address offsets corresponding to the target storage spaces at the N levels.

[0177] It can be seen that in the embodiment of the present application, since each storage space of each level is pre-divided, its offset relative to the starting address of the storage space of the previous level is fixed. Therefore, the addressing result of the target file can be obtained by superimposing the address offsets of the target storage spaces of each level, which can quickly and efficiently determine the address corresponding to the target file and improve the efficiency of reading the target file.

[0178] In one embodiment of the present application, the device further includes:

[0179] The verification module is used to read the resource locator corresponding to the storage file from the target storage space of the Nth level, and perform accuracy verification based on the read resource locator and the resource locator carried in the read request. If the verification passes, it is determined that the storage file is the target file.

[0180] It can be seen that by storing the resource locator in the storage space, the correctness of the target file can be further verified, the risk of incorrect data reading due to overlapping hash values ​​can be avoided, and the security of the storage system can be improved.

[0181] The present application also provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0182] Memory 603, used for storing computer programs;

[0183] The processor 601 is used to execute the program stored in the memory 603 to implement the following steps:

[0184] In response to a storage request for a target file, a hash value is calculated based on a resource locator using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; wherein the target file is obtained based on the resource locator;

[0185] The target storage space of the i-th level under the i-1-th level storage space is calculated based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space;

[0186] The target file is stored in the target storage space of the Nth level.

[0187] or,

[0188] Obtaining a read request for a target file, wherein the read request carries a resource locator of the target file;

[0189] Based on the resource locator, a hash value is calculated using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1;

[0190] The target storage space of the i-th level under the i-1-th level storage space is calculated based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space;

[0191] The target file is read from the target storage space at the Nth level.

[0192] The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0193] The communication interface is used for communication between the above terminal and other devices.

[0194] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0195] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0196] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the data storage method and the data reading method described in any one of the above embodiments are implemented.

[0197] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the data storage method and data reading method described in any one of the above embodiments.

[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0199] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0200] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer storage medium, and computer program product embodiments, since they are basically similar to the data storage method and data reading method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0201] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data storage method, characterized in that: The method comprises: In response to a storage request for a target file, a hash value is calculated based on a resource locator using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; wherein the target file is obtained based on the resource locator; The target storage space of the i-th level under the i-1-th level storage space is calculated based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space; The target file is stored in the target storage space of the Nth level.

2. The method according to claim 1, characterized in that The step of calculating the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment includes: Determine a first value of a target base corresponding to the i-th bit segment; Determine the number of storage spaces at level i; A modulo operation is performed on the number of storage spaces at the i-th level based on the first value, and a storage space with a sequence number as a result of the modulo operation is determined as a target storage space at the i-th level.

3. The method according to claim 1 or 2, characterized in that: Each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs; The address of the target storage space at the Nth level is: the sum of the address offsets corresponding to the target storage spaces at the N levels.

4. A data reading method, characterized in that: The method comprises: Obtaining a read request for a target file, wherein the read request carries a resource locator of the target file; Based on the resource locator, a hash value is calculated using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; The target storage space of the i-th level under the i-1-th level storage space is calculated based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space; The target file is read from the target storage space at the Nth level.

5. The method according to claim 4, characterized in that The step of calculating the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment includes: Determine a first value of a target base corresponding to the i-th bit segment; Determine the number of storage spaces at level i; A modulo operation is performed on the number of storage spaces at the i-th level based on the first value, and a storage space with a sequence number as a result of the modulo operation is determined as a target storage space at the i-th level.

6. The method according to claim 4 or 5, characterized in that: Each storage space of the i-th level corresponds to an address offset; the address offset represents the offset of the address of the storage space relative to the starting address of the storage space of the i-1th level to which the storage space belongs; The address of the target storage space at the Nth level is: the sum of the address offsets corresponding to the target storage spaces at the N levels.

7. The method according to claim 4, characterized in that The method further comprises: The resource locator corresponding to the storage file is read from the target storage space of the Nth level, and the accuracy is verified based on the read resource locator and the resource locator carried in the read request. If the verification passes, the storage file is determined to be the target file.

8. A data storage device, characterized in that: The device comprises: A first calculation module, configured to calculate a hash value based on a resource locator and in response to a storage request for a target file, using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; wherein the target file is obtained based on the resource locator; The second calculation module is used to calculate the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space; The storage module is used to store the target file in the target storage space of the Nth level.

9. A data reading device, characterized in that: The device comprises: An acquisition module, used for acquiring a read request for a target file, wherein the read request carries a resource locator of the target file; A third calculation module, configured to calculate a hash value based on the resource locator using a preset hash function; the hash value is composed of N bit segments; N is a positive integer greater than or equal to 1; A fourth calculation module is used to calculate the target storage space of the i-th level under the i-1-th level storage space based on the i-th bit segment in sequence; i is a positive integer greater than or equal to 1 and less than or equal to N; when i is 1, the i-1-th level storage space is the total storage space; The reading module is used to read the target file from the target storage space at the Nth level.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-3 or 4-7 when executing a program stored in a memory.

Images