Data storage method and device, equipment, medium and program product

Abstract

The embodiment of the invention discloses a data storage method and device, equipment, a medium and a program product. The method comprises the steps of obtaining to-be-stored target data; based on multiple first compression modes, performing parallel compression processing on the target data, and based on the actual compression speed of each first compression mode, determining a target first compression mode corresponding to the target data and a first compressed file compressed based on the target first compression mode; determining a first index value corresponding to the first compressed file, wherein the first index value comprises identification information of a first storage bucket allocated at the time; and based on the first index value, storing the first compressed file into a first storage bucket in the object storage service, establishing a target mapping relationship between the first index value and the identification information of the target data, and storing the target mapping relationship into a database. Through the technical scheme of the embodiment of the invention, the data storage cost can be effectively reduced, and the data storage efficiency and management convenience are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to Internet technology, and in particular to a data storage method, device, equipment, medium and program product. Background Art

[0002] With the in-depth popularization of Internet technology and the explosive growth of digital information, we have entered an era of massive data. In this era, data storage is not only related to the operational efficiency of enterprises, but also becomes a major cost expenditure.

[0003] At present, facing such a huge amount of data, cloud storage such as Object Storage Service (OSS) can be used for data storage. However, in the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] As the amount of data continues to grow, the cost of OSS storage is also showing a trend of gradual increase. It can be seen that there is an urgent need for a method that can effectively reduce the cost of OSS storage while improving data storage efficiency and management convenience. Summary of the invention

[0005] Embodiments of the present invention provide a data storage method, apparatus, device, medium and program product to effectively reduce data storage costs and improve data storage efficiency and management convenience.

[0006] In a first aspect, an embodiment of the present invention provides a data storage method, including:

[0007] Obtain target data to be stored;

[0008] Based on multiple first compression methods, the target data is compressed in parallel, and based on the actual compression speed of each first compression method, a target first compression method corresponding to the target data and a first compressed file compressed based on the target first compression method are determined;

[0009] Determine a first index value corresponding to the first compressed file, where the first index value includes identification information of a first storage bucket allocated at that time;

[0010] Based on the first index value, the first compressed file is stored in the first storage bucket in the object storage service, and a target mapping relationship between the first index value and the identification information of the target data is established, and the target mapping relationship is stored in a database.

[0011] In a second aspect, an embodiment of the present invention further provides a data storage device, including:

[0012] A target data acquisition module, used to acquire target data to be stored;

[0013] A first compression processing module, configured to perform parallel compression processing on the target data based on a plurality of first compression methods, and determine a target first compression method corresponding to the target data and a first compressed file compressed based on the target first compression method based on an actual compression speed of each first compression method;

[0014] A first index value determination module, configured to determine a first index value corresponding to the first compressed file, wherein the first index value includes identification information of a first storage bucket allocated at that time;

[0015] The first compressed file storage module is used to store the first compressed file in the first storage bucket in the object storage service based on the first index value, establish a target mapping relationship between the first index value and the identification information of the target data, and store the target mapping relationship in a database.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] one or more processors;

[0018] A memory for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method provided by any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data storage method provided by any embodiment of the present invention.

[0021] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data storage method provided by any embodiment of the present invention.

[0022] One embodiment of the above invention has the following advantages or beneficial effects:

[0023] By performing parallel compression processing on the target data to be stored based on multiple first compression methods, and determining the target first compression method corresponding to the target data and the first compressed file compressed based on the target first compression method based on the actual compression speed of each first compression method, the dynamic selection of the first compression method is realized, and the first compressed file with the fastest actual compression speed is obtained, thereby improving the data compression efficiency and storage efficiency. By determining the first index value corresponding to the first compressed file, the first index value includes the identification information of the first storage bucket allocated at that time, and based on the first index value, the first compressed file is stored in the first storage bucket in the object storage service, so that the storage space can be reduced by storing the compressed file, and the data storage cost can be further effectively reduced. By using the index value to allocate the storage bucket, the efficient management of multiple storage buckets in the object storage service can be realized, and by establishing the target mapping relationship between the first index value and the identification information of the target data, the target mapping relationship is stored in the database, so that the data can be efficiently read later, thereby improving the convenience of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 is a flow chart of a data storage method provided by an embodiment of the present invention;

[0026] Figure 2 is an example diagram of a data upload and download process involved in an embodiment of the present invention;

[0027] Figure 3 This is an example diagram of a storage bucket allocation storage involved in an embodiment of the present invention;

[0028] Figure 4 is a flow chart of another data storage method provided by an embodiment of the present invention;

[0029] Figure 5 is an example diagram of a conversion process in a data storage phase involved in an embodiment of the present invention;

[0030] Figure 6 It is a structural schematic diagram of a data storage device provided by an embodiment of the present invention;

[0031] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0033] Figure 1 This is a flowchart of a data storage method provided by an embodiment of the present invention. This embodiment is applicable to the case of storing any data in an object storage service. The method can be executed by a data storage device, which can be implemented by software and / or hardware and integrated in an electronic device. Figure 1 As shown, the method specifically comprises the following steps:

[0034] S110: Obtain target data to be stored.

[0035] The target data may refer to data in any format that needs to be stored, for example, audio or video data, pictures or documents.

[0036] Specifically, the server can receive a data storage request sent by the client, and the data storage request can also be regarded as a data upload request. The server parses the received data storage request to obtain the target data that needs to be stored currently.

[0037] S120, performing parallel compression processing on the target data based on multiple first compression methods, and determining a target first compression method corresponding to the target data and a first compressed file compressed based on the target first compression method based on an actual compression speed of each first compression method.

[0038] Among them, each first compression method may refer to a compression algorithm that focuses more on compression speed. The compression speed of the first compression method is better than the compression rate. The minimum compression speed of the first compression method is greater than or equal to the preset compression speed. In other words, the first compression method refers to a compression algorithm with a relatively fast compression speed. In addition to the minimum compression speed of the first compression method being greater than or equal to the preset compression speed, the minimum compression rate of the first compression method may also be required to be greater than or equal to the preset compression rate, so as to avoid the situation where the compression volume does not change significantly due to the compression rate being too small, and further ensure the effect of reducing storage costs. For example, the first compression method may be but is not limited to the LZ4 compression algorithm. The actual compression speed of the first compression method refers to the actual compression speed of the target data actually compressed on the server side. The target first compression method refers to the first compression method with the fastest compression speed. The first compressed file refers to a data file obtained by compressing the target data using the target first compression method with the fastest compression speed.

[0039] It should be noted that factors such as data type and data volume will affect the compression speed of the first compression method. That is to say, different compression speeds may be achieved when compressing different data using the same first compression method. Therefore, it is necessary to determine the actual compression speed of each first compression method in real time and dynamically determine the target first compression method that compresses the current data the fastest.

[0040] Specifically, based on a plurality of different first compression methods, the target data is compressed in parallel, the actual compression speed of each first compression method when compressing the target data is determined, and the first compression method with the fastest actual compression speed is determined as the target first compression method corresponding to the target data, and the file after the target data is compressed using the target first compression method is used as the first compressed file, thereby realizing dynamic and intelligent selection of the target first compression method, ensuring the high efficiency of data compression, and thereby improving data storage efficiency.

[0041] Exemplarily, step S120 may include: starting multiple compression processes that are executed concurrently, and performing parallel compression processing on the target data through different compression processes based on different first compression methods; in response to the target compression process that is completed first, terminating other compression processes, and determining the first compression method used by the target compression process as the target first compression method corresponding to the target data, and obtaining the first compressed file compressed by the target compression process.

[0042] Specifically, the server can start multiple compression processes that are executed concurrently. The compression processes correspond to the first compression methods one by one. In each compression process, a first compression method is used to compress the target data. By executing multiple compression processes concurrently, parallel compression processing of multiple first compression methods can be achieved. By detecting the completion progress of each compression process, it is determined whether there is a completed compression process at present. If there is a compression process that is completed first, that is, the target compression process, then other compression processes except the target compression process, that is, the compression process that has not yet completed compression, can be immediately terminated to save compression resources. The target compression process that is completed first is the compression process with the fastest actual compression speed, so that the first compression method used by the target compression process is determined as the target first compression method with the fastest actual compression speed, and the file compressed by the target compression process is used as the first compressed file. By compressing the target data using the target first compression method with the fastest compression speed, the efficiency of writing the target data can be guaranteed, thereby improving the overall performance of data storage.

[0043] S130: Determine a first index value corresponding to the first compressed file, where the first index value includes identification information of a first storage bucket allocated at that time.

[0044] Bucket refers to the container used to store data in the object storage service OSS. Buckets can store various types of data, such as pictures, audio, video, and documents. It should be noted that since using a single bucket for data storage may be limited by network traffic, resulting in insufficient throughput, multiple buckets in the object storage service OSS are used for data storage to improve data read and write performance.

[0045] The first storage bucket refers to a storage bucket allocated for the target data. The identification information of the first storage bucket may be information uniquely identified by at least one of numbers, letters, and words, so as to distinguish different storage buckets based on the identification information. The first index value may be an index value used to query the first compressed file. Different compressed files correspond to different index values, so as to distinguish different compressed files by using the index values.

[0046] Specifically, when the first compressed file needs to be stored, a first storage bucket may be allocated for the first compressed file for storage. For example, the storage bucket may be allocated based on the current remaining storage capacity of each storage bucket in the object storage service. The storage bucket with the largest amount of current remaining space is used as the first storage bucket allocated at the time, and the first index value corresponding to the first compressed file is determined based on the identification information of the first storage bucket. For example, the identification information of the first storage bucket may be combined with a randomly generated string to obtain the first index value.

[0047] S140. Based on the first index value, store the first compressed file in a first storage bucket in the object storage service, establish a target mapping relationship between the first index value and identification information of the target data, and store the target mapping relationship in a database.

[0048] The identification information of the target data may be unique identification information for distinguishing the target data. For example, if the target data is order data, the identification information of the target data may refer to the order number. The database may be a relational database for storing data. For example, the database may be, but is not limited to, a Mysql database.

[0049] Specifically, the server determines the address information of the first storage bucket based on the identification information of the first storage bucket in the first index value, and stores the first compressed file in the first storage bucket under the address information, thereby realizing intelligent bucket storage of compressed data, avoiding the large amount of physical storage space usage caused by directly storing the original data, and thus reducing the data storage cost. By using the first index value as key information and the identification information of the target data as value information, the key-value pair information obtained can be used as the target mapping relationship between the first index value and the identification information of the target data. By storing the key-value pair information, the target mapping relationship can be stored in the database, so that the data can be efficiently read based on the mapping relationship in the database later, thereby improving the convenience of data management.

[0050] For example, see Figure 2 , the server compresses the target data that needs to be stored uploaded by the client by executing the above steps S110-S140, and stores the compressed first compressed file in the object storage service, thereby automatically completing the compression process before data storage without the need for client participation. When the client needs to download the target data, the server can retrieve the compressed first compressed file from the object storage service, decompress the first compressed file, obtain the target data, and send the target data to the client, thereby automatically completing the decompression process before data download without the need for client participation. Among them, the download and reading process of the target data can be: based on the identification information of the target data, query the first index value corresponding to the identification information of the target data from the database, and based on the identification information of the first storage bucket in the first index value, retrieve the first compressed file from the first storage bucket, and decompress the first compressed file based on the decompression method corresponding to the target first compression method to obtain the decompressed target data. Exemplarily, the first index value or the remaining information in the first index value except the identification information of the first storage bucket can be used as the file name of the first compressed file, and the identification information of the target first compression method can be used as the extension of the first compressed file, so that the compression method used by the first compressed file can be determined more quickly based on the extension, thereby achieving efficient and accurate decompression operations.

[0051] It should be noted that since the data to be stored with a smaller data volume often has a higher reading frequency or access frequency, this embodiment is more suitable for the storage of data to be stored with a smaller data volume. For example, the data volume of the target data to be stored is less than the preset data volume, thereby effectively reducing the data storage cost increased by frequent reading or access in the object storage service.

[0052] The technical solution of this embodiment is to perform parallel compression processing on the target data to be stored based on multiple first compression methods, and determine the target first compression method corresponding to the target data and the first compressed file compressed based on the target first compression method based on the actual compression speed of each first compression method, so as to realize the dynamic selection of the first compression method, obtain the first compressed file with the fastest actual compression speed, and thus improve the data compression efficiency and storage efficiency. By determining the first index value corresponding to the first compressed file, the first index value includes the identification information of the first storage bucket allocated at that time, and based on the first index value, the first compressed file is stored in the first storage bucket in the object storage service, so that the storage space can be reduced by storing the compressed file, and the data storage cost can be further effectively reduced. By using the index value to allocate the storage bucket, the efficient management of multiple storage buckets in the object storage service can be realized, and by establishing the target mapping relationship between the first index value and the identification information of the target data, the target mapping relationship is stored in the database, so that the data can be efficiently read later, thereby improving the convenience of data management.

[0053] Based on the above technical solution, step S130 may include: generating a target hash value and a target string corresponding to the target data; performing modulo processing on the target hash value based on the number of buckets in the object storage service, and using the modulo result as the identification information of the first bucket allocated at that time; determining the first index value corresponding to the first compressed file based on the identification information of the first bucket and the target string.

[0054] Specifically, a target hash value can be randomly generated for the target data through a hash function, and the target hash value is a random value. The target hash value is divided by the number of buckets for modulo processing, and the remainder obtained is used as the identification information of the first bucket allocated at that time. The identification information of each bucket is represented by a value from 0 to the number of buckets - 1. For example, if the number of buckets is 4, the identification information corresponding to the first bucket is 0, the identification information corresponding to the second bucket is: 1, the identification information corresponding to the third bucket is: 2, and the identification information corresponding to the fourth bucket is: 3. A target string is randomly generated for the target data, and the identification information and underscore of the first bucket are concatenated with the target string to obtain the first index value corresponding to the first compressed file, such as Figure 3 The target string can be used to distinguish different compressed files stored in the same bucket. Figure 3, the target mapping relationship between each index value and the data identification information can be stored in the database. Accordingly, when it is necessary to read the target data from the object storage service, the first index value corresponding to the identification information of the target data can be queried from the database, and the first index value can be segmented based on the underscore to extract the identification information of the first storage bucket in the first index value, and the address information of the first storage bucket can be determined based on the identification information, and the first compressed file corresponding to the target data can be taken out from the first storage bucket corresponding to the address information in the object storage service, such as taking out the first compressed file named as the first index value, and decompressing the first compressed file based on the decompression method corresponding to the target first compression method to obtain the final target data. By utilizing the index value method, multiple storage buckets can be efficiently managed, and the overall read and write performance can be fully improved.

[0055] Figure 4 This is a flowchart of another data storage method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment describes in detail the process of recompressing and storing the stored first compressed file when the current storage duration of the target data exceeds the first preset duration. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.

[0056] See also Figure 4 Another data storage method provided in this embodiment specifically includes the following steps:

[0057] S410: Acquire target data to be stored.

[0058] S420, based on multiple first compression methods, compress the target data in parallel, and determine the target first compression method corresponding to the target data and the first compressed file compressed based on the target first compression method based on the actual compression speed of each first compression method.

[0059] S430: Determine a first index value corresponding to the first compressed file, where the first index value includes identification information of a first storage bucket allocated at that time.

[0060] S440: Based on the first index value, store the first compressed file in a first storage bucket in the object storage service, establish a target mapping relationship between the first index value and identification information of the target data, and store the target mapping relationship in a database.

[0061] It should be noted that, since the newly stored data may be frequently accessed, when the first compressed file is stored in the first storage bucket, the first compressed file may be stored in a standard type of storage method, so that no excessive fees are charged when retrieving the first compressed file, further reducing storage costs. Figure 5The data storage process described in the above steps S410-S440 is equivalent to the first storage stage of the data. The first storage stage may refer to a hot data storage stage, that is, a stage in which the data storage duration is less than the first preset duration. In the first storage stage, the target data is quickly compressed in the first compression method with the fastest compression speed, and the first compressed file is stored in the first storage bucket in a standard type, thereby achieving efficient data storage and saving storage costs.

[0062] S450: In response to the current storage duration of the target data exceeding the first preset duration, based on the identification information of the target data and the target mapping relationship, retrieve the first compressed file from the object storage service, decompress the first compressed file, and obtain the target data.

[0063] The first preset duration refers to a critical duration for switching the data compression method.

[0064] Specifically, it is possible to detect in real time whether the current storage duration of the target data exceeds the first preset duration to determine whether to enter the second storage stage from the first storage stage. For example, in the process of storing the first compressed file corresponding to the target data, the data detection task corresponding to the target data can be generated synchronously based on the identification information and creation time (i.e., storage start time) of the target data, and the data detection task can be stored in the task table, and the current storage duration of the target data is detected by executing the data detection task in the task table in real time or periodically. In the case where it is detected that the current storage duration of the target data exceeds the first preset duration, it indicates that it is necessary to switch the compression method, that is, switch from a compression method focusing on compression speed to a compression method focusing on compression rate, so as to further reduce the storage cost, that is, it is necessary to enter the second storage stage from the first storage stage. The second storage stage may refer to a warm data storage stage, that is, a stage where the data storage duration is greater than the first preset duration. In the second storage stage, it is necessary to first retrieve the first compressed file from the object storage service based on the identification information of the target data and the target mapping relationship stored in the database. For example, based on the identification information of the target data and the target mapping relationship stored in the database, determine the first index value corresponding to the identification information of the target data, determine the first storage bucket in the object storage service client based on the identification information of the first storage bucket in the first index value, and take out the first compressed file from the first storage bucket. The file name of the first compressed file can be determined based on the first index value. For example, directly use the first index value as the file name of the first compressed file, or use the remaining information in the first index value except the identification information of the first storage bucket as the file name of the first compressed file. By determining the file name of the first compressed file based on the first index value, the first compressed file is taken out from the first storage bucket based on the file name. Based on the decompression method corresponding to the target first compression method, the first compressed file is decompressed to obtain the decompressed target data.

[0065] Exemplarily, the extension of the first compressed file is identification information of the target first compression mode; "decompressing the first compressed file to obtain target data" in step S450 may include: determining the target first compression mode corresponding to the first compressed file based on the extension of the first compressed file; and decompressing the first compressed file based on the decompression mode corresponding to the target first compression mode to obtain the target data.

[0066] Specifically, by using the identification information of the target first compression method as the extension of the first compressed file, the compression method used by the first compressed file, that is, the target first compression method, can be determined more quickly and accurately based on the extension, and based on the decompression method corresponding to the target first compression method, the first compressed file is decompressed to obtain the target data, thereby achieving efficient and accurate decompression operations.

[0067] Exemplarily, the identification information of the target first compression method can also be used as the extension of the first index value, and the first index value with the extension can be stored in the database, so that the compression method used for the current compressed target data can be quickly and accurately determined based on the extension of the first index value, thereby improving the decompression speed of the compressed file and obtaining the target data efficiently and quickly.

[0068] S460: Based on the multiple second compression methods, the target data is compressed in parallel, and based on the actual compression rate of each second compression method, a target second compression method corresponding to the target data and a second compressed file compressed based on the target second compression method are determined.

[0069] Among them, each second compression method may refer to a compression algorithm that focuses more on compression rate. The compression rate of the second compression method is better than the compression speed. The minimum compression rate of the second compression method is greater than or equal to the preset compression rate. In other words, the second compression method refers to a compression algorithm with a relatively high compression rate. For example, the second compression method may be but is not limited to the Deflate compression algorithm and the ZIP compression algorithm in the Zlib compression library. The actual compression rate of the second compression method refers to the actual compression rate of the target data actually compressed on the server side. The target second compression method refers to the second compression method with the highest compression rate. The second compressed file refers to a data file obtained by compressing the target data using the target second compression method with the highest compression rate.

[0070] It should be noted that factors such as data type and data volume will affect the compression rate of the second compression method. That is to say, different compression rates may be achieved when compressing different data using the same second compression method. Therefore, it is necessary to determine the actual compression rate of each second compression method in real time and dynamically determine the target second compression method with the highest current compression rate.

[0071] Specifically, after the first compressed file corresponding to the target data is stored in the object storage service for a period of time, the access frequency of the first compressed file will gradually decrease. At this time, it is necessary to take out the first compressed file with the fastest compression speed and decompress it to obtain the original target data. Then, based on a plurality of different second compression methods, the decompressed target data is compressed in parallel, and the actual compression rate of each second compression method when compressing the target data is determined, and the second compression method with the highest actual compression rate is determined as the target second compression method corresponding to the target data, and the file after the target data is compressed using the target second compression method is used as the second compressed file, so as to further reduce the data volume of the compressed file and ensure that the storage cost reaches the optimal level.

[0072] Exemplarily, step S460 may include: starting multiple compression processes that are executed concurrently, and performing parallel compression processing on the target data through different compression processes based on different second compression methods; in response to the completion of all compression processes, obtaining the file volume of the compressed file compressed by each compression process, and using the compressed file with the smallest file volume as the second compressed file, and using the second compression method used by the second compressed file as the target second compression method corresponding to the target data.

[0073] Specifically, the server can start multiple compression processes that are executed concurrently. The compression processes correspond to the second compression methods one by one. In each compression process, a second compression method is used to compress the target data. By executing multiple compression processes concurrently, parallel compression processing of multiple second compression methods can be achieved. By detecting the completion progress of each compression process, it is determined whether each compression process has completed the compression operation. If all compression processes have been completed, the file volume (i.e., file size) of the compressed file compressed by each compression process can be obtained, and the compressed file with the smallest file volume is used as the second compressed file. The second compression method used is the target second compression method with the highest actual compression rate, so that the target second compression method with the highest actual compression rate and the second compressed file with the smallest file volume can be obtained. By compressing the target data using the target second compression method with the highest compression rate, the data storage amount can be further reduced, thereby further optimizing the storage cost.

[0074] S470: Based on the first index value, store the second compressed file in a first storage bucket in the object storage service.

[0075] Specifically, since the first compressed file has been taken out of the first storage bucket, the object storage service does not store the first compressed file, so the second compressed file can be directly stored in the first storage bucket in the object storage service instead of the first compressed file based on the identification information of the first storage bucket in the first index value, without the need to determine a new index value, thereby improving storage efficiency. It should be noted that the file names of the first compressed file and the second compressed file can be the same, but the extension of the second compressed file and the extension of the first index value need to be updated to the identification information of the target second compression method, so that the compression method of the stored second compressed file can be accurately and quickly obtained based on the updated extension.

[0076] Exemplarily, step S470 may include: determining a second index value corresponding to the second compressed file, the second index value including identification information of the second storage bucket allocated at that time; based on the second index value, storing the second compressed file in the second storage bucket in the object storage service, and updating the target mapping relationship in the database based on the second index value.

[0077] Specifically, a more suitable storage bucket can be reassigned to the second compressed file, such as the storage bucket with the largest amount of remaining space as the second storage bucket, and a new second index value can be obtained based on the identification information of the second storage bucket and a randomly generated target string, so as to further improve the overall storage performance. The server can determine the address information of the second storage bucket based on the identification information of the second storage bucket in the second index value, and store the second compressed file with a smaller compressed volume in the second storage bucket under the address information, thereby further reducing the data storage cost. Since the second index value is different from the first index value, it is necessary to update the first index value in the target mapping relationship stored in the database to the second index value, so that the second compressed file can be accurately read based on the updated target mapping relationship.

[0078] It should be noted that see Figure 5 In the second storage stage, the target data is compressed at a greater rate using the target second compression method with the highest compression rate to further reduce the amount of storage space, and the second compressed file can continue to be stored in the object storage service in a standard type, thereby further saving storage costs.

[0079] Exemplarily, after the target data is stored in the object storage service as a second compressed file, the target data is read in the same manner as the data when stored as a first compressed file. For example, based on the identification information of the target data, the second index value corresponding to the identification information of the target data is queried from the database, and based on the identification information of the second storage bucket in the second index value, the second compressed file is retrieved from the second storage bucket. Based on the extension of the second compressed file or the extension of the second index value, the target second compression method used by the second compressed file is determined, and based on the decompression method corresponding to the target second compression method, the second compressed file is decompressed to obtain the decompressed target data.

[0080] The technical solution of this embodiment is to obtain the target data by taking out the first compressed file from the object storage service and decompressing the first compressed file when the current storage time of the target data exceeds the first preset time; based on a plurality of different second compression methods, the target data is compressed in parallel, and the target second compression method with the highest actual compression rate and the second compressed file with the smallest compression volume are determined based on the actual compression rate of each second compression method, thereby realizing the dynamic intelligent selection of the second compression method. By storing the second compressed file with a smaller compression volume in the object storage service, the data volume of the compressed file can be further reduced, and dynamic storage conversion can be performed as the data life cycle evolves to ensure that the storage cost reaches the optimal level.

[0081] Based on the above technical solutions, after storing the second compressed file in the object storage service, it can also include: in response to the current storage time of the target data exceeding the second preset time, or the current access frequency of the target data is lower than the preset access frequency, converting the storage type of the second compressed file from the standard type to the archive type for storage.

[0082] The second preset duration refers to the critical value of the duration for which the data storage type needs to be converted. Both the standard type and the archive type are data storage types provided by the object storage service. The archive type may include an archive type, a cold archive type, or a deep cold archive type provided by the object storage service.

[0083] Specifically, see Figure 5 As time goes by, the data changes from the first storage stage to the second storage stage, that is, from the mode focusing on compression speed to the mode focusing on high compression rate. It is possible to continue to perform data detection tasks to detect whether the current storage duration of the target data exceeds the second preset duration, or whether the current access frequency of the target data is lower than the preset access frequency. When it is detected that the current storage duration of the target data exceeds the second preset duration, or the current access frequency of the target data is lower than the preset access frequency, it indicates that the data utilization rate is low, and it is necessary to enter the third storage stage from the second storage stage. At this time, the data storage type can be changed, and the storage type of the second compressed file can be converted from the standard type to the archive type and stored in the object storage service to achieve efficient storage management and further reduce the data storage cost. Among them, the third storage stage can refer to the cold data storage stage, that is, the stage where the data storage duration is greater than the second preset duration or the current access frequency of the target data is lower than the preset access frequency. By using three storage stages, a smooth transition of data from hot storage to warm storage and then to cold storage can be achieved, thereby achieving the optimization of cost use. As time goes by and the data usage changes, the technical solution of this embodiment can automatically and flexibly convert data between different compression methods and storage types in a closed loop, thereby significantly reducing the data storage cost.

[0084] The following is an embodiment of a data storage device provided in an embodiment of the present invention. The device and the data storage methods of the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the data storage device, reference can be made to the embodiments of the above-mentioned data storage methods.

[0085] Figure 6 This is a schematic diagram of the structure of a data storage device provided by an embodiment of the present invention. This embodiment is applicable to the case where any data is stored in an object storage service. Figure 6As shown, the device specifically includes: a target data acquisition module 610, a first compression processing module 620, a first index value determination module 630 and a first compressed file storage module 640.

[0086] Among them, the target data acquisition module 610 is used to acquire the target data to be stored; the first compression processing module 620 is used to perform parallel compression processing on the target data based on multiple first compression methods, and determine the target first compression method corresponding to the target data and the first compressed file compressed based on the target first compression method based on the actual compression speed of each first compression method; the first index value determination module 630 is used to determine the first index value corresponding to the first compressed file, and the first index value includes the identification information of the first storage bucket allocated at that time; the first compressed file storage module 640 is used to store the first compressed file in the first storage bucket in the object storage service based on the first index value, and establish a target mapping relationship between the first index value and the identification information of the target data, and store the target mapping relationship in the database.

[0087] The technical solution of this embodiment is to perform parallel compression processing on the target data to be stored based on multiple first compression methods, and determine the target first compression method corresponding to the target data and the first compressed file compressed based on the target first compression method based on the actual compression speed of each first compression method, so as to realize the dynamic selection of the first compression method, obtain the first compressed file with the fastest actual compression speed, and thus improve the data compression efficiency and storage efficiency. By determining the first index value corresponding to the first compressed file, the first index value includes the identification information of the first storage bucket allocated at that time, and based on the first index value, the first compressed file is stored in the first storage bucket in the object storage service, so that the storage space can be reduced by storing the compressed file, and the data storage cost can be further effectively reduced. By using the index value to allocate the storage bucket, the efficient management of multiple storage buckets in the object storage service can be realized, and by establishing the target mapping relationship between the first index value and the identification information of the target data, the target mapping relationship is stored in the database, so that the data can be efficiently read later, thereby improving the convenience of data management.

[0088] Optionally, the first compression processing module 620 is specifically configured to:

[0089] Start multiple compression processes for concurrent execution, and perform parallel compression processing on the target data through different compression processes based on different first compression methods; in response to the target compression process that is completed first, terminate other compression processes, and determine the first compression method used by the target compression process as the target first compression method corresponding to the target data, and obtain the first compressed file compressed by the target compression process.

[0090] Optionally, the first index value determining module 630 is specifically configured to:

[0091] Generate a target hash value and a target character string corresponding to the target data; perform modulo processing on the target hash value based on the number of buckets in the object storage service, and use the modulo result as the identification information of the first bucket allocated at that time; determine a first index value corresponding to the first compressed file based on the identification information of the first bucket and the target character string.

[0092] Optionally, the device further comprises:

[0093] A first compressed file decompression module is configured to retrieve the first compressed file from the object storage service based on the identification information of the target data and the target mapping relationship in response to the current storage duration of the target data exceeding a first preset duration, and decompress the first compressed file to obtain the target data;

[0094] A second compression processing module, configured to perform parallel compression processing on the target data based on a plurality of second compression methods, and determine a target second compression method corresponding to the target data and a second compressed file compressed based on the target second compression method based on an actual compression ratio of each second compression method;

[0095] The second compressed file storage module is used to store the second compressed file in the first storage bucket in the object storage service based on the first index value.

[0096] Optionally, the extension of the first compressed file is identification information of the target first compression mode;

[0097] The first compressed file decompression module is specifically used to: determine the target first compression mode corresponding to the first compressed file based on the extension name of the first compressed file; and decompress the first compressed file based on the decompression mode corresponding to the target first compression mode to obtain the target data.

[0098] Optionally, the second compression processing module is specifically configured to:

[0099] Start multiple compression processes for concurrent execution, and perform parallel compression processing on the target data through different compression processes based on different second compression methods; in response to the completion of all compression processes, obtain the file size of the compressed file compressed by each compression process, and use the compressed file with the smallest file size as the second compressed file, and use the second compression method used by the second compressed file as the target second compression method corresponding to the target data.

[0100] Optionally, the second compressed file storage module is specifically used to:

[0101] Determine a second index value corresponding to the second compressed file, where the second index value includes identification information of the second storage bucket allocated at that time; based on the second index value, store the second compressed file in the second storage bucket in the object storage service, and update the target mapping relationship in the database based on the second index value.

[0102] Optionally, the device further comprises:

[0103] A storage type conversion module is used to, after storing the second compressed file in the object storage service, convert the storage type of the second compressed file from a standard type to an archive type for storage in response to the current storage duration of the target data exceeding a second preset duration, or the current access frequency of the target data being lower than a preset access frequency; wherein the second preset duration is greater than the first preset duration.

[0104] The data storage device provided in the embodiment of the present invention can execute the data storage method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data storage method.

[0105] It is worth noting that in the embodiment of the above-mentioned data storage device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0106] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 7 A block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 7 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0107] like Figure 7 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0108] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0109] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0110] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory 30 (RAM) and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0111] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0112] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0113] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a data storage method provided by any embodiment of the present invention, the method comprising:

[0114] Obtain target data to be stored;

[0115] Based on multiple first compression methods, the target data is compressed in parallel, and based on the actual compression speed of each first compression method, a target first compression method corresponding to the target data and a first compressed file compressed based on the target first compression method are determined;

[0116] Determine a first index value corresponding to the first compressed file, where the first index value includes identification information of a first storage bucket allocated at that time;

[0117] Based on the first index value, the first compressed file is stored in the first storage bucket in the object storage service, and a target mapping relationship between the first index value and the identification information of the target data is established, and the target mapping relationship is stored in a database.

[0118] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data storage method provided by any embodiment of the present invention.

[0119] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the data storage method provided in any embodiment of the present invention are implemented. The method includes:

[0120] Obtain target data to be stored;

[0121] Based on multiple first compression methods, the target data is compressed in parallel, and based on the actual compression speed of each first compression method, a target first compression method corresponding to the target data and a first compressed file compressed based on the target first compression method are determined;

[0122] Determine a first index value corresponding to the first compressed file, where the first index value includes identification information of a first storage bucket allocated at that time;

[0123] Based on the first index value, the first compressed file is stored in the first storage bucket in the object storage service, and a target mapping relationship between the first index value and the identification information of the target data is established, and the target mapping relationship is stored in a database.

[0124] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0125] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0126] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0127] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data storage method provided by any embodiment of the present invention.

[0129] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0130] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0131] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data storage method, characterized in that: include: Obtain target data to be stored; Based on multiple first compression methods, the target data is compressed in parallel, and based on the actual compression speed of each first compression method, a target first compression method corresponding to the target data and a first compressed file compressed based on the target first compression method are determined; Determine a first index value corresponding to the first compressed file, where the first index value includes identification information of a first storage bucket allocated at that time; Based on the first index value, the first compressed file is stored in the first storage bucket in the object storage service, and a target mapping relationship between the first index value and the identification information of the target data is established, and the target mapping relationship is stored in a database.

2. The data storage method according to claim 1, characterized in that: The method of compressing the target data in parallel based on multiple first compression methods, and determining a target first compression method corresponding to the target data and a first compressed file compressed based on the target first compression method based on an actual compression speed of each first compression method, includes: Initiate a plurality of compression processes for concurrent execution, and perform parallel compression processing on the target data through different compression processes based on different first compression modes; In response to the first completed target compression process, other compression processes are terminated, and the first compression method used by the target compression process is determined as the target first compression method corresponding to the target data, and the first compressed file compressed by the target compression process is obtained.

3. The data storage method according to claim 1, characterized in that: The determining a first index value corresponding to the first compressed file includes: Generate a target hash value and a target character string corresponding to the target data; Based on the number of storage buckets in the object storage service, the target hash value is modulo processed, and the modulo result is used as the identification information of the first storage bucket allocated at that time; Based on the identification information of the first storage bucket and the target character string, a first index value corresponding to the first compressed file is determined.

4. The data storage method according to any one of claims 1 to 3, characterized in that: The method further comprises: In response to a current storage duration of the target data exceeding a first preset duration, based on the identification information of the target data and the target mapping relationship, taking out the first compressed file from the object storage service, and decompressing the first compressed file to obtain the target data; Based on multiple second compression methods, the target data is compressed in parallel, and based on the actual compression rate of each second compression method, a target second compression method corresponding to the target data and a second compressed file compressed based on the target second compression method are determined; Based on the first index value, the second compressed file is stored in the first storage bucket in the object storage service.

5. The data storage method according to claim 4, characterized in that: The extension of the first compressed file is identification information of the target first compression mode; The decompressing the first compressed file to obtain the target data includes: Determining the target first compression mode corresponding to the first compressed file based on the extension of the first compressed file; Based on the decompression method corresponding to the target first compression method, the first compressed file is decompressed to obtain the target data.

6. The data storage method according to claim 4, characterized in that: The method of compressing the target data in parallel based on multiple second compression methods, and determining a target second compression method corresponding to the target data and a second compressed file compressed based on the target second compression method based on an actual compression rate of each second compression method, includes: Initiate a plurality of compression processes for concurrent execution, and perform parallel compression processing on the target data through different compression processes based on different second compression modes; In response to the completion of all compression processes, the file size of the compressed file compressed by each compression process is obtained, and the compressed file with the smallest file size is used as the second compressed file, and the second compression method used by the second compressed file is used as the target second compression method corresponding to the target data.

7. The data storage method according to claim 4, characterized in that: The storing the second compressed file in the first storage bucket in the object storage service based on the first index value includes: Determine a second index value corresponding to the second compressed file, where the second index value includes identification information of the second storage bucket allocated at that time; Based on the second index value, the second compressed file is stored in the second storage bucket in the object storage service, and the target mapping relationship in the database is updated based on the second index value.

8. The data storage method according to claim 4, characterized in that: After storing the second compressed file in the object storage service, the method further includes: In response to the current storage duration of the target data exceeding a second preset duration, or the current access frequency of the target data being lower than a preset access frequency, converting the storage type of the second compressed file from a standard type to an archive type for storage; Among them, the second preset time length is greater than the first preset time length.

9. A data storage device, characterized in that: include: A target data acquisition module, used to acquire target data to be stored; A first compression processing module, configured to perform parallel compression processing on the target data based on a plurality of first compression methods, and determine a target first compression method corresponding to the target data and a first compressed file compressed based on the target first compression method based on an actual compression speed of each first compression method; A first index value determination module, configured to determine a first index value corresponding to the first compressed file, wherein the first index value includes identification information of a first storage bucket allocated at that time; The first compressed file storage module is used to store the first compressed file in the first storage bucket in the object storage service based on the first index value, establish a target mapping relationship between the first index value and the identification information of the target data, and store the target mapping relationship in a database.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data storage method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data storage method according to any one of claims 1 to 8.

Images