A distributed storage system, data storage method and product
By introducing a business management module in a distributed storage system, the data storage mode is dynamically determined based on the classification level of the business system, the balance between multi-copy and erasure modes is solved, and the overall system is improved Storage efficiency and adaptability.
Patent Information
- Application Number
- CN202410992952.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-23
AI Technical Summary
In distributed storage systems, data protection mechanisms such as multi-copy and erasure have their own advantages and disadvantages, and cannot find a balance between storage efficiency and read and write efficiency, and it is difficult to meet high concurrency or high I/O requirements.
By introducing a business management module into the distributed storage system, the data storage mode is dynamically determined based on the classification level of the business system, and a multi-copy storage mode or erasure mode is adopted to achieve secure storage of data.
It improves the storage efficiency of distributed storage systems, takes into account data security and read and write efficiency, and adapts to the needs of different business systems.
Smart Images

Figure CN118567573B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present application relate to the technical field of data storage, and in particular, to a distributed storage system, a data storage method, and a product. Background Art
[0002] In the process of digital development, data storage has also become a top priority. The current mainstream data storage includes two types: centralized storage and distributed storage. Distributed storage can deploy distributed storage software based on general server devices to meet the storage needs of large data centers, and its distributed technical architecture also provides convenience for rapid capacity expansion in large-scale scenarios. With the continuous optimization of underlying technologies, its storage processing capabilities have also been greatly improved, and its application scope is getting wider and wider. At the same time, in order to ensure the security of data, distributed storage provides a variety of data protection mechanisms, such as multiple copies and error correction and deletion.
[0003] However, different data protection mechanisms in distributed storage systems have their own storage characteristics and defects. For example, while the multi-copy data protection mechanism improves data security, the storage space utilization rate is low; the erasure data protection mechanism has high storage space utilization, but low read and write efficiency, especially when writing data, the efficiency will be much lower than the multi-copy mechanism, and it cannot meet the needs of high concurrency or high I / O (Input / Output) of the business.
[0004] Therefore, how to reasonably select a data protection mechanism during data storage to achieve storage efficiency of a distributed storage system is an urgent problem to be solved. Summary of the invention
[0005] The embodiments of the present application provide a distributed storage system, a data storage method, and a product, aiming to improve the storage efficiency of the distributed storage system.
[0006] In a first aspect, an embodiment of the present application provides a distributed storage system, wherein the distributed storage system provides storage services for multiple business systems, and the distributed storage system includes a business management module, wherein:
[0007] The business management module is used to respond to a storage request for business data to be stored sent by the current business system, determine the classification level corresponding to the current business system; determine the data storage mode corresponding to the business data to be stored according to the classification level corresponding to the current business system; and store the business data to be stored in the distributed storage system according to the data storage mode corresponding to the business data to be stored;
[0008] The business management module stores classification levels corresponding to different business systems and a mapping relationship between each classification level and a data storage mode, and the data storage mode is used to securely store the business data to be stored.
[0009] Optionally, the service management module includes:
[0010] A business interface, used to obtain the classification level of any business system, the classification level includes key data, important data and general data;
[0011] A classification level database, used to store classification levels corresponding to each of the plurality of business systems;
[0012] An execution unit is used to search the classification level corresponding to the current business system in the classification level database, determine the data storage mode corresponding to the business data to be stored according to the classification level corresponding to the current business system and the mapping relationship between each classification level and the data storage mode, and store the business data to be stored in the distributed storage system according to the data storage mode corresponding to the business data to be stored.
[0013] Optionally, the execution unit is used to:
[0014] When the classification level corresponding to the current business system is key data, determining that the data storage mode corresponding to the business data to be stored is a multiple copy storage mode;
[0015] When the classification level corresponding to the current business system is important data or general data, it is determined that the data storage mode corresponding to the business data to be stored is an erasure mode.
[0016] Optionally, the business management module is further used to receive classification level information sent by any business system when connecting to the distributed storage system, and obtain and store the classification level corresponding to the business system in the classification level information.
[0017] Optionally, the distributed storage system further includes a monitoring and analysis module, and the monitoring and analysis module is used to:
[0018] Monitor the analytical dimension information of any business system, including the number of read and write operations per second, throughput, and storage capacity;
[0019] Analyze and obtain the classification level of the business system according to the analysis dimension information;
[0020] The classification level of the business system is sent to the business management module.
[0021] Optionally, the monitoring and analysis module includes:
[0022] A monitoring unit, used to monitor multiple historical business data of any business system and obtain analysis dimension information of each of the multiple historical business data;
[0023] A storage unit, used to store analysis dimension information of each of the plurality of historical business data of the business system in a first target storage space in the distributed storage system;
[0024] The analysis unit is used to determine the classification level of the business system according to the analysis dimension information of each of the multiple historical business data of the business system, and send the classification level to the business management module.
[0025] Optionally, the analysis unit is used to:
[0026] For any business system, determining a first average value according to the number of read and write operations per second of each of a plurality of historical business data of the business system;
[0027] Determine a second average value according to the throughput of each of the plurality of historical business data of the business system;
[0028] Determine the storage capacity according to the storage size of each of the multiple historical business data of the business system;
[0029] Calculate a grade score according to the first average value, the second average value and the storage capacity;
[0030] The classification level of the business system is determined according to the level score of the business system and the level thresholds corresponding to different classification levels.
[0031] Optionally, the formula for the analysis unit to determine the grade score of any business system is:
[0032] Grade score = (first average value*50%+second average value*30%) / (storage capacity*10%).
[0033] Optionally, the monitoring unit is further used to monitor the analysis dimension information of each business storage data of any business system within a preset monitoring period;
[0034] The storage unit is used to store the analysis dimension information of each business storage data of the business system within a preset monitoring period in the second target storage space of the distributed storage system;
[0035] An analysis unit, configured to obtain an updated classification level of the business system according to analysis dimension information of each business storage data of the business system within a preset monitoring period, and send the updated classification level to the business management module;
[0036] The business management module is used to replace the current classification level of the business system according to the updated classification level of the business system.
[0037] Optionally, the analysis unit is used to send an analysis completion instruction to the storage unit after obtaining the updated classification level of the business system;
[0038] The storage unit is used to delete the second analysis dimension information of the business system within a preset monitoring period stored in the second target storage space in response to the analysis completion instruction.
[0039] Optionally, the distributed storage system further includes a conversion module, wherein:
[0040] The conversion module is used to redetermine the updated data storage mode corresponding to each business data stored in the business system in the distributed storage system according to the transformed classification level when a change in the classification level of any business system in the business management module is detected, and to change the storage mode of each business data stored in the business system in the distributed storage system according to the updated data storage mode.
[0041] In a second aspect, an embodiment of the present application provides a data storage method of a distributed storage system, the method being applied to the distributed storage system, the method comprising:
[0042] In response to a storage request for business data to be stored sent by a current business system, determining a classification level corresponding to the current business system;
[0043] Determine the data storage mode corresponding to the business data to be stored according to the classification level corresponding to the current business system;
[0044] Storing the business data to be stored in the distributed storage system according to the data storage mode corresponding to the business data to be stored;
[0045] The distributed storage system stores classification levels corresponding to different business systems and a mapping relationship between each classification level and a data storage mode, and the data storage mode is used to securely store the business data to be stored.
[0046] In a third aspect, an embodiment of the present application provides a computer device, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the data storage method of the distributed storage system described in the second aspect of the embodiment is executed.
[0047] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the data storage method of the distributed storage system described in the second aspect of the embodiment is executed.
[0048] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the data storage method of the distributed storage system described in the second aspect of the embodiment.
[0049] Beneficial effects:
[0050] The distributed storage system provides storage services for multiple business systems. The distributed storage system includes a business management module. The business management module first responds to a storage request for business data to be stored sent by a current business system, and determines a classification level corresponding to the current business system; then, based on the classification level corresponding to the current business system, determines a data storage mode corresponding to the business data to be stored; finally, according to the data storage mode corresponding to the business data to be stored, the business data to be stored is stored in the distributed storage system.
[0051] The business management module stores the classification levels corresponding to different business systems and the mapping relationship between each classification level and the data storage mode. By determining the classification level of the current business system and determining the data storage mode of the business data to be stored sent by the current business system according to the classification level, the data protection mechanism can be reasonably selected during the data storage process, rather than all business data using the same data protection mechanism, which can effectively improve the efficient storage of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.
[0053] Figure 1 A schematic diagram of a distributed storage system provided by an embodiment of the present application is shown;
[0054] Figure 2 A schematic diagram of a distributed storage system provided by an embodiment of the present application is shown;
[0055] Figure 3 A schematic diagram of a distributed storage system provided by an embodiment of the present application is shown;
[0056] Figure 4 A schematic diagram of a monitoring and analysis module provided in an embodiment of the present application is shown;
[0057] Figure 5 A schematic diagram of a distributed storage system provided by an embodiment of the present application is shown;
[0058] Figure 6 A flowchart showing a data storage method of a distributed storage system provided by an embodiment of the present application is shown;
[0059] Figure 7 A schematic diagram of a computer device provided in an embodiment of the present application is shown;
[0060] Figure 8 A schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application is shown;
[0061] Fig. 9 A schematic diagram of a computer program product provided by an embodiment of the present application is shown. DETAILED DESCRIPTION
[0062] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0063] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in each embodiment of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not constitute any limitation, and the various embodiments can be combined with each other and referenced on the premise of no contradiction.
[0064] I / O: Input / Output, input / output;
[0065] API: Application Programming Interface, application programming interface;
[0066] ETCD database: Extended Distributed Key Value, is an open source distributed key-value storage database;
[0067] IOPS: Input / Output Operations Per Second, the number of read and write operations per second.
[0068] In the process of digital development, the development of data center informatization is increasingly becoming a key force in promoting digital transformation. With the rapid development of technologies such as cloud computing, big data and artificial intelligence, data centers not only carry massive amounts of data, but are also the core support for intelligent operations and innovative development of enterprises. Data centers are also undergoing a transformation from traditional physical architecture to virtualization and cloud architecture, achieving dynamic allocation and efficient utilization of resources, and greatly improving data processing capabilities and service response speed. At the same time, the development of data center informatization has also brought about a comprehensive improvement in security, adopting a multi-level security protection strategy, including physical security, network security and data security, to ensure the integrity and confidentiality of data during transmission, storage and processing.
[0069] Currently, mainstream data storage includes two types: centralized storage and distributed storage. Among them, centralized storage was applied earlier than distributed storage. Centralized storage has advantages over distributed storage in processing high I / O and high-concurrency businesses, but its procurement cost and management cost are also much higher than distributed storage.
[0070] Distributed storage can deploy distributed storage software based on general server devices to meet the storage needs of large data centers. Its distributed technical architecture also provides convenience for rapid capacity expansion in large-scale scenarios. With the continuous optimization of underlying technologies, its storage processing capabilities have been greatly improved, and its application scope is getting wider and wider. At the same time, distributed storage provides a variety of data protection mechanisms to ensure data security. The two representative solutions are multiple copies and erasure.
[0071] Multiple copies are a data protection mechanism for copying data, that is, a copy of a piece of data is fully copied and then stored. The higher the security requirement, the more copies are made to ensure data security in extreme cases; however, the multiple copy mechanism sacrifices the available storage capacity of the storage device in the process of ensuring data security; for example, a distributed storage system provides 100TB of available storage space. If the multiple copy data protection mechanism is not adopted, the distributed storage system can store 100TB of data information, but if a two-copy data protection mechanism is adopted, since a copy of the data needs to be stored, only 50TB can be used to store data, and the other 50TB is used to store the copied data; when the multiple copy data protection mechanism is adopted, storage space will be lost, and the more copies there are, the greater the storage space lost.
[0072] Erasure is a data protection mechanism that divides the original data into blocks and stores them after encoding through a coding algorithm. By generating coding blocks containing redundant information based on the original data blocks, the coding blocks are stored together with the original data blocks to facilitate recovery when data is lost. Compared with multi-copy redundancy, erasure codes have higher disk utilization. However, due to the data block storage mechanism and coding fault-tolerant implementation scheme of erasure, the data reading and writing efficiency will be greatly reduced, and it is not suitable for scenarios with high I / O requirements.
[0073] In summary, different data protection mechanisms in distributed storage systems have their own storage characteristics and defects. For example, the multi-copy data protection mechanism has low storage space utilization while improving data security; the erasure data protection mechanism has high storage space utilization, but low read and write efficiency, especially when writing data, the efficiency will be much lower than the multi-copy mechanism, and it cannot meet the needs of high concurrency or high I / O of the business; therefore, how to reasonably select the data protection mechanism in the process of data storage to achieve efficient storage of the distributed storage system is an urgent problem to be solved.
[0074] In order to achieve efficient storage in a distributed storage system, an embodiment of the present application provides a distributed storage system.
[0075] Reference Figure 1 , showing a schematic diagram of a distributed storage system provided in an embodiment of the present application, the distributed storage system provides storage services for multiple business systems, such as business system 1...business system n, and the distributed storage system includes multiple data storage modes, the data storage mode is used to securely store the business data to be stored, by way of example, the data storage mode may include a multi-copy storage mode and an erasure mode, and in the actual implementation process, other data storage modes may also be expanded, which is not limited in this embodiment.
[0076] The distributed storage system includes a business management module, which is a software thread set in the distributed storage system and can run in the storage management device of the distributed storage system. The actual deployment location of the business management module in the distributed storage system can be customized according to the needs of actual applications.
[0077] The business management module stores the classification levels corresponding to different business systems and the mapping relationship between each classification level and the data storage mode. The business management module is used to respond to a storage request for business data to be stored sent by the current business system, determine the classification level corresponding to the current business system; then determine the data storage mode corresponding to the business data to be stored according to the classification level corresponding to the current business system; finally, store the business data to be stored in the distributed storage system according to the data storage mode corresponding to the business data to be stored.
[0078] Reference Figure 2 , showing a schematic diagram of a distributed storage system provided in an embodiment of the present application, in a feasible implementation manner, the business management module includes: a business interface (restAPI interface), a classification level database and an execution unit.
[0079] Specifically, the business interface is used to obtain the classification level of any business system. For example, the classification level in this embodiment includes key data, important data and general data. In the actual implementation process, the classification level can be customized according to the needs of the actual application.
[0080] The classification level database is used to store the classification levels corresponding to the multiple business systems. By way of example, the classification level database may adopt the ETCD database. In the actual implementation process, other types of databases may also be adopted, which is not limited in this embodiment. In the classification level database, the mapping relationship between the identification of each business system and the classification level may be stored.
[0081] The execution unit is used to search the classification level corresponding to the current business system in the classification level database, determine the data storage mode corresponding to the business data to be stored according to the classification level corresponding to the current business system and the mapping relationship between each classification level and the data storage mode, and store the business data to be stored in the distributed storage system according to the data storage mode corresponding to the business data to be stored.
[0082] Specifically, the storage request for the business data to be stored issued by the current business system carries the identifier of the current business system, and the execution unit searches the classification level corresponding to the current business system in the classification level database according to the identifier of the current business system.
[0083] Then, the execution unit determines the data storage mode corresponding to the business data to be stored issued by the current business system according to the classification level corresponding to the current business system. Specifically, when the classification level corresponding to the current business system is key data, the data storage mode corresponding to the business data to be stored is determined to be a multi-copy storage mode; when the classification level corresponding to the current business system is important data or general data, the data storage mode corresponding to the business data to be stored is determined to be an erasure mode.
[0084] In a feasible implementation, when the business management module stores the classification levels of various business systems, any business system responds to the user's classification level setting operation and sets a fixed classification level for the business system; when the business system is connected to a distributed storage system, the classification level information sent by the business system to the distributed storage system will carry the classification level of the business system; the business management module is also used to receive the classification level information sent by any business system when connecting to the distributed storage system, and obtain and store the classification level corresponding to the business system in the classification level information.
[0085] For example, when the business system connects to the distributed storage system, it calls the business interface in the business management module and sends classification level information, namely, stor-level-param information, which carries the classification level of the business system; then the classification level of the business system is stored in the classification level database.
[0086] In another feasible implementation, in addition to the classification level of any business system set by the user, the classification level of any business system can also be determined according to historical business data of the business system.
[0087] Reference Figure 3 , showing a schematic diagram of a distributed storage system provided in an embodiment of the present application, the distributed storage system also includes a monitoring and analysis module, the monitoring and analysis module is used to: monitor the analysis dimension information of any business system, the analysis dimension information including the number of read and write operations per second, throughput and storage capacity; according to the analysis dimension information, analyze and obtain the classification level of the business system; send the classification level of the business system to the business management module.
[0088] Specifically, the monitoring and analysis module includes a monitoring unit, a storage unit and an analysis unit.
[0089] Reference Figure 4 , shows a schematic diagram of a monitoring and analysis module provided in an embodiment of the present application. When the monitoring and analysis module determines the classification level of any business system, the execution process of the monitoring unit, the storage unit and the analysis unit includes:
[0090] First, the monitoring unit monitors a plurality of historical business data of any business system, and obtains analysis dimension information of each of the plurality of historical business data.
[0091] For example, for business system A, multiple historical business data from business system A stored in the distributed storage system can be monitored in real time and obtained. The number of historical business data can be customized, and it can be part of the historical business data of business system A or all of the historical business data of business system A. For any historical business data, the number of read and write operations per second, throughput and storage capacity corresponding to the historical business data are obtained.
[0092] Then, the storage unit stores the analysis dimension information of each of the multiple historical business data of the business system in the first target storage space in the distributed storage system; the number of read and write operations per second (IOPS, unit k / s), throughput (the amount of data transmitted per unit time, unit MB / s) and storage capacity (occupied storage space, unit GB) corresponding to the historical business data obtained through real-time monitoring can be stored in a physical storage space in the distributed storage system. For example, a solid-state hard drive can be selected in the distributed storage system to specifically store the analysis dimension information of any historical business data of each business system.
[0093] Next, the analysis unit determines the classification level of the business system according to the analysis dimension information of each of the plurality of historical business data of the business system, and sends the classification level to the business management module.
[0094] Specifically, when determining the classification level of the business system, the analysis unit includes the following steps:
[0095] A1: For any business system, determine a first average value according to the number of read and write operations per second of each of a plurality of historical business data of the business system.
[0096] For example, assuming that the number of historical business data is x, the average number of read and write operations per second for each of the x historical business data is calculated to obtain the average number of read and write operations per second as the first average value.
[0097] A2: Determine a second average value according to the throughput of each of the plurality of historical business data of the business system.
[0098] Then, the throughput of each of the x historical service data is averaged to obtain the average throughput as the second average value.
[0099] A3: Determine the storage capacity based on the storage size of multiple historical business data of the business system.
[0100] The storage size corresponding to each of the x historical business data is summed up to obtain the storage capacity.
[0101] A4: Calculate a grade score according to the first average value, the second average value and the storage capacity.
[0102] For example, the formula for determining the grade score of any business system is:
[0103] Grade score = (first average value*50%+second average value*30%) / (storage capacity*10%).
[0104] A5: Determine the classification level of the business system according to the level score of the business system and the level thresholds corresponding to different classification levels.
[0105] For example, different classification levels can set corresponding level thresholds, which can be customized according to the needs of actual applications; for example: if the level score is ≥1, the classification level of the business system is the key data level; if the level score is 0.5≤<1, the classification level of the business system is important data; if the level score is <0.5, the classification level of the business system is general data.
[0106] During the actual implementation process, the monitoring and analysis module can monitor the business systems whose classification levels are not included in the classification level database of the business management module, monitor and analyze the business systems that do not include the classification levels, and after obtaining the classification levels of the business systems that do not include the classification levels, send them to the business interface of the business management module, and then store the classification levels of the business systems in the classification level database.
[0107] In a feasible implementation, the classification level of a business system that adopts user-defined classification levels usually does not change until it changes in response to the user's level modification operation. However, the classification level of a business system whose classification level is determined by a monitoring and analysis module may change due to changes in the business of the business system. Therefore, the classification level of the business system can also be dynamically updated to correct the classification level of the business system and obtain a more accurate classification level.
[0108] Specifically, the monitoring and analysis module may also update the classification levels obtained by the monitoring and analysis at regular intervals; the monitoring unit is used to monitor the analysis dimension information of each business storage data of any business system within a preset monitoring period.
[0109] The preset monitoring period can be customized according to the needs of the actual application. For example, the preset monitoring period can be set to one month, that is, the business storage data of any business system within one month is monitored. The business storage data refers to the data written into the distributed storage system by the business system within the preset monitoring period; the analysis dimension information can still be the number of read and write operations per second, throughput and storage capacity.
[0110] The storage unit is used to store the analysis dimension information of each business storage data of the business system within a preset monitoring period in the second target storage space in the distributed storage system; by way of example, another solid-state hard drive is selected in the distributed storage system to specifically store the analysis dimension information of any business storage data of each business system within a preset monitoring period.
[0111] Finally, the analysis unit obtains the updated classification level of the business system based on the analysis dimension information of each business storage data of the business system within the preset monitoring period, and sends the updated classification level to the business management module. The analysis unit determines the level score of the business system in the preset monitoring period based on the analysis dimension information of each business storage data within the preset monitoring period, and then determines the updated classification level of the business system based on the level score in the preset monitoring period and the level thresholds corresponding to different classification levels; the calculation process of determining the level score and classification level of the business system in the preset monitoring period based on the analysis dimension information of each business storage data is shown in steps A1-A5.
[0112] The business management module is used to replace the current classification level of the business system according to the updated classification level of the business system, and then update the classification level of the business system stored in the classification level database of the business management module; in the actual implementation process, it is also possible to compare whether the updated classification level is consistent with the current classification level for any business system. If not, it will be replaced, thereby reducing data processing costs.
[0113] The analysis unit is used to send an analysis completion instruction to the storage unit after obtaining the updated classification level of the business system; the storage unit is used to delete the second analysis dimension information of the business system within a preset monitoring period stored in the second target storage space in response to the analysis completion instruction, thereby releasing the second target storage space, and storing the analysis dimension information detected within the next preset monitoring period in the second target storage space.
[0114] Reference Figure 5 , showing a schematic diagram of a distributed storage system provided by an embodiment of the present application. In a feasible implementation, the distributed storage system also includes a conversion module, which is used to, when detecting a change in the classification level of any business system in the business management module, re-determine the updated data storage mode corresponding to each business data stored in the business system in the distributed storage system according to the transformed classification level, and change the storage mode of each business data stored in the business system in the distributed storage system according to the updated data storage mode.
[0115] When the classification level of any business system changes, the data storage mode for the business data of the business system will also change, and the data storage mode corresponding to the data of the business system that has been stored in the distributed storage system before the classification level changes can also be modified; for example, the initial classification level of business system A is critical data, and business data a1 is stored at this time; but with the changes in the business, the access volume and corresponding demand of business system A have decreased. Assuming that the updated classification level is downgraded to important data, the conversion module can convert the business data a1 originally stored in the multi-copy storage mode of business system A when the classification level is critical data into the corresponding correction and erasure mode for storage when the classification level is downgraded to important data. Specifically, based on the master data in the multi-copy storage mode, the business data a1 is converted to the correction and erasure mode for storage, and then all copies of the business data a1 are deleted from the storage pool of the distributed storage system to release storage space.
[0116] When the classification level of the business system is upgraded, assuming that the classification level is upgraded from important data to critical data, for the business data a2 stored in the erasure mode, the data sharding and erasure code of the business data a2 in the erasure mode are used to recover the business data a2, and a multi-copy storage mode is adopted for the business data a2, and multiple copies of the business data a2 are generated and stored in the storage pool of the distributed storage system; thereby, it can be ensured that the business data of the business system with the classification level as critical data always adopts the most efficient and reliable multi-copy storage mode.
[0117] In the distributed storage system provided in this embodiment, by adding a business management module, a monitoring and analysis module, and a conversion module, the level of the business served by the distributed storage system can be perceived, and different data storage modes can be provided for data of business systems of different classification levels, thereby achieving storage requirements that take into account both security and efficiency, and greatly improving the capabilities of the distributed storage system.
[0118] Reference Figure 6 , shows a flowchart of a data storage method of a distributed storage system provided in an embodiment of the present application, the method is applied to a distributed storage system, the method comprises the following steps:
[0119] S101: In response to a storage request for business data to be stored sent by a current business system, determining a classification level corresponding to the current business system.
[0120] The storage request for the business data to be stored issued by the current business system carries the identification of the current business system. According to the identification of the current business system, the classification level corresponding to the current business system can be searched in the classification level database. The classification levels include key data, important data and general data.
[0121] The classification level of the business system can be a user-defined level obtained from the business system, or it can be obtained by analyzing multiple historical business data of the business system in a distributed storage system; and the classification level of the business system obtained through analysis can also be dynamically updated at preset monitoring cycles, so as to timely update the classification level of the business system and reasonably store the business data of the business system.
[0122] S102: Determine a data storage mode corresponding to the business data to be stored according to the classification level corresponding to the current business system.
[0123] For example, when the classification level corresponding to the current business system is critical data, the data storage mode corresponding to the business data to be stored is determined to be a multi-copy storage mode; when the classification level corresponding to the current business system is important data or general data, the data storage mode corresponding to the business data to be stored is determined to be an erasure mode.
[0124] S103: storing the business data to be stored in the distributed storage system according to the data storage mode corresponding to the business data to be stored.
[0125] The distributed storage system stores classification levels corresponding to different business systems and mapping relationships between classification levels and data storage modes. For example, the distributed storage system is provided with the business management module, monitoring and analysis module and conversion module described in the embodiment.
[0126] By determining the classification level of the current business system and determining the data storage mode of the business data to be stored sent by the current business system according to the classification level, it is possible to reasonably select a data protection mechanism during the data storage process, rather than using the same data protection mechanism for all business data. This can effectively improve the efficient storage of the distributed storage system.
[0127] Reference Figure 7 , shows a schematic diagram of a computer device provided in an embodiment of the present application, the computer device includes: at least one processor 701, and a memory 702, the memory 702 stores a computer program that can be run on the processor 701, wherein the processor 701 executes the data storage method of the distributed storage system described in the embodiment when executing the computer program.
[0128] Reference Figure 8 , shows a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium 800 stores a computer program 801, wherein the computer program 701, when executed by a processor, executes the data storage method of the distributed storage system described in the embodiment.
[0129] Reference Fig. 9 , showing a schematic diagram of a computer program product provided in an embodiment of the present application, wherein the computer program product 900 includes a computer program / instruction 901, which, when executed by a processor, implements the data storage method of the distributed storage system described in the embodiment.
[0130] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0131] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0132] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0133] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0135] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0136] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0137] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A distributed storage system, characterized in that: The distributed storage system provides storage services for multiple business systems. The distributed storage system includes a business management module, wherein: The business management module is used to respond to a storage request for business data to be stored sent by the current business system, determine the classification level corresponding to the current business system; determine the data storage mode corresponding to the business data to be stored according to the classification level corresponding to the current business system; and store the business data to be stored in the distributed storage system according to the data storage mode corresponding to the business data to be stored; Among them, the business management module stores the classification levels corresponding to different business systems and the mapping relationship between each classification level and the data storage mode, and the data storage mode is used to securely store the business data to be stored; the classification level of any business system is determined based on the historical business data of the business system, and the historical business data includes the number of read and write operations per second, throughput and storage capacity.
2. The distributed storage system according to claim 1, characterized in that: The business management module includes: A business interface, used to obtain the classification level of any business system, the classification level includes key data, important data and general data; A classification level database, used to store classification levels corresponding to each of the plurality of business systems; An execution unit is used to search the classification level corresponding to the current business system in the classification level database, determine the data storage mode corresponding to the business data to be stored according to the classification level corresponding to the current business system and the mapping relationship between each classification level and the data storage mode, and store the business data to be stored in the distributed storage system according to the data storage mode corresponding to the business data to be stored.
3. The distributed storage system according to claim 2, characterized in that: The execution unit is used for: When the classification level corresponding to the current business system is key data, determining that the data storage mode corresponding to the business data to be stored is a multiple copy storage mode; When the classification level corresponding to the current business system is important data or general data, it is determined that the data storage mode corresponding to the business data to be stored is an erasure mode.
4. The distributed storage system according to claim 1, characterized in that: The business management module is also used to receive classification level information sent by any business system when connecting to the distributed storage system, and obtain and store the classification level corresponding to the business system in the classification level information.
5. The distributed storage system according to claim 1, characterized in that: The distributed storage system further includes a monitoring and analysis module, which is used to: Monitor the analytical dimension information of any business system, including the number of read and write operations per second, throughput, and storage capacity; Analyze and obtain the classification level of the business system according to the analysis dimension information; The classification level of the business system is sent to the business management module.
6. The distributed storage system according to claim 5, characterized in that: The monitoring and analysis module includes: A monitoring unit, used to monitor multiple historical business data of any business system and obtain analysis dimension information of each of the multiple historical business data; A storage unit, used to store analysis dimension information of each of the plurality of historical business data of the business system in a first target storage space in the distributed storage system; The analysis unit is used to determine the classification level of the business system according to the analysis dimension information of each of the multiple historical business data of the business system, and send the classification level to the business management module.
7. The distributed storage system according to claim 6, characterized in that: The analysis unit is used for: For any business system, determining a first average value according to the number of read and write operations per second of each of a plurality of historical business data of the business system; Determine a second average value according to the throughput of each of the plurality of historical business data of the business system; Determine the storage capacity according to the storage size of each of the multiple historical business data of the business system; Calculate a grade score according to the first average value, the second average value and the storage capacity; The classification level of the business system is determined according to the level score of the business system and the level thresholds corresponding to different classification levels.
8. The distributed storage system according to claim 7, characterized in that: The formula for determining the grade score of any business system by the analysis unit is: Grade score = (first average value*50%+second average value*30%) / (storage capacity*10%).
9. The distributed storage system according to claim 6, characterized in that: The monitoring unit is further used to monitor the analysis dimension information of each business storage data of any business system within a preset monitoring period; The storage unit is used to store the analysis dimension information of each business storage data of the business system within a preset monitoring period in the second target storage space of the distributed storage system; An analysis unit, configured to obtain an updated classification level of the business system according to analysis dimension information of each business storage data of the business system within a preset monitoring period, and send the updated classification level to the business management module; The business management module is used to replace the current classification level of the business system according to the updated classification level of the business system.
10. The distributed storage system according to claim 9, characterized in that: The analysis unit is used to send an analysis completion instruction to the storage unit after obtaining the updated classification level of the business system; The storage unit is used to delete the second analysis dimension information of the business system within a preset monitoring period stored in the second target storage space in response to the analysis completion instruction.
11. The distributed storage system according to claim 7, characterized in that: The distributed storage system further includes a conversion module, wherein: The conversion module is used to redetermine the updated data storage mode corresponding to each business data stored in the business system in the distributed storage system according to the transformed classification level when a change in the classification level of any business system in the business management module is detected, and to change the storage mode of each business data stored in the business system in the distributed storage system according to the updated data storage mode.
12. A data storage method for a distributed storage system, characterized in that: The method is applied to the distributed storage system according to any one of claims 1 to 11, wherein the distributed storage system provides storage services for multiple business systems, and the method includes: In response to a storage request for business data to be stored sent by a current business system, determining a classification level corresponding to the current business system; Determine the data storage mode corresponding to the business data to be stored according to the classification level corresponding to the current business system; Storing the business data to be stored in the distributed storage system according to the data storage mode corresponding to the business data to be stored; Among them, the distributed storage system stores the classification levels corresponding to different business systems and the mapping relationship between each classification level and the data storage mode, and the data storage mode is used to securely store the business data to be stored; the classification level of any business system is determined based on the historical business data of the business system, and the historical business data includes the number of read and write operations per second, throughput and storage capacity.
13. A computer device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the data storage method of the distributed storage system according to claim 12 is executed.
14. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein the computer program, when executed by a processor, executes the data storage method of the distributed storage system according to claim 12.
15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the data storage method of the distributed storage system described in claim 12 is implemented.
