Data file storage system based on combination of distributed storage and blue light storage

Abstract

The invention discloses a data file storage system based on combination of distributed storage and blue-ray storage, relates to the technical field of data file storage, and not only ensures quick access of high-frequency data, but also realizes long-term reliable storage of low-frequency and important data through combination of distributed storage nodes and blue-ray storage equipment. The distributed storage nodes provide efficient writing and reading performance and are suitable for the data requirement of frequent access, and blue light storage has low storage cost and excellent durability and is particularly suitable for storing low-frequency and long-term reserved data; besides, the data management software layer intelligently classifies the importance and access frequency of data files, so that dynamic allocation and optimal management of resources are realized, and reasonable storage of different data is ensured; according to the whole system, the data access efficiency is improved, the dual requirements of real-time performance and safety are met, and the storage cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data file storage, and in particular to a data file storage system based on the combination of distributed storage and blue-ray storage. Background Art

[0002] In the current information age, with the continuous acceleration of the digitalization process, the generation and storage demand of massive data continues to rise, especially in various data-intensive industries such as video, audio, archives, backup, etc., which put forward higher requirements for the long-term storage and efficient management of large-scale data. Due to the limitations of storage media, cost and management methods, traditional storage systems are difficult to achieve an ideal balance between high-frequency access needs and long-term storage needs. Therefore, in order to adapt to the storage needs of different data characteristics, there is an urgent need for a new storage system that can take into account data access efficiency, storage cost and persistence.

[0003] In the prior art, distributed storage is widely used in cloud storage, big data analysis and other scenarios with high real-time requirements due to its efficient reading performance and scalability. However, its cost for long-term data preservation is high, and the high-frequency access requirements also occupy a large amount of system resources. Blu-ray storage technology, due to its low storage cost, stable durability and tamper-proof characteristics, is suitable for storing low-frequency access data that needs to be retained for a long time, such as archives, backups, historical data, etc. However, a single distributed storage or Blu-ray storage system cannot fully meet the data needs that require both high-frequency access and economical long-term preservation in actual applications. Therefore, there is an urgent need for a system architecture that combines distributed storage and Blu-ray storage, which can not only meet the needs of different data characteristics, but also optimize resource utilization, and realize intelligent hierarchical storage and efficient management of data.

[0004] Based on this, the present invention proposes a "data file storage system based on the combination of distributed storage and Blu-ray storage", which organically combines distributed storage nodes with Blu-ray storage devices, and uses the data management software layer to classify data according to the importance and access frequency of data files, thereby determining the data storage method. While ensuring fast access to high-frequency data, the system effectively reduces the storage cost of low-frequency data, meeting the multiple requirements of storage performance, cost control and data security in different data scenarios. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a data file storage system based on a combination of distributed storage and Blu-ray storage, which solves the problems in the background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a data file storage system based on a combination of distributed storage and Blu-ray storage, comprising: Data center platform, used to write, store, manage and read data files; The data center platform includes data writing module, data management software layer, Blu-ray storage device, and distributed storage nodes; A data writing module is used to first write the data file into the distributed storage node; The data management software layer is used to classify data files according to their importance index and access frequency index, and classify them into important high-frequency data files, unimportant high-frequency data files, important non-high-frequency data files, and unimportant non-high-frequency data files, and store the data files in a corresponding manner according to the classification of the data files; Distributed storage nodes; used to store data files classified as important high-frequency data files and non-important high-frequency data files by the data management software layer; The Blu-ray storage device is used to store data files classified by the data management software layer as important high-frequency data files, important non-high-frequency data files, and non-important non-high-frequency data files.

[0007] As a further solution of the present invention: in the data management software layer, the corresponding storage method includes storage in distributed storage nodes and Blu-ray storage devices.

[0008] As a further solution of the present invention: the specific method of classifying data files according to the importance index and access frequency index of data files is: The importance index of the data file is determined by the following key factors. The key factors in the importance index include the value attribute of the file content, the creation time of the file, and the permission level of the data file, which are recorded as ; The importance index of the data file is determined and evaluated by the following formula based on the key factors of the file content value attribute, the file creation time, and the permission level of the data file. ;

[0009] In the formula, is the weight factor; Importance index of data files With the default value For comparison: like , then the data file is marked as an important data file; like , then the data file is marked as a non-important data file; The access frequency index of the data file is determined by the following key factors. The key factors in the access frequency index include historical access frequency, recent access frequency, and access volatility, which are recorded as , , ; Based on the key factors of historical access frequency, recent access frequency, and access volatility, the access frequency index of the data file is determined and evaluated through the following formula ;

[0010] In the formula, is the weight factor; The access frequency index of the data file With the default value For comparison: like , then the data file is marked as a high-frequency data file; like , then the data file is marked as a non-high-frequency data file.

[0011] As a further solution of the present invention: the specific method of classifying data files according to the importance index and access frequency index of the data files also includes: Importance index for combining data files Frequency of visit index The data files are divided into four categories: important high-frequency data files, unimportant high-frequency data files, important non-high-frequency data files and unimportant non-high-frequency data files.

[0012] As a further solution of the present invention: the specific parameter calculation method of each key factor in the importance index is: File content value attribute : Assign a range to the content value attribute of the data file , where 1 represents the information with the highest value and 0 represents ordinary data. The value is assigned based on keyword calculation: a preset key business vocabulary is used, and different weights are assigned to each keyword in the key business vocabulary. The data file content is scanned and the number and weight of these keywords are identified. The value attribute of the file content is determined by the following formula :

[0013] In the formula, Indicates The weight coefficient of each keyword, Indicates the first The number of times a keyword appears; It is a parameter used for normalization and is set by the staff; How long ago the file was created? :Calculated by the following aging formula:

[0014] In the formula, is the time-dependent attenuation factor, The time period from the creation time of the data file to the current time; Permission levels for data files : Scoring is performed based on the access permission level of the data file. The permission level of the scoring is set to a value from 0 to 1. The higher the permission, the higher the score. The access permission of the specific data file is the preset value.

[0015] As a further solution of the present invention: the specific parameter calculation method of each key factor in the access frequency index is: Historical visit frequency , select a longer time interval , get the time interval The total number of times the data file is accessed , through the following formula Determine historical visit frequency ,in, Indicates the highest number of accesses among all data files in time interval T1; Recent visit frequency , choose a shorter time interval , get the time interval The total number of times the data file is accessed , through the following formula Determine historical visit frequency ,in, Indicates time interval The highest number of accesses among all data files in the Visit Volatility , select a time interval , through the formula Determining access volatility ,in, The table shows the average daily access frequency, that is, the average access frequency in the selected interval, and the calculation formula is , where v is the time interval The total number of days in ; Indicates the standard deviation of visit frequency.

[0016] As a further solution of the present invention: the data center platform also includes a data reading module; The data reading module is used to check whether the data file exists on the distributed storage node through the data management software layer when the user accesses a data file; if not, the data file is retrieved from the Blu-ray storage device.

[0017] The present invention provides a data file storage system based on the combination of distributed storage and blue-ray storage. Compared with the prior art, it has the following beneficial effects: This data file storage system based on the combination of distributed storage and Blu-ray storage, through the combination of distributed storage nodes and Blu-ray storage devices, not only ensures fast access to high-frequency data, but also realizes the long-term and reliable preservation of low-frequency and important data. Distributed storage nodes provide efficient writing and reading performance, suitable for frequently accessed data needs, and Blu-ray storage has low storage costs and excellent durability, especially suitable for storing low-frequency, long-term retention data.

[0018] In addition, the data management software layer intelligently classifies the importance and access frequency of data files, realizes dynamic allocation and optimized management of resources, and ensures the reasonable storage of different data. The overall system not only improves data access efficiency and meets the dual requirements of real-time and security, but also reduces storage costs, providing efficient, economical and flexible solutions for various data storage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described below in conjunction with the accompanying drawings.

[0020] Figure 1 The structural framework diagram of the data file storage system based on the combination of distributed storage and Blu-ray storage of the present invention; DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Embodiment 1 See also Figure 1 , the present invention provides a data file storage system based on the combination of distributed storage and blue-ray storage, including; Data center platform, used to write, store, manage and read data files; The data center platform includes a data writing module, a data management software layer, a Blu-ray storage device, distributed storage nodes, and a data reading module; The data writing module is used to first write the data files to the distributed storage nodes, using their efficient writing speed to meet real-time storage needs; The data management software layer is used to classify data files according to their importance index and access frequency index, and classify them into important high-frequency data files, unimportant high-frequency data files, important non-high-frequency data files, and unimportant non-high-frequency data files, and store the data files in a corresponding manner according to the classification of the data files; The corresponding storage method includes storage by distributed storage nodes and Blu-ray storage devices; The specific method of classifying data files according to the importance index and access frequency index of data files is as follows: The importance index of the data file is determined by the following key factors. The key factors in the importance index include the value attribute of the file content, the creation time of the file, and the permission level of the data file, which are recorded as ; The specific parameter calculation method for each key factor in the importance index is: File content value attribute : Assign a range to the content value attribute of the data file , where 1 represents the information with the highest value and 0 represents ordinary data. The value is assigned by keyword-based calculation: a key business vocabulary is preset, and the keywords in the key business vocabulary are specifically set by the staff. For example, the key business vocabulary can include words such as "contract", "financial statement", "strategic planning", etc., and different weights are assigned to each keyword in the key business vocabulary. The data file content is scanned and the number and weight of these keywords are identified. The value attribute of the file content is determined by the following formula :

[0023] In the formula, Indicates The weight coefficient of each keyword, Indicates the first The number of times a keyword appears; It is a parameter used for normalization and is set by the staff; It should be noted that For example, suppose there are 10 keywords in the keyword library and the maximum possible weight of each keyword is 1; if it is assumed that each keyword appears 100 times in the data file (a set maximum number of occurrences), the possible maximum value can be calculated as follows:

[0024] Here, 1000 is the possible maximum value; this means that if the content in a file fully meets the conditions of the highest value (the frequency and weight of each keyword reach the maximum value), its The score is 1; How long ago the file was created? : Generally, the newer the data file, the more likely it is to be used recently; it is calculated using the following aging formula:

[0025] In the formula, is the time-dependent attenuation factor, The time period from the creation time of the data file to the current time; Permission levels for data files :Score the access rights level of the data files. The access rights level is set from 0 to 1. The higher the rights, the higher the score. The specific access rights of the data files are set by professional staff. For example, the access rights of ordinary users are 0.2, middle management is 0.5, and core management is 1. The importance index of the data file is determined and evaluated by the following formula based on the key factors of the file content value attribute, the file creation time, and the permission level of the data file. ;

[0026] In the formula, is the weight factor; Importance index of data files With the default value For comparison: like , then the data file is marked as an important data file; like , then the data file is marked as a non-important data file; Among them, the preset value The specific parameters are set by professional staff; The access frequency index of the data file is determined by the following key factors. The key factors in the access frequency index include historical access frequency, recent access frequency, and access volatility, which are recorded as , , ; The specific parameter calculation method for each key factor in the access frequency index is: Historical visit frequency , select a longer time interval , get the time interval The total number of times the data file is accessed , through the following formula Determine historical visit frequency ,in, Indicates the highest number of accesses among all data files in the time interval T1; select a longer time interval The time interval is selected by the staff. for half a year; Recent visit frequency , choose a shorter time interval , get the time interval The total number of times the data file is accessed , through the following formula Determine historical visit frequency ,in, Indicates time interval The highest number of accesses among all data files in the The time interval is selected by the staff. for one week; Visit Volatility , select a time interval , through the formula Determining access volatility ,in, The table shows the average daily access frequency, that is, the average access frequency in the selected interval, and the calculation formula is , where v is the time interval The total number of days in ; Indicates the standard deviation of access frequency; select a time interval The time interval is selected by the staff. 30 days; Based on the key factors of historical access frequency, recent access frequency, and access volatility, the access frequency index of the data file is determined and evaluated through the following formula ;

[0027] In the formula, is the weight factor; The access frequency index of the data file With the default value For comparison: like , then the data file is marked as a high-frequency data file; like , then the data file is marked as a non-high-frequency data file; Among them, the preset value The specific parameters are set by professional staff; Importance index for combining data files Frequency of visit index The data files are divided into four categories: important high-frequency data files, unimportant high-frequency data files, important non-high-frequency data files and unimportant non-high-frequency data files; The content of the corresponding storage method of the data file according to the classification of the data file is: if the data file is classified as an important high-frequency data file, an important non-high-frequency data file or a non-important non-high-frequency data file, the data file is stored through a Blu-ray storage device; If the data file is classified as an important high-frequency data file or a non-important high-frequency data file, the data file is stored through a distributed storage node; It should be noted that distributed storage nodes are suitable for storing frequently accessed data files, providing fast reading for frequently accessed data files to ensure system access efficiency; Blu-ray storage devices are suitable for storing data files that need to be stored for a long time and have a low access frequency, especially data of high importance; non-important and infrequently accessed data can also be placed in Blu-ray storage to reduce resource usage; dual storage: for "important and frequently accessed data files", distributed storage nodes and Blu-ray storage devices are used at the same time to achieve a balance between access speed and data security; The data management software layer plays an intelligent management role in the entire system. It classifies files according to their "importance index" and "access frequency index" and determines the storage method for each type of file. This classification mechanism effectively optimizes resource allocation and data access efficiency. For example, the system will store important and high-frequency data in distributed nodes to speed up access, while low-frequency files can be stored in Blu-ray storage devices to save costs. By dynamically evaluating the importance and access frequency of data, the data management software layer can also automatically adjust the data storage location, realize dynamic data management, reduce data redundancy, and avoid resource waste. At the same time, the classified storage strategy helps to balance long-term storage and high access speed, so that the system can meet performance requirements while ensuring the security of important data and the rationality of storage costs. Distributed storage nodes: used to store data files classified as important high-frequency data files and non-important high-frequency data files by the data management software layer. Distributed storage nodes usually use hard disk drives (HDDs) or solid-state drives (SSDs) and are deployed in a cluster manner so that data is distributed and stored among multiple nodes. A Blu-ray storage device for storing data files classified by the data management software layer as important high-frequency data files, important non-high-frequency data files, and non-important non-high-frequency data files, the Blu-ray storage device having an optical disc burner and a management system for reading and writing data; The data reading module is used to check whether the data file exists on the distributed storage node through the data management software layer when the user accesses a data file; if not, the data file is retrieved from the Blu-ray storage device.

[0028] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0029] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A data file storage system based on the combination of distributed storage and Blu-ray storage, characterized in that: include: Data center platform, used to write, store, manage and read data files; The data center platform includes data writing module, data management software layer, Blu-ray storage device, and distributed storage nodes; A data writing module is used to first write the data file into the distributed storage node; The data management software layer is used to classify data files according to their importance index and access frequency index, and classify them into important high-frequency data files, unimportant high-frequency data files, important non-high-frequency data files, and unimportant non-high-frequency data files, and store the data files in a corresponding manner according to the classification of the data files; Distributed storage nodes; used to store data files classified as important high-frequency data files and non-important high-frequency data files by the data management software layer; The Blu-ray storage device is used to store data files classified by the data management software layer as important high-frequency data files, important non-high-frequency data files, and non-important non-high-frequency data files.

2. The data file storage system based on the combination of distributed storage and Blu-ray storage according to claim 1 is characterized in that: In the data management software layer, the corresponding storage method includes storage using distributed storage nodes and Blu-ray storage devices.

3. The data file storage system based on the combination of distributed storage and Blu-ray storage according to claim 2 is characterized in that: The specific method of classifying data files according to the importance index and access frequency index of data files is as follows: The importance index of the data file is determined by the following key factors. The key factors in the importance index include the value attribute of the file content, the creation time of the file, and the permission level of the data file, which are recorded as ; The importance index of the data file is determined and evaluated by the following formula based on the key factors of the file content value attribute, the file creation time, and the permission level of the data file. ; In the formula, is the weight factor; Importance index of data files With the default value For comparison: like , then the data file is marked as an important data file; like , then the data file is marked as a non-important data file; The access frequency index of the data file is determined by the following key factors. The key factors in the access frequency index include historical access frequency, recent access frequency, and access volatility, which are recorded as , , ; Based on the key factors of historical access frequency, recent access frequency, and access volatility, the access frequency index of the data file is determined and evaluated through the following formula ; In the formula, is the weight factor; In the formula, is the weight coefficient, and ; The access frequency index of the data file With the default value For comparison: like , then the data file is marked as a high-frequency data file; like , then the data file is marked as a non-high-frequency data file.

4. The data file storage system based on the combination of distributed storage and Blu-ray storage according to claim 3 is characterized in that: The specific method of classifying the data files according to the importance index and the access frequency index of the data files also includes: Importance index for combining data files Frequency of visit index The data files are divided into four categories: important high-frequency data files, unimportant high-frequency data files, important non-high-frequency data files and unimportant non-high-frequency data files.

5. The data file storage system based on the combination of distributed storage and Blu-ray storage according to claim 3 is characterized in that: The specific parameter calculation method for each key factor in the importance index is: File content value attribute : Assign a range to the content value attribute of the data file , where 1 represents the information with the highest value and 0 represents ordinary data. The value is assigned based on keyword calculation: a preset key business vocabulary is used, and different weights are assigned to each keyword in the key business vocabulary. The data file content is scanned and the number and weight of these keywords are identified. The value attribute of the file content is determined by the following formula : In the formula, Indicates The weight coefficient of each keyword, Indicates the first The number of times a keyword appears; It is a parameter used for normalization and is set by the staff; How long ago the file was created? :Calculated by the following aging formula: In the formula, is the time-dependent attenuation factor, The time period from the creation time of the data file to the current time; Permission levels for data files : Scoring is performed based on the access permission level of the data file. The permission level of the scoring is set to a value from 0 to 1. The higher the permission, the higher the score. The access permission of the specific data file is the preset value.

6. The data file storage system based on the combination of distributed storage and Blu-ray storage according to claim 3 is characterized in that: The specific parameter calculation method of each key factor in the access frequency index is as follows: Historical visit frequency , select a longer time interval , get the time interval The total number of times the data file is accessed , through the following formula Determine historical visit frequency ,in, Indicates the highest number of accesses among all data files in time interval T1; Recent visit frequency , choose a shorter time interval , get the time interval The total number of times the data file is accessed , through the following formula Determine historical visit frequency ,in, Indicates time interval The highest number of accesses among all data files in the Visit Volatility , select a time interval , through the formula Determining access volatility ,in, The table shows the average daily access frequency, that is, the average access frequency in the selected interval, and the calculation formula is , where v is the time interval The total number of days in ; Indicates the standard deviation of visit frequency.

7. The data file storage system based on the combination of distributed storage and Blu-ray storage according to claim 1, characterized in that: The data center platform also includes a data reading module; The data reading module is used to check whether the data file exists on the distributed storage node through the data management software layer when the user accesses a data file; if not, the data file is retrieved from the Blu-ray storage device.

Images