Distributed storage resource scheduling modeling method

Abstract

The invention relates to the technical field of data processing, in particular to a distributed storage resource scheduling modeling method, which comprises the following steps of data classification, equipment definition, distribution matrix construction, objective function setting, constraint condition setting and solution optimization. The method has the beneficial effects that by optimizing the allocation strategy of the data among the storage devices, the overhead of data migration and duplication is reduced, and the utilization rate of storage resources and the overall storage efficiency are improved; by introducing a data value coefficient function, the storage priority of the data is dynamically adjusted according to factors such as timeliness, importance and sensitivity of the data, and key data is ensured to be timely and effectively stored and protected, so that the overall value of the data is improved; and through a distributed storage and redundant backup mechanism, the influence of a single-point fault on the system is reduced, and the reliability and stability of the system are improved. Meanwhile, the system performance and the data storage condition are monitored in real time, potential problems are found and processed in time, and the robustness of the system is further enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a distributed storage resource scheduling modeling method. Background Art

[0002] During the multi-domain mission operation, the mission action end uses distributed storage to connect various storage devices through the network. i Connect and aggregate data to form a data pool. Assume that the set of mobile edge storage devices for a certain mission action is Ω = {Q1, Q2, Q3, .... Q n}, n is the total number of storage devices, for a storage device Q i Generally speaking, there are two speeds for storage device data throughput, namely sequential read and write speed u i And random read and write speed v i , only from the perspective of data storage, the types of files can be classified into "large files" and "small files". "Large files" use sequential read and write speed u i , "Small File" D * Using random read and write speed v i The type of a file is not determined by its size, but by whether the data file has internal logic.

[0003] In modern mission operations environments, real-time, efficient, and secure storage of data is essential for the formulation and execution of mission operations decisions. With the continuous improvement of the informatization of mission operations, the amount of data generated by mission operations has increased dramatically, and the data types are diversified, including large files (such as high-definition videos, large images, etc.) and small files (such as sensor data, text information, etc.). These data have different requirements for storage resources. Large files require high-speed, large-capacity storage devices to ensure their integrity and availability, while small files focus more on storage flexibility and access speed.

[0004] However, traditional centralized storage solutions have many limitations in mission action environments, such as high risk of single point failure, poor scalability, and difficulty in adapting to dynamically changing mission action requirements. Therefore, distributed storage systems have become an ideal choice for mission action mobile escort support due to their high availability, scalability, and fault tolerance. Distributed storage systems deploy multiple storage nodes in the network and store data in multiple nodes in a dispersed manner, thereby achieving redundant backup and load balancing of data. However, distributed storage systems also face challenges in resource scheduling and management. How to reasonably allocate storage resources based on the characteristics of data and the performance of storage devices to achieve the dual goals of storage efficiency and data value maximization is one of the current hot topics in distributed storage technology research. Summary of the invention

[0005] The purpose of the present invention is to provide a distributed storage resource scheduling modeling method to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a distributed storage resource scheduling modeling method, the method comprising the following steps:

[0007] Data classification: divide the data to be stored into large file data and small file data;

[0008] Device definition: define a data acquisition and transmission device set and a storage device set, where the storage device has a storage speed function based on the data type;

[0009] Allocation matrix construction: construct a matrix that represents the distribution of data among storage devices;

[0010] Objective function setting: define an objective function that takes storage time and data value into consideration;

[0011] Constraint setting: Set storage limits for storage devices and total amount of data;

[0012] Solving optimization: Use mathematical programming methods to solve the optimal data allocation strategy.

[0013] Preferably, each device in the storage device set has independent storage speed and capacity characteristics.

[0014] Preferably, the allocation matrix includes a large file allocation matrix and a small file allocation matrix, which respectively record in detail the storage status of each type of file on each storage device.

[0015] Preferably, the objective function balances the importance of storage time and data value in a weighted manner, and the weight coefficient can be adjusted to meet the needs of different mission actions.

[0016] Preferably, the constraints include minimum and maximum storage capacity limits of the storage device, and a total amount limit of the storage data required for the entire mission action.

[0017] Preferably, real-time monitoring of storage device performance is also included to dynamically adjust data allocation strategies.

[0018] Preferably, the data value coefficient function is dynamically adjusted according to factors such as timeliness, importance and sensitivity of the data.

[0019] Preferably, a fault recovery mechanism is also included, which automatically migrates the data stored on a storage device to other available devices when a storage device fails.

[0020] Preferably, a user interface is also included to allow the user to adjust model parameters, such as weight coefficients and value coefficient functions, according to actual needs.

[0021] Preferably, the method is integrated into a distributed storage management system that automatically performs data storage, allocation, monitoring and optimization tasks to improve data storage efficiency and data value utilization.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The distributed storage resource scheduling modeling method proposed in the present invention reduces the overhead of data migration and replication by optimizing the data allocation strategy among storage devices, thereby improving the utilization rate of storage resources and the overall storage efficiency. By introducing the data value coefficient function, the storage priority of the data is dynamically adjusted according to factors such as the timeliness, importance and sensitivity of the data, thereby ensuring that key data is stored and protected in a timely and effective manner, thereby improving the overall value of the data. Through distributed storage and redundant backup mechanisms, the impact of single point failures on the system is reduced, and the reliability and stability of the system are improved. At the same time, the system performance and data storage status are monitored in real time, potential problems are discovered and handled in a timely manner, and the robustness of the system is further enhanced.

[0024] It supports dynamic adjustment of model parameters and scheduling strategies according to changes in the mission action environment and mission action requirements to adapt to different storage requirements and optimization goals. This flexibility makes this method more adaptable and practical in complex and changing mission action environments; by providing a user interface, it allows users to adjust model parameters and view system operation status according to actual needs, thereby improving user participation and satisfaction. At the same time, automated data storage, allocation, monitoring and optimization tasks reduce the possibility of manual intervention and errors, and improve the ease of use and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] See also Figure 1The present invention provides a technical solution: a distributed storage resource scheduling modeling method, the method comprising the following steps:

[0028] Data classification: divide the data to be stored into large file data and small file data;

[0029] Device definition: defines a data acquisition and transmission device set and a storage device set. The storage device has a storage speed function based on the data type. Each device in the storage device set has independent storage speed and capacity characteristics.

[0030] Allocation matrix construction: construct a matrix that represents the distribution of data among storage devices; the allocation matrix includes a large file allocation matrix and a small file allocation matrix, which respectively record in detail the storage status of each type of file on each storage device.

[0031] Objective function setting: Define an objective function that comprehensively considers storage time and data value; the objective function balances the importance of storage time and data value in a weighted manner, and the weight coefficient can be adjusted to meet the needs of different task actions; the data value coefficient function is dynamically adjusted according to the timeliness, importance and sensitivity of the data.

[0032] Constraint setting: Set storage limits for storage devices and total amount of data; constraints include minimum and maximum storage capacity limits for storage devices, and total amount of storage data required for the entire mission action.

[0033] Solving optimization: Use mathematical programming methods to solve the optimal data allocation strategy.

[0034] It also includes real-time monitoring of storage device performance to dynamically adjust data allocation strategies.

[0035] It also includes a fault recovery mechanism that automatically migrates the data stored on a storage device to other available devices when a storage device fails.

[0036] It also includes a user interface that allows users to adjust model parameters such as weight coefficients and value coefficient functions according to actual needs.

[0037] The method is integrated into a distributed storage management system, which automatically performs data storage, allocation, monitoring and optimization tasks to improve data storage efficiency and data value utilization.

[0038] Embodiment 2, based on Embodiment 1, specifically proposes:

[0039] For the amount of data ΔD generated by a certain task action, from the perspective of data storage, it can be divided into large files D^ and small files D * They are stored at different speeds, and their composition can be expressed as i and j are the number of large and small data files respectively, and the set of data acquisition and transmission equipment is expressed as Where eqt is the number of acquisition devices, storage device Q i Storage speed function Depends on the type of data:

[0040]

[0041] The collected data is stored in different devices Q n Use matrix and The composition is expressed as:

[0042]

[0043] Represents a large file data set D stored on the nth storage device i ^The amount of data. The composition is expressed as:

[0044]

[0045] Represents a collection of small file data stored on the nth storage device The amount of data. Task-oriented mobile accompanying support distributed storage nodes, different nodes have different functions, capacity nodes store as much data as possible, performance nodes save data as short a time as possible, objective function F fit (D ij )for:

[0046]

[0047] F time (D i ) function represents the time required to store data, F data (D) function represents the value of the amount of data stored, where g(D j ) represents data D j The value coefficient size is subject to the constraint that for a storage device Q i The amount of data stored must be greater than the minimum military storage capacity and less than the maximum storage capacity The amount of data ΔD required to store for the entire mission action is greater than the minimum storage capacity of the storage device and less than the sum of the maximum capacity of the storage device. j ) represents the value coefficient of the amount of stored data, the range of the function is (0,1), and λ is the range of its value (0,1).

[0048] During the data storage process, the speed of data reading is affected by the pointer, chip control, and temperature in the storage device, and the influence is negligible; the time for switching the reading speed for different file types is negligible during the data storage process; the effect of the wear and tear of hard disk devices and other devices during the data storage reading and writing process on the reading and writing speed is negligible.

[0049] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A distributed storage resource scheduling modeling method, characterized in that: The method comprises the following steps: Data classification: divide the data to be stored into large file data and small file data; Device definition: define a collection of data acquisition and transmission devices and a collection of storage devices, where the storage devices have a storage speed function based on the data type; Allocation matrix construction: construct a matrix that represents the distribution of data among storage devices; Objective function setting: define an objective function that takes into account storage time and data value; Constraint setting: Set storage limits for storage devices and total amount of data; Solving optimization: Use mathematical programming methods to solve the optimal data allocation strategy.

2. The distributed storage resource scheduling modeling method according to claim 1 is characterized in that: Each device in the storage device set has independent storage speed and capacity characteristics.

3. The distributed storage resource scheduling modeling method according to claim 2 is characterized in that: The allocation matrix includes a large file allocation matrix and a small file allocation matrix, which respectively record in detail the storage status of each type of file on each storage device.

4. The distributed storage resource scheduling modeling method according to claim 1, characterized in that: The objective function balances the importance of storage time and data value in a weighted manner, and the weight coefficient can be adjusted to meet the needs of different mission actions.

5. The distributed storage resource scheduling modeling method according to claim 1, characterized in that: The constraints include the minimum and maximum storage capacity limits of the storage device, and the total amount of storage data required for the entire mission action.

6. The distributed storage resource scheduling modeling method according to claim 1, characterized in that: It also includes real-time monitoring of storage device performance to dynamically adjust data allocation strategies.

7. The distributed storage resource scheduling modeling method according to claim 4 is characterized in that: The data value coefficient function is dynamically adjusted according to the timeliness, importance and sensitivity factors of the data.

8. The distributed storage resource scheduling modeling method according to claim 1, characterized in that: It also includes a fault recovery mechanism that automatically migrates the data stored on a storage device to other available devices when a storage device fails.

9. The distributed storage resource scheduling modeling method according to claim 1, characterized in that: It also includes a user interface that allows users to adjust model parameters such as weight coefficients and value coefficient functions according to actual needs.

10. The distributed storage resource scheduling modeling method according to claim 1, characterized in that: The method is integrated into a distributed storage management system, which automatically performs data storage, allocation, monitoring and optimization tasks to improve data storage efficiency and data value utilization.

Images