Method and apparatus for optimizing data locality in distributed storage systems

Abstract

The application provides a data locality optimization method and device in a distributed storage system, and relates to the technical field of data access, wherein the method comprises the following steps: collecting data access history records and server temperature monitoring data, then extracting access mode data and hot spot information and constructing a candidate path set; then combining a heat dissipation control parameter set and a path optimization algorithm to generate path evaluation parameters, screening out an optimized path meeting the heat dissipation requirement, and generating a score result from the dimensions of spatial proximity and time continuity through an evaluation function, and finally dynamically adjusting the path configuration according to the score result. The application improves the data access continuity of the distributed storage system and the stability of hardware operation.

Description

Technical Field

[0001] This application relates to the field of data access technology, and in particular to a method and apparatus for optimizing data locality in a distributed storage system. Background Technology

[0002] With the rapid development of high-performance computing and cloud computing technologies, distributed storage systems face the challenge of handling large-scale concurrent data access requests. At the same time, in practical application scenarios such as scientific computing and artificial intelligence training, the system not only needs to maintain high throughput and low latency data access capabilities, but also must ensure the stability of hardware devices under long-term high load operation, which undoubtedly places higher demands on data locality optimization.

[0003] To address this technical requirement, existing solutions employ a dynamic path selection method based on access history analysis. This method monitors the frequency of metadata access and path response time, then uses machine learning algorithms to identify high-frequency access patterns and dynamically selects the optimal data transmission path based on real-time network conditions. This solution also introduces a load balancing mechanism to distribute access requests to different storage nodes, thereby avoiding the problem of overheating or overload of a single node.

[0004] However, this solution has limitations in hardware thermal management. Its path selection mainly considers network performance indicators, but it is not timely enough in responding to changes in server temperature. Under long-term high load operation, the increase in hardware temperature may affect system stability. The existing solution lacks a coordinated consideration of heat dissipation requirements in the path optimization process, which may lead to local overheating and trigger frequency reduction protection, thus affecting the overall performance. Therefore, this optimization method that focuses on a single dimension may be difficult to maintain stable performance output in continuous high load scenarios. Summary of the Invention

[0005] This application provides a method and apparatus for optimizing data locality in a distributed storage system, in order to solve the problems of poor data access continuity and low hardware operation stability in the existing distributed storage system.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for optimizing data locality in a distributed storage system, comprising:

[0007] Collect historical data access data of the distributed storage system and temperature monitoring data of the metadata server. The historical data access data includes access frequency data and path latency data.

[0008] Big data pattern analysis is performed on the access frequency data and the path delay data to extract access pattern data and hotspot information, and a candidate path set is constructed based on the hotspot information;

[0009] A set of heat dissipation control parameters corresponding to the temperature monitoring data is generated. A path optimization algorithm is used to collaboratively process the access mode data and the set of heat dissipation control parameters to generate path evaluation parameters.

[0010] Based on the path evaluation parameters, an optimized path that meets the preset heat dissipation control requirements is selected from the candidate path set, and an evaluation function is used to comprehensively score the optimized path in terms of spatial proximity and temporal continuity to generate a score result.

[0011] Generate a path adjustment instruction corresponding to the scoring result, and adjust the optimized path according to the path adjustment instruction to achieve local optimization of metadata storage.

[0012] Optionally, the step of generating a heat dissipation control parameter set corresponding to the temperature monitoring data, and using a path optimization algorithm to collaboratively process the access pattern data and the heat dissipation control parameter set to generate path evaluation parameters, includes:

[0013] A sliding window scan is performed on the temperature monitoring data to calculate the temperature change rate of each of the processing unit and the storage controller;

[0014] Based on the temperature change rate and the current temperature value, a set of heat dissipation control parameters is generated;

[0015] Dynamic weight allocation is performed between the access mode data and the heat dissipation control parameter set;

[0016] The transmission efficiency score and heat dissipation adaptation score of each candidate path in the candidate path set are calculated using a path optimization algorithm.

[0017] Based on the dynamic weight allocation results, the transmission efficiency score and the heat dissipation adaptation score are weighted and fused to generate path evaluation parameters.

[0018] Secondly, this application provides a data locality optimization apparatus in a distributed storage device, comprising:

[0019] The acquisition module is used to acquire the data access history of the distributed storage system and the temperature monitoring data of the metadata server. The data access history includes access frequency data and path delay data.

[0020] The analysis module is used to perform big data pattern analysis on the access frequency data and the path delay data to extract access pattern data and hotspot information, and to construct a candidate path set based on the hotspot information.

[0021] The generation module is used to generate a set of heat dissipation control parameters corresponding to the temperature monitoring data, and to use a path optimization algorithm to collaboratively process the access mode data and the set of heat dissipation control parameters to generate path evaluation parameters.

[0022] The filtering module is used to filter out optimized paths that meet preset heat dissipation control requirements from the candidate path set according to the path evaluation parameters, and use the evaluation function to comprehensively score the optimized paths in terms of spatial proximity and temporal continuity to generate a score result.

[0023] The adjustment module is used to generate path adjustment instructions corresponding to the scoring results, and adjust the optimized path according to the path adjustment instructions to achieve local optimization of metadata storage.

[0024] Thirdly, this application provides an electronic device, comprising:

[0025] Memory, used to store computer programs;

[0026] A processor, configured to implement the steps of the data locality optimization method in a distributed storage device as described in the first aspect above when executing the computer program.

[0027] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the data locality optimization method in the distributed storage device described in the first aspect above.

[0028] The technical solution provided in this application has the following beneficial effects:

[0029] This application achieves a holistic understanding of the system's operating status and hardware working environment by comprehensively collecting data access history records and temperature monitoring data from the metadata server of the distributed storage system. Then, it utilizes big data pattern analysis technology to process access frequency and path latency information, accurately identifying hot access areas and frequent access periods in the system, and constructing a candidate path set to provide a clear basis for path optimization. Next, it generates a set of heat dissipation control parameters corresponding to the temperature monitoring data, and combines this with a path optimization algorithm to perform collaborative analysis of access patterns and heat dissipation status, generating path evaluation parameters. This organically combines data access characteristics with hardware heat dissipation status, ensuring that path selection improves performance while also considering hardware safety and heat dissipation requirements.

[0030] Based on this, optimized paths that meet preset heat dissipation control requirements are selected according to path evaluation parameters. The paths are then comprehensively scored in terms of spatial proximity and temporal continuity using an evaluation function. This avoids performance degradation caused by overheating, improves system stability, and enhances the continuity and locality of data access from both spatial and temporal dimensions. Finally, path adjustment instructions are generated based on the scoring results, driving the system to dynamically configure and adjust the optimized paths. This enables dynamic optimization of path configuration and continuously improves the overall system performance and data access efficiency.

[0031] Furthermore, this application also monitors hardware temperature changes in real time and, combined with data access pattern characteristics, employs a dynamic weight allocation mechanism to coordinate transmission efficiency and heat dissipation requirements. It then uses a multi-dimensional scoring system to generate optimal path evaluation parameters. Therefore, this application achieves synergistic optimization of data access performance and hardware heat dissipation status, improving data access efficiency while ensuring stable system operation, thereby effectively avoiding performance fluctuations caused by overheating.

[0032] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating a data locality optimization method in a distributed storage system, as provided in this application embodiment;

[0035] Figure 2 This application provides a schematic diagram illustrating a specific implementation of a data locality optimization method in a distributed storage system.

[0036] Figure 3 This is a schematic diagram of the structure of a data locality optimization device in a distributed storage system provided in an embodiment of this application. Detailed Implementation

[0037] In the field of data locality optimization in distributed storage systems, existing technical solutions mainly rely on dynamic path selection mechanisms based on network performance indicators. These solutions optimize path selection by analyzing historical access patterns and data transmission latency. However, in practical applications, they ignore the impact of hardware thermal effects on system stability under high-concurrency access. Due to the lack of coordinated consideration of server temperature status, existing solutions may experience local overheating during long-term high-load operation, which may trigger hardware protection mechanisms, leading to performance fluctuations and decreased system reliability. This makes it difficult to meet the stringent requirements for continuous and stable operation in high-performance computing scenarios.

[0038] To address the aforementioned issues, this application proposes a data locality optimization method for distributed storage systems. The core idea of ​​this method is to innovatively coordinate heat dissipation monitoring with data path optimization. Specifically, the method first collects multiple state parameters during system operation, including data access characteristics and hardware temperature data. Then, a collaborative analysis mechanism is used to simultaneously consider data transmission efficiency and heat dissipation requirements, dynamically generating the optimal path selection scheme. By introducing a heat-aware path evaluation system, this method can effectively control hardware operating temperature while ensuring data transmission efficiency, avoiding performance degradation caused by overheating.

[0039] Therefore, this collaborative optimization mechanism ensures that the system can maintain stable performance output under high load conditions, while extending the lifespan of hardware devices and providing more reliable storage system support for high-performance computing scenarios.

[0040] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] The core of this application is to provide a method for optimizing data locality in a distributed storage system, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0042] Step 101: Collect the data access history of the distributed storage system and the temperature monitoring data of the metadata server. The data access history includes access frequency data and path delay data.

[0043] In step 101, the data access history refers to the historical log information recorded in the distributed storage system that records the access of users or applications to file metadata. The aforementioned metadata may include file name, file size, file location, etc.

[0044] The metadata server is the core management component of a distributed storage system. It is responsible for maintaining metadata information such as file system namespaces, access permissions, and data block location mappings. A distributed storage system is a complete storage architecture composed of a metadata server and multiple data storage nodes.

[0045] Access frequency data refers to the statistical information on the number of times a certain metadata item or access path is accessed within a specific time period, which is used to identify hot data.

[0046] Path latency data refers to the time it takes for data to be transmitted through different network paths in a storage system, and is used to measure path performance.

[0047] Temperature monitoring data refers to the real-time operating temperature data of the main hardware components on the metadata server, collected through sensors.

[0048] For example, in a distributed storage system used for scientific computing, a monitoring agent continuously collects metadata access logs and calculates the access frequency of path P1 in the last 5 minutes as 1200 times, with a path latency of 15 milliseconds. Additionally, temperature sensors collect data showing the processor temperature of metadata server node A at 75 degrees Celsius and the storage controller temperature at 68 degrees Celsius. This data is then packaged into data packets and sent to the analysis center. The access frequency and path latency data are derived from historical data access records, while the temperature monitoring data comes from hardware sensors.

[0049] Step 102: Perform big data pattern analysis on the access frequency data and the path delay data to extract access pattern data and hotspot information, and construct a candidate path set based on the hotspot information.

[0050] In step 102, access pattern data refers to regular access characteristic information obtained by performing time-series analysis on access frequency data, such as periodic access patterns.

[0051] Hotspot information refers to high-performance, high-traffic path segments identified through correlation analysis of path latency data and access frequency. The candidate path set refers to a collection of potentially high-performance paths selected from all available paths based on hotspot information.

[0052] In this embodiment of the application, after receiving the data uploaded in step 101, the big data analysis engine is immediately started to perform time series analysis on the access frequency data to identify recurring periodic high-frequency access periods, thereby forming access pattern data. At the same time, spatial clustering analysis is performed on the path delay data to mark the path segments with low latency and high access frequency as hotspot information.

[0053] Next, based on the physical locations corresponding to these hotspots, the network topology of the distributed storage system is traversed to find all complete paths that pass through these hotspot areas. Then, the paths are sorted according to the number of hops and historical bandwidth capacity, and finally, a set of candidate paths is selected from the top-ranked high-quality paths.

[0054] For example, after conducting in-depth analysis of the received data, the analysis center found that the access frequency increased significantly between 10:00 and 11:00 every day. Therefore, the access pattern data for this specific time period was extracted. At the same time, the analysis also showed that the path segment connecting node A, node B and node C had a latency of less than 20 milliseconds and was frequently accessed. Therefore, this path segment was marked as hotspot information.

[0055] Based on this hotspot information, three complete paths passing through this segment are selected from all paths: path X passes through nodes A, B, C, and D; path Y passes through nodes A, B, C, and E; and path Z passes through nodes A, F, and C. Then, they are sorted according to the rules of fewer hops and larger bandwidth capacity. Path X scores 90 points, path Y scores 85 points, and path Z scores 80 points. Finally, the first two paths are selected to construct a candidate path set.

[0056] Step 103: Generate a set of heat dissipation control parameters corresponding to the temperature monitoring data, and use a path optimization algorithm to perform collaborative processing on the access mode data and the set of heat dissipation control parameters to generate path evaluation parameters.

[0057] In step 103, the heat dissipation control parameter set refers to the set derived from temperature monitoring data, whose core includes heat dissipation level, such as high, medium and low, and can be further mapped to specific heat dissipation control commands, such as fan speed and heat dissipation device power level; the path evaluation parameter refers to the quantitative index value calculated by the algorithm and used to comprehensively evaluate the merits of a path.

[0058] In this embodiment of the application, based on the temperature monitoring data collected in step 101, the temperature change trend and the current temperature value are first calculated, and then a set of heat dissipation control parameters containing heat dissipation level and expected heat dissipation performance is generated. Then, the access mode data obtained in step 102 and the heat dissipation control parameter set generated in this step are input into the path optimization algorithm. The algorithm adopts a multi-objective optimization method, which simultaneously considers the data transmission efficiency reflected by the access mode data and the heat dissipation requirements reflected by the heat dissipation control parameter set, and calculates a comprehensive score that balances performance and heat dissipation for each path in the candidate path set, i.e., path evaluation parameters.

[0059] For example, based on the received temperature monitoring data, where the processor temperature is 75 degrees Celsius and the storage controller temperature is 68 degrees Celsius, the calculated set of thermal control parameters indicates a high thermal performance level and a required cooling power of 80 watts. Then, the previously obtained access pattern data, i.e., the data from the high-frequency access period from 10:00 AM to 11:00 AM, and this set of thermal control parameters are input into the path optimization algorithm. This algorithm calculates the values ​​for path X and path Y in the candidate path set. Although path X has high transmission efficiency, the node temperatures it passes through are relatively high, while path Y has slightly lower transmission efficiency but better node temperatures. After weighted calculation, the path evaluation parameters for path X are 85 points and those for path Y are 88 points. This score comprehensively considers both performance and thermal requirements.

[0060] Step 104: Based on the path evaluation parameters, select the optimized path that meets the preset heat dissipation control requirements from the candidate path set, and use the evaluation function to comprehensively score the optimized path in terms of spatial proximity and temporal continuity to generate a score result.

[0061] In step 104, the preset heat dissipation control requirements refer to the temperature threshold range set to ensure the stable operation of the metadata server. For example, the heat dissipation control requirements specifically include hardware thermal specifications such as: the processor core temperature does not exceed 85 degrees Celsius, the storage controller chip temperature does not exceed 70 degrees Celsius, and the temperature change rate does not exceed 5 degrees Celsius / minute.

[0062] An optimized path refers to the best path selected from the set of candidate paths that satisfies both performance and heat dissipation requirements.

[0063] The evaluation function is a mathematical function used to quantify the strength of locality of access to data. It should be noted that the specific expression used for this function is not specifically limited in this embodiment, and can be set according to the actual situation.

[0064] The scoring result refers to the score calculated by the evaluation function, which reflects the quality of data access locality.

[0065] In this embodiment, the path evaluation parameters of each path calculated in step 103 are compared with preset heat dissipation control requirements such as temperature thresholds. The path with the highest score and meeting the heat dissipation requirements is selected from the candidate path set as the optimized path. Then, for this selected optimized path, an evaluation function is used to conduct a deeper evaluation. Specifically, the concentration of physical locations of data blocks stored on this path and the continuity of historical access time are analyzed. The concentration of physical locations reflects spatial proximity, and the continuity of historical access time reflects temporal continuity. Finally, a comprehensive score result is calculated to predict the efficiency of accessing data along this path.

[0066] For example, the preset heat dissipation control requirements stipulate that the temperature of all nodes on the path must be below 80 degrees Celsius. According to the path evaluation parameters, the score of 88 for path Y is better than that of 85 for path X, and the temperature of all nodes along path Y is below 75 degrees Celsius. Therefore, path Y is selected as the optimized path. Subsequently, the storage of data blocks on path Y was analyzed and it was found that they are concentrated on three adjacent racks with high spatial proximity. At the same time, the analysis of access records showed that these data blocks are often accessed continuously in a short period of time, and the temporal continuity is good.

[0067] The expression for the evaluation function can be: ,in Indicates the scoring result. Represents spatial proximity score, This represents the score for continuity over time.

[0068] For example, when 90 points and Given a score of 95, the scoring result is... .

[0069] Step 105: Generate a path adjustment instruction corresponding to the scoring result, and adjust the optimized path according to the path adjustment instruction to achieve local optimization of metadata storage.

[0070] In step 105, the path adjustment instruction refers to the control command generated based on the scoring results, which is used to guide the distributed storage system to adjust the data storage path or access strategy;

[0071] Locality optimization of metadata storage refers to the process of adjusting the physical storage of frequently accessed metadata to reduce access latency and improve efficiency.

[0072] In this embodiment of the application, after receiving the score result calculated in step 104, it is first compared with a preset performance threshold, and then a corresponding path adjustment instruction is generated based on the comparison result. That is, if the score is high, it indicates that the optimized path should be used first; if the score is low, it indicates that the use of the path should be reduced or a new path should be selected; if the score is medium, it indicates that the path should be used with a specific weight. Finally, the control component of the distributed storage system executes these instructions to optimize the access locality of metadata by adjusting the data storage location or routing strategy.

[0073] For example, the preset highest score threshold is 90 points, the lowest score threshold is 70 points, and the score obtained in step 104 is 92 points, which is higher than the highest score threshold. Therefore, a path adjustment instruction is generated, which strengthens the use of path Y and increases its priority weight. After receiving the instruction, the storage system controller immediately adjusts the existing routing table to prioritize subsequent requests for metadata in this area to path Y, so that relevant data access is more concentrated through this efficient and stable path, thereby realizing local optimization of metadata storage and improving overall access performance and system reliability.

[0074] This method collects and analyzes the data access characteristics and hardware heat dissipation status of the system in real time, and then dynamically selects and adjusts the data transmission path. This enables the distributed storage system to intelligently balance data processing efficiency and hardware heat dissipation requirements in high-load scenarios such as high-performance computing. Ultimately, it improves the continuity of data access and the overall system performance while ensuring the long-term stable operation of the hardware.

[0075] To address the issue of coordinating system performance and hardware heat dissipation during path optimization and further improve the stability of distributed storage systems under high loads, some embodiments include step 103: generating a set of heat dissipation control parameters corresponding to the temperature monitoring data, and using a path optimization algorithm to collaboratively process the access pattern data and the heat dissipation control parameter set to generate path evaluation parameters, such as... Figure 2 The above includes:

[0076] Step 201: Perform a sliding window scan on the temperature monitoring data to calculate the temperature change rate of the processing unit and the storage controller respectively.

[0077] In step 201, the processing unit refers to the central processing unit deployed in the metadata server, which means that it is the core computing component that performs metadata access request processing and path calculation tasks.

[0078] A storage controller is a disk array control unit deployed in a metadata server. It is a storage control component that manages metadata read / write operations and data caching.

[0079] The rate of temperature change refers to the amount of temperature change of the processing unit and storage controller per unit time. This parameter reflects the dynamic trend of hardware temperature change.

[0080] In this embodiment, a fixed-duration time window is used to perform a sliding scan of the temperature monitoring data continuously collected in step 101. After each window slide, the slope of the temperature data of the processing unit and the storage controller within the window is calculated, i.e., the rate of temperature change, to obtain the latest temperature change trend data of both. It should be noted that this embodiment does not limit the size of the fixed duration and can be set according to the actual situation.

[0081] Step 202: Generate a set of heat dissipation control parameters based on the temperature change rate and the current temperature value.

[0082] In step 202, the current temperature value is the monitoring data obtained in real time by the temperature sensor. It means the instantaneous temperature reading on the surface of the processing unit and the storage controller chip, which is used to reflect the real-time operating temperature status of the hardware components.

[0083] In this embodiment of the application, after comprehensively analyzing the temperature change rate calculated in step 201 and the current temperature value collected in real time, a corresponding heat dissipation control parameter set is generated according to the speed of temperature change and the current temperature level through a predefined mapping rule. This parameter set includes the heat dissipation level and specific control parameters required by the system at present.

[0084] The logic of the above mapping rule can include: if the current temperature is higher than the threshold and the rate of temperature change is positive, then the generated heat dissipation level is high. The mapping rule can be expressed in the form of a lookup table, calculation formula, decision tree, or configuration parameters. It should be noted that this embodiment does not limit the specific setting basis or expression form of the mapping rule, and can be set accordingly according to the actual situation.

[0085] Step 203: Perform dynamic weight allocation on the access mode data and the heat dissipation control parameter set.

[0086] In this embodiment of the application, the weight ratio of access mode data and heat dissipation control parameter set in subsequent scoring is dynamically adjusted according to the urgency of heat dissipation level in heat dissipation control parameter set. Specifically, when the heat dissipation level is higher than the preset heat dissipation level, the weight of heat dissipation control parameter set is greater than the weight of access mode data, and when the heat dissipation level is lower than the preset heat dissipation level, the weight of access mode data is greater than the weight of heat dissipation control parameter set.

[0087] Step 204: Calculate the transmission efficiency score and heat dissipation adaptation score of each candidate path in the candidate path set using a path optimization algorithm.

[0088] In step 204, this embodiment does not limit the type of path optimization algorithm. The transmission efficiency score is a quantitative value that evaluates the data transmission capability of the path, and the heat dissipation adaptation score is a quantitative value that evaluates the heat dissipation adaptability of the path.

[0089] In this embodiment, the transmission efficiency score and heat dissipation adaptation score of each path in the candidate path set are calculated by the path optimization algorithm. The transmission efficiency score can be calculated based on the historical delay data and bandwidth data of the path, while the heat dissipation adaptation score can be calculated based on the degree of matching between the temperature data of the nodes through which the path passes and the heat dissipation control parameter set. This embodiment does not limit the calculation formula of the above two scores, and can set them according to the actual situation.

[0090] Step 205: Based on the dynamic weight allocation result, the transmission efficiency score and the heat dissipation adaptation score are weighted and fused to generate path evaluation parameters.

[0091] In this embodiment of the application, based on the dynamic weight allocation result determined in step 203, the transmission efficiency score and heat dissipation adaptation score of each path are weighted and summed. The two scores are then fused according to the weight ratio to generate the final path evaluation parameter, which takes into account both the performance of the path and its heat dissipation adaptability.

[0092] Here is a specific example:

[0093] In the scenario of the distributed storage system for scientific research computing, the received temperature monitoring data is scanned using a sliding window with a time window of 5 minutes. The temperature change rate of the processing unit is calculated to be 2 degrees Celsius per minute, while the temperature change rate of the storage controller is 1.5 degrees Celsius per minute.

[0094] Then, based on these rates of temperature change, combined with the current temperature values—75 degrees Celsius for the processing unit and 68 degrees Celsius for the storage controller—a predefined thermal urgency scoring formula is used. The system calculates a heat dissipation urgency score, then determines the final heat dissipation level based on a preset mapping rule between the score and the level, and finally generates a set of heat dissipation control parameters. Indicates the urgency of heat dissipation. Indicates the rate of temperature change. Indicates the current temperature value;

[0095] For example, coefficient The value is 2, and the coefficient is... With a value of 0.1, the heat dissipation urgency score of the processing unit is [value missing]. The thermal urgency rating of the storage controller is [missing information]. Under the mapping rule that O is greater than or equal to 10, corresponding to a high heat dissipation level; O is greater than or equal to 5 and less than 10, corresponding to a medium heat dissipation level; and O is less than 5, corresponding to a low heat dissipation level, the heat dissipation level of the storage controller is determined to be high, and the heat dissipation level of the storage controller is determined to be medium.

[0096] Subsequently, dynamic weight allocation is performed on the access mode data and the heat dissipation control parameter set. For example, the weight is set according to the heat dissipation level quantization result: the "high" heat dissipation level is quantized to a value of 10, the "medium" level is quantized to 5, and the total heat dissipation quantization value is 15. The maximum heat dissipation quantization threshold is set to 25, then the heat dissipation weight is calculated as 15 ÷ 25 = 0.6, and the access mode data weight is 1 − 0.6 = 0.4 accordingly.

[0097] Then, the transmission efficiency score and heat dissipation adaptation score of path X and path Y in the candidate path set are calculated by the path optimization algorithm. The transmission efficiency score of path X is 90 points and the heat dissipation adaptation score is 70 points. The transmission efficiency score of path Y is 85 points and the heat dissipation adaptation score is 95 points. Based on the dynamic weight allocation result, the transmission efficiency score and heat dissipation adaptation score are weighted and fused to generate path evaluation parameters.

[0098] Where path X The path evaluation parameters for path Y are: Finally, path evaluation parameters are generated for subsequent path selection.

[0099] In this embodiment of the application, the above-mentioned collaborative processing process achieves a dynamic balance between data access characteristics and hardware heat dissipation status, so that path selection can ensure data transmission efficiency while fully considering hardware heat dissipation requirements, thereby effectively improving the system's stable operation capability under high temperature conditions.

[0100] To address the challenge of simultaneously quantifying transmission performance and heat dissipation adaptability in path evaluation and further improve the accuracy of path selection, in some embodiments, step 204 involves calculating the transmission efficiency score and heat dissipation adaptability score for each candidate path in the candidate path set using a path optimization algorithm, including:

[0101] Step 301: Obtain the first bandwidth utilization data, historical latency data, and temperature monitoring data for each candidate path in the candidate path set.

[0102] In step 301, the first bandwidth utilization data refers to the proportion of the bandwidth currently actually used by the candidate path to its maximum available bandwidth, the historical latency data refers to the average latency of data transmission by the candidate path over a period of time, and the temperature monitoring data refers to the real-time temperature readings of the hardware devices on each storage node through which the candidate path passes.

[0103] Step 302: Based on the first bandwidth utilization data and the historical latency data, calculate the data transmission efficiency benchmark value for each candidate path.

[0104] In step 302, the data transmission efficiency benchmark value is a quantitative indicator value that comprehensively reflects the basic transmission capability of the path. This value is determined by both bandwidth utilization and latency.

[0105] In this embodiment of the application, the first bandwidth utilization data and historical latency data are input into a preset standardized calculation formula. This formula takes into account the positive impact of high bandwidth and low latency on transmission efficiency, and then calculates a basic performance score for each path as a benchmark value for data transmission efficiency.

[0106] It should be noted that the embodiments of this application do not specifically limit the specific form of the standardized calculation formula, and can be set accordingly according to the actual situation.

[0107] Step 303: Based on the hotspot information, determine the number of hotspot areas and load intensity traversed by each candidate path.

[0108] In step 303, the number of hotspot areas refers to the number of areas marked as hotspot information that a candidate path passes through, and the load intensity refers to the current data access pressure level of these hotspot areas.

[0109] In this embodiment of the application, based on the hotspot information obtained from the previous analysis, the number of hotspot areas traversed by each candidate path is counted, and the current access busyness of each hotspot area is evaluated, thereby determining the load intensity parameters of each path.

[0110] Step 304: Based on the data transmission efficiency benchmark value, the number of hotspot areas, and the load intensity, a transmission efficiency score is generated using a weighted fusion algorithm.

[0111] In this embodiment, the data transmission efficiency benchmark value is used as the primary reference, while also considering the positive impact of the number of hotspot areas and the negative impact of load intensity. A weighted fusion algorithm is used to combine these three factors to generate the final transmission efficiency score. The implementation process of the weighted fusion algorithm will not be detailed in this embodiment; relevant technologies can be consulted.

[0112] Step 305: Calculate the heat dissipation demand index based on the difference between the temperature monitoring data and the preset temperature threshold.

[0113] In step 305, the heat dissipation demand index is a quantitative value that reflects the urgency of heat dissipation along the path. This value is calculated based on the difference between the current temperature and the safe temperature. The preset temperature threshold is the upper limit of the safe operating temperature of the hardware device set by the system.

[0114] The embodiments of this application do not specifically limit the value of the preset temperature threshold; it can be set according to the actual situation.

[0115] In this embodiment of the application, the collected temperature monitoring data is compared with a preset temperature threshold, the temperature difference of each node is calculated, and then the heat dissipation demand index of the entire path is calculated based on these differences. The higher the index, the more urgent the heat dissipation demand.

[0116] Step 306: Based on the heat dissipation control parameter set and the heat dissipation demand index, generate a heat dissipation adaptation score through a multi-objective optimization algorithm.

[0117] In this embodiment of the application, the quantized value of the heat dissipation level and the heat dissipation demand index provided by the heat dissipation control parameter set are input into the formula corresponding to the multi-objective optimization algorithm. The algorithm calculates the heat dissipation adaptation score of a path under the objective of balancing heat dissipation effect and energy consumption efficiency.

[0118] It should be noted that this embodiment does not limit the specific implementation process of the multi-objective optimization algorithm, and can be set accordingly according to the actual situation.

[0119] In this embodiment of the application, by calculating the transmission efficiency score and the heat dissipation adaptation score respectively, a precise quantitative evaluation of the performance and heat dissipation dimensions can be provided for each path. This allows subsequent path selection to be based on comprehensive and objective evaluation data, ensuring data transmission efficiency while also fully considering the system's heat dissipation requirements, ultimately effectively improving the scientific nature of path decision-making and the overall stability of system operation.

[0120] To address the issue of simultaneously optimizing heat dissipation efficiency and energy consumption during the heat dissipation adaptation score generation process, and to further improve the scientific rigor and practicality of the score results, in some embodiments, step 306: based on the heat dissipation control parameter set and the heat dissipation demand index, a heat dissipation adaptation score is generated using a multi-objective optimization algorithm, including:

[0121] Step 401: Obtain the path temperature distribution data of the heat dissipation demand index, and extract the working status parameters of the heat dissipation device from the heat dissipation control parameter set.

[0122] In step 401, the path temperature distribution data refers to the temperature status data formed by arranging the temperature values ​​of each node on the candidate path in sequence;

[0123] The operating status parameters of a heat dissipation device refer to the current operating parameter data of the heat dissipation equipment. These operating parameter data include: fan speed, pump power level, air duct opening command, heat sink activation strategy, etc.

[0124] Step 402: Establish a multi-objective optimization function with the goal of maximizing heat dissipation efficiency and minimizing energy consumption.

[0125] In step 402, the multi-objective optimization function is a mathematical function that considers multiple optimization objectives simultaneously. Maximizing heat dissipation efficiency is used to achieve the best heat dissipation effect, while minimizing energy consumption is used to reduce energy consumption to the minimum.

[0126] This embodiment does not limit the expression of the function, and can be set according to actual needs. The calculation formulas for maximizing heat dissipation efficiency and minimizing energy consumption can be referred to relevant technologies, and will not be repeated here.

[0127] Step 403: Input the operating state parameters of the heat dissipation device and the path temperature distribution data into the multi-objective optimization function, and perform iterative solution through the multi-objective optimization function to obtain the Pareto optimal solution set. Select the optimal solution that balances heat dissipation efficiency and energy consumption from the Pareto optimal solution set.

[0128] In step 403, the Pareto optimal solution set refers to the set of solutions in multi-objective optimization that can no longer improve any one objective without harming the others, and the optimal solution refers to the solution selected from the Pareto optimal solution set that best meets the actual needs.

[0129] Step 404: Calculate the heat dissipation adaptation score based on the heat dissipation efficiency weight and energy consumption weight in the optimal solution.

[0130] In step 404, the heat dissipation efficiency weight and energy consumption weight are quantified parameters obtained through Pareto optimal solution set analysis of a multi-objective optimization algorithm. The heat dissipation efficiency weight represents the relative importance of the heat dissipation performance optimization objective in the overall score, and the energy consumption weight represents the relative importance of the energy consumption control objective of the heat dissipation device. These two weight parameters together determine the balance between heat dissipation efficiency and energy consumption in the final heat dissipation adaptation score.

[0131] In this embodiment of the application, the final heat dissipation adaptation score is generated by weighted calculation formula based on the heat dissipation efficiency weight and energy consumption weight ratio given in the optimal solution and combined with the basic scoring parameters.

[0132] In this embodiment, a multi-objective optimization algorithm is used to balance the relationship between heat dissipation efficiency and energy consumption, so that the generated heat dissipation adaptation score can not only reflect the heat dissipation effect, but also take into account energy consumption. This provides a more comprehensive and reasonable evaluation basis for path selection, and ultimately helps to achieve the best balance between efficient heat dissipation and energy-saving operation of distributed storage systems.

[0133] To automatically identify system access patterns and performance bottlenecks from massive access data and further improve the accuracy of path optimization, in some embodiments, step 102 involves: performing big data pattern analysis on the access frequency data and the path latency data to extract access pattern data and hotspot information, and constructing a candidate path set based on the hotspot information, including:

[0134] Step 501: Perform sliding statistical analysis on the access frequency data using a fixed-duration time window to obtain the peak access frequency within each time window.

[0135] In step 501, the time window refers to a fixed-length time period, and the peak access frequency refers to the highest number of accesses that occur within that time period.

[0136] In this embodiment of the application, a fixed-duration time window is used to slide the access frequency data. After each window slide, the number of accesses within the window time is counted, and the highest number of accesses within each time window is recorded as the peak access frequency.

[0137] Step 502: Compare the peak access frequency with a preset access threshold, identify periodic time intervals based on the comparison results, and generate access pattern data based on the periodic time intervals.

[0138] In step 502, the preset access threshold is a pre-set critical value used to determine whether access is frequent; the periodic time interval refers to the time range in which high-frequency access occurs repeatedly.

[0139] This application does not impose a specific limitation on the value of the preset access threshold; it can be set according to the actual situation.

[0140] In this embodiment of the application, the peak access frequency of each time window is compared with a preset access threshold. When the peak value continues to exceed the threshold, the common time characteristics of these time windows are identified, thereby forming a periodic time interval, and based on this, access pattern data containing time regularity information is generated.

[0141] Step 503: Divide the path delay data into spatial regions, and calculate the average delay value and access frequency in each spatial region.

[0142] In step 503, the average latency value refers to the average latency of all paths within a certain area, and the area access frequency refers to the total number of accesses occurring within a certain area.

[0143] In this embodiment of the application, the storage nodes are divided into regions according to their physical deployment locations. The average path latency data in each region is averaged to obtain the average latency value of the region. At the same time, the total number of accesses in each region is counted to obtain the region access frequency.

[0144] Step 504: Mark the information corresponding to the regions where the average latency value is less than a preset latency threshold and the access frequency exceeds a preset access frequency threshold as hotspot information.

[0145] In step 504, the preset delay threshold is the maximum allowable delay threshold, and the preset access frequency threshold is the access volume threshold for determining whether a region is popular.

[0146] The embodiments of this application do not specifically limit the values ​​of the preset delay threshold and the preset access frequency threshold; they can be set according to the actual situation.

[0147] In this embodiment of the application, the average latency value of each region is compared with a preset latency threshold, and the region access frequency is compared with a preset access frequency threshold. The region information that simultaneously meets the conditions of low latency and high access is marked as hotspot information.

[0148] Step 505: Based on the regional location in the hotspot information, retrieve all available paths in the distributed storage system that pass through the regional location.

[0149] In step 505, the area location refers to the specific physical location information of the area marked as a hotspot, and the available path refers to the actual available data transmission path that exists in the system.

[0150] In this embodiment of the application, based on the regional location information recorded in the hotspot information, all complete paths passing through these hotspot areas are searched in the network topology of the distributed storage system, thereby forming a list of available paths.

[0151] Step 506: Sort all available paths according to node hop count and bandwidth capacity to form a candidate path set.

[0152] In step 506, the node hop count refers to the number of intermediate nodes traversed by the path, and the bandwidth capacity refers to the maximum data transmission capacity supported by the path.

[0153] In this embodiment of the application, all available paths are sorted in ascending order of node hop count, and paths with the same hop count are sorted in descending order of bandwidth capacity. Finally, a number of high-quality paths with the highest ranking are selected to form a candidate path set.

[0154] In this embodiment of the application, a systematic data analysis process is used to automatically identify the system's access patterns and performance hotspots, and based on this, a high-quality candidate path set is constructed, providing an accurate data foundation for subsequent path optimization, thereby improving the efficiency and quality of path selection.

[0155] To comprehensively consider real-time load conditions and data access characteristics in path selection, and further improve the comprehensiveness and accuracy of path optimization, in some embodiments, step 104: based on the path evaluation parameters, optimize paths that meet preset heat dissipation control requirements are selected from the candidate path set, and the optimized paths are comprehensively scored in terms of spatial proximity and temporal continuity using an evaluation function to generate a scoring result, including:

[0156] Step 601: Obtain the second bandwidth utilization data of the network interface and the queue length data of the storage controller.

[0157] In step 601, the network interface refers to the network communication component deployed in the metadata server, which means that it is a hardware interface unit responsible for data communication between the metadata server and other nodes in the distributed storage system.

[0158] The second bandwidth utilization data refers to the actual bandwidth utilization data of the network interface at the current moment, and the queue length data refers to the number of input / output requests waiting to be processed in the storage controller.

[0159] Step 602: Normalize the second bandwidth utilization data and the queue length data to generate load assessment coefficients.

[0160] In step 602, the load assessment coefficient is a comprehensive index value that reflects the current load status of the system.

[0161] In this embodiment of the application, the second bandwidth utilization data and queue length data are converted into standardized values ​​between 0 and 1, and then the average value of these two standardized values ​​is calculated to generate a load assessment coefficient that represents the overall load status of the system.

[0162] Step 603: Multiply the path evaluation parameters with the load evaluation coefficients to obtain the overall path score.

[0163] In step 603, the comprehensive path score is a combined score that integrates the path's own evaluation value and the system's real-time load status.

[0164] In this embodiment, the path evaluation parameters and the load evaluation coefficient are multiplied to enable the path score to be dynamically adjusted according to the current load condition, thereby generating the final comprehensive path score.

[0165] Step 604: Select the path with the best overall score and a temperature value less than a preset temperature threshold from the candidate path set as the optimized path.

[0166] In this embodiment of the application, the path with the highest comprehensive score is selected from the candidate path set, and at the same time, it is checked whether the temperature data of all nodes on the path are lower than the preset temperature threshold. The path that simultaneously meets the optimal score and temperature requirements is determined as the optimized path.

[0167] Step 605: Perform spatial locality analysis on the optimized path, and calculate the proximity of the data block storage location based on the first analysis result.

[0168] In step 605, the meaning of data block storage location refers to the physical storage address information of the actual data block corresponding to the metadata on each storage node in the distributed storage system. This address information is obtained through the metadata-data block mapping relationship table maintained by the metadata server; proximity is an indicator value that quantifies the degree of concentration of data storage locations.

[0169] In this embodiment of the application, the physical location of the data blocks stored on the optimized path is analyzed, the average physical distance between these data blocks is calculated, and then converted into a corresponding proximity score based on the distance.

[0170] Step 606: Perform temporal locality analysis on the optimized path, and calculate the regularity quantification value of the historical access time interval based on the second analysis result.

[0171] In step 606, the regularity quantification value is a numerical value that reflects the degree of regularity in access time.

[0172] In this embodiment of the application, the historical access time records of data blocks on the optimization path are analyzed, the time interval between adjacent access requests is calculated, and the volatility of these intervals, such as variance, standard deviation or coefficient of variation, is analyzed to quantify the regularity of access behavior in the time dimension, and finally generate regular quantitative values.

[0173] Step 607: Based on the proximity and the regularity quantification value, output the scoring result using the evaluation function.

[0174] In this embodiment, the proximity and regularity quantification values ​​are input into a preset evaluation function. The function performs a weighted calculation on the two parameters according to preset weights, thereby outputting the final path score result.

[0175] In this embodiment, by comprehensively considering real-time load conditions and data access characteristics, the path selection is not only based on static performance indicators, but also adapts to dynamic changes in the system, while ensuring that data access has good spatial and temporal locality, thereby improving the overall system performance and access efficiency.

[0176] To intelligently generate corresponding path adjustment strategies based on the scoring results and further improve the system's adaptability, in some embodiments, step 105: generating path adjustment instructions corresponding to the scoring results includes:

[0177] Step 701: Compare the scoring results with the preset highest scoring threshold and the lowest scoring threshold respectively.

[0178] In step 701, the highest scoring threshold is the excellent performance standard set by the system, and the lowest scoring threshold is the qualified performance standard set by the system.

[0179] This application does not impose specific limitations on the numerical values ​​of the highest and lowest scoring thresholds; these can be set according to actual circumstances.

[0180] Step 702: When the comparison result indicates that the score result is greater than the highest score threshold, generate a path maintenance instruction.

[0181] In step 702, the path maintenance instruction refers to the control instruction that keeps the current path configuration unchanged.

[0182] In this embodiment of the application, when the score result is higher than the highest score threshold, a path maintenance instruction is generated, instructing the system to continue using the current optimized path without adjustment.

[0183] Step 703: Alternatively, when the comparison result indicates that the score result is less than the minimum score threshold, generate a path reselection trigger instruction.

[0184] In step 703, the path reselection trigger instruction refers to the control instruction that initiates the path reselection process.

[0185] In this embodiment of the application, when the score result is lower than the minimum score threshold, a path reselection trigger instruction is generated, instructing the system to restart the path optimization process.

[0186] Step 704: Alternatively, when the comparison result indicates that the score result is greater than or equal to the lowest score threshold and less than or equal to the highest score threshold, calculate the adjustment amount of the path weight, and generate a path adjustment instruction containing weight update parameters based on the adjustment amount of the path weight.

[0187] In step 704, the adjustment amount of the path weight refers to the priority weight value that needs to be adjusted, and the weight update parameter refers to the parameter containing specific adjustment information.

[0188] In this embodiment of the application, when the scoring result is between two thresholds, a specific weight adjustment value is calculated based on the scoring value, and a path adjustment instruction containing adjustment information is generated.

[0189] In this embodiment, the intelligent threshold comparison and instruction generation mechanism enables the system to automatically adopt corresponding adjustment strategies based on the path performance score, which not only ensures the timeliness of path optimization but also avoids unnecessary frequent adjustments, thus realizing intelligent and refined system performance maintenance.

[0190] Figure 3 This application provides a schematic diagram of the structure of a data locality optimization device in a distributed storage device, and the specific implementation section describes the following:

[0191] The acquisition module 31 is used to acquire the data access history of the distributed storage system and the temperature monitoring data of the metadata server. The data access history includes access frequency data and path delay data.

[0192] The analysis module 32 is used to perform big data pattern analysis on the access frequency data and the path delay data to extract access pattern data and hotspot information, and to construct a candidate path set based on the hotspot information.

[0193] The generation module 33 is used to generate a set of heat dissipation control parameters corresponding to the temperature monitoring data, and to use a path optimization algorithm to perform collaborative processing on the access mode data and the set of heat dissipation control parameters to generate path evaluation parameters.

[0194] The filtering module 34 is used to filter out optimized paths that meet preset heat dissipation control requirements from the candidate path set according to the path evaluation parameters, and use the evaluation function to comprehensively score the optimized paths in terms of spatial proximity and temporal continuity to generate a scoring result.

[0195] The adjustment module 35 is used to generate a path adjustment instruction corresponding to the scoring result, and adjust the optimized path according to the path adjustment instruction to achieve local optimization of metadata storage.

[0196] The data locality optimization device in the distributed storage device of this application embodiment is used to implement the aforementioned data locality optimization method in the distributed storage device. Therefore, the specific implementation of the data locality optimization device in the distributed storage device can be found in the embodiment section of the data locality optimization method in the distributed storage device above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0197] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the data locality optimization method in any of the above-described distributed storage devices.

[0198] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data locality optimization method in any of the above-described distributed storage devices.

[0199] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0200] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the data locality optimization method in a distributed storage device.

[0201] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0202] The foregoing has provided a detailed description of a data locality optimization method and apparatus in a distributed storage device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for optimizing data locality in a distributed storage system, characterized in that, include: Collect historical data access data of the distributed storage system and temperature monitoring data of the metadata server. The historical data access data includes access frequency data and path latency data. Big data pattern analysis is performed on the access frequency data and the path delay data to extract access pattern data and hotspot information, and a candidate path set is constructed based on the hotspot information; A set of heat dissipation control parameters corresponding to the temperature monitoring data is generated. A path optimization algorithm is used to collaboratively process the access mode data and the set of heat dissipation control parameters to generate path evaluation parameters. Based on the path evaluation parameters, an optimized path that meets the preset heat dissipation control requirements is selected from the candidate path set, and an evaluation function is used to comprehensively score the optimized path in terms of spatial proximity and temporal continuity to generate a score result. Generate a path adjustment instruction corresponding to the scoring result, and adjust the optimized path according to the path adjustment instruction to achieve local optimization of metadata storage.

2. The method according to claim 1, characterized in that, The process of generating a heat dissipation control parameter set corresponding to the temperature monitoring data involves using a path optimization algorithm to collaboratively process the access pattern data and the heat dissipation control parameter set to generate path evaluation parameters, including: A sliding window scan is performed on the temperature monitoring data to calculate the temperature change rate of each of the processing unit and the storage controller; Based on the temperature change rate and the current temperature value, a set of heat dissipation control parameters is generated; Dynamic weight allocation is performed between the access mode data and the heat dissipation control parameter set; The transmission efficiency score and heat dissipation adaptation score of each candidate path in the candidate path set are calculated using a path optimization algorithm. Based on the dynamic weight allocation results, the transmission efficiency score and the heat dissipation adaptation score are weighted and fused to generate path evaluation parameters.

3. The method according to claim 2, characterized in that, The process of calculating the transmission efficiency score and heat dissipation adaptation score for each candidate path in the candidate path set using a path optimization algorithm includes: Obtain the first bandwidth utilization data, historical latency data, and temperature monitoring data for each candidate path in the candidate path set; Based on the first bandwidth utilization data and the historical latency data, calculate the data transmission efficiency benchmark value for each candidate path; Based on the hotspot information, determine the number of hotspot areas and load intensity traversed by each candidate path; Based on the data transmission efficiency benchmark, the number of hotspot areas, and the load intensity, a transmission efficiency score is generated using a weighted fusion algorithm. The heat dissipation demand index is calculated based on the difference between the temperature monitoring data and the preset temperature threshold. Based on the set of heat dissipation control parameters and the heat dissipation demand index, a heat dissipation adaptation score is generated through a multi-objective optimization algorithm.

4. The method according to claim 3, characterized in that, The step of generating a heat dissipation adaptation score based on the heat dissipation control parameter set and the heat dissipation demand index using a multi-objective optimization algorithm includes: Obtain the path temperature distribution data of the heat dissipation demand index, and extract the working status parameters of the heat dissipation device from the heat dissipation control parameter set; Establish a multi-objective optimization function with the goal of maximizing heat dissipation efficiency and minimizing energy consumption; The operating state parameters of the heat dissipation device and the path temperature distribution data are input into a multi-objective optimization function. The multi-objective optimization function is used to iteratively solve the problem and obtain a Pareto optimal solution set. The optimal solution that balances heat dissipation efficiency and energy consumption is selected from the Pareto optimal solution set. The heat dissipation adaptation score is calculated based on the heat dissipation efficiency weight and energy consumption weight in the optimal solution.

5. The method according to claim 1, characterized in that, The step involves performing big data pattern analysis on the access frequency data and the path delay data to extract access pattern data and hotspot information, and constructing a candidate path set based on the hotspot information, including: A sliding statistical analysis was performed on the access frequency data using a fixed-duration time window to obtain the peak access frequency within each time window. The peak access frequency is compared with a preset access threshold. Based on the comparison result, a periodic time interval is identified, and access pattern data is generated based on the periodic time interval. The path delay data is divided into spatial regions, resulting in multiple spatial regions. The average delay value and access frequency within each spatial region are then calculated. The information corresponding to the regions where the average latency value is less than a preset latency threshold and the access frequency exceeds a preset access frequency threshold is marked as hotspot information; Based on the regional location in the hotspot information, retrieve all available paths in the distributed storage system that pass through the regional location; All available paths are sorted according to node hop count and bandwidth capacity to form a candidate path set.

6. The method according to claim 1, characterized in that, The process involves selecting optimized paths that meet preset heat dissipation control requirements from the candidate path set based on the path evaluation parameters, and using an evaluation function to comprehensively score the optimized paths in terms of spatial proximity and temporal continuity to generate a scoring result, including: Obtain the second bandwidth utilization data of the network interface and the queue length data of the storage controller; The second bandwidth utilization data and the queue length data are normalized to generate load assessment coefficients; The path evaluation parameters are multiplied by the load evaluation coefficients to obtain the overall path score. The path with the best overall score and a temperature value less than a preset temperature threshold is selected from the candidate path set as the optimized path. Spatial locality analysis is performed on the optimized path, and the proximity of the data block storage location is calculated based on the first analysis result; Furthermore, a temporal locality analysis is performed on the optimized path, and based on the second analysis result, a quantitative value of the regularity of historical access time intervals is calculated. Based on the proximity and the regularity quantification value, the scoring result is output using the evaluation function.

7. The method according to claim 1, characterized in that, The path adjustment instruction for generating the scoring result includes: The scoring results are compared with the preset highest and lowest scoring thresholds, respectively. When the comparison result indicates that the score is greater than the highest score threshold, a path maintenance instruction is generated; Alternatively, when the comparison result indicates that the score result is less than the minimum score threshold, a path reselection trigger instruction is generated; Alternatively, when the comparison result indicates that the score result is greater than or equal to the lowest score threshold and less than or equal to the highest score threshold, the adjustment amount of the path weight is calculated, and based on the adjustment amount of the path weight, a path adjustment instruction containing weight update parameters is generated.

8. A data locality optimization device in a distributed storage device, characterized in that, include: The acquisition module is used to acquire the data access history of the distributed storage system and the temperature monitoring data of the metadata server. The data access history includes access frequency data and path delay data. The analysis module is used to perform big data pattern analysis on the access frequency data and the path delay data to extract access pattern data and hotspot information, and to construct a candidate path set based on the hotspot information. The generation module is used to generate a set of heat dissipation control parameters corresponding to the temperature monitoring data, and to use a path optimization algorithm to collaboratively process the access mode data and the set of heat dissipation control parameters to generate path evaluation parameters. The filtering module is used to filter out optimized paths that meet preset heat dissipation control requirements from the candidate path set according to the path evaluation parameters, and use the evaluation function to comprehensively score the optimized paths in terms of spatial proximity and temporal continuity to generate a score result. The adjustment module is used to generate path adjustment instructions corresponding to the scoring results, and adjust the optimized path according to the path adjustment instructions to achieve local optimization of metadata storage.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data locality optimization method in a distributed storage device as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the data locality optimization method in a distributed storage device as described in any one of claims 1 to 7.

Images