Distributed storage management system for real estate data

Abstract

This invention discloses a distributed storage management system for real estate data, relating to the field of real estate data storage management technology. It includes a high-concurrency identification and load status acquisition module, a node load analysis and classification module, a hierarchical load adjustment mechanism construction module, a dynamic feedback and optimization evaluation module, and a system performance optimization report generation module. The node load analysis and classification module analyzes the load status information of each node acquired in real time, determines whether there is an imbalance in node load, and classifies each node into high-load nodes, normal-load nodes, and low-load nodes based on the judgment results. The hierarchical load adjustment mechanism construction module constructs a load adjustment mechanism based on the classification results of each node, for different adjustments to high-load nodes, normal-load nodes, and low-load nodes. This invention solves the problem of load imbalance under high-concurrency access, realizing dynamic optimization allocation of system resources and improving user experience.

Description

Technical Field

[0001] This invention relates to the field of real estate data storage and management technology, specifically to a distributed real estate data storage and management system. Background Technology

[0002] Real estate data refers to various information related to real estate, mainly including basic information, transaction records, market prices, property rights changes, and spatial planning of entities such as land, buildings, and properties. This data not only covers the physical characteristics of real estate, such as geographical location, building area, and use, but also includes detailed information on legal, economic, and social aspects related to real estate, such as property owner information, tax status, market demand, and policies and regulations. With the rapid development of the real estate market, the amount of data is increasing rapidly. A single storage system can hardly effectively support the storage and management of large-scale data. Furthermore, real estate data is characterized by its distributed, heterogeneous, and high-concurrency nature, making distributed storage management of real estate data particularly important. Distributed storage can distribute data across multiple nodes, providing not only higher storage capacity and processing power but also improving data access speed and system fault tolerance. In addition, distributed storage management, through replication mechanisms and data redundancy, ensures high availability and reliability of data, ensuring that data is not lost even in the event of node failure. Distributed technology also enables load balancing and high-concurrency access, thereby improving system performance and meeting the needs of large-scale users and massive data queries. Ultimately, distributed storage provides a more flexible and scalable storage solution for real estate data, supporting efficient processing and analysis of complex data and providing a solid foundation for the digital transformation of the real estate industry.

[0003] Existing distributed storage management technologies for real estate data achieve efficient distributed storage and management by distributing data across multiple storage nodes and combining data partitioning and replication mechanisms. First, data is partitioned according to specific rules (such as region, property type, or time period), distributing real estate data across different storage nodes. This ensures that each node stores a moderate amount of data, avoiding overload issues for any single node. Second, to guarantee high availability and fault tolerance, the system typically generates multiple replicas for each piece of data, distributed across different storage nodes. When a node fails, replicas on other nodes automatically take over, ensuring no data loss or service interruption. Furthermore, distributed storage management systems are usually equipped with a robust metadata management module responsible for recording key information such as data storage location and status, enabling efficient data querying and location. By employing mechanisms such as consistent hashing, load balancing, and efficient indexing, distributed storage systems ensure fast access and query response for large-scale data. Simultaneously, the system further improves data storage performance and query efficiency by using distributed databases and caching technologies. By combining these technologies, existing distributed storage management for real estate data can provide flexible, scalable, highly reliable, and high-performance storage solutions, supporting the real estate industry in the efficient processing and analysis of massive, multi-dimensional data.

[0004] The existing technology has the following shortcomings:

[0005] During peak hours on a real estate platform (such as holidays or promotional events for specific properties), a large number of users simultaneously access the platform's video viewing function, requiring the platform's distributed storage system to process and transmit massive amounts of video data. When multiple users simultaneously request video files for the same property, some nodes in the system face enormous bandwidth and storage transmission pressure. Due to load balancing issues in the platform's distributed storage architecture when handling these high-concurrency requests, the resources of some nodes (such as bandwidth, cache, and storage capacity) are quickly exhausted, while the resources of other nodes are not fully utilized. As a result, some users experience loading delays or video stuttering when watching videos. The root cause of this phenomenon is that the system has failed to effectively balance the load across nodes, causing some nodes to be overloaded while others are idle. Existing technology has not been effectively optimized for high-concurrency scenarios, resulting in inconsistent data transmission speeds between nodes, slower video loading speeds, and severely impacting the user's viewing experience. In this situation, users may abandon watching due to a poor experience, or even choose to browse other platforms, leading to the loss of potential homebuyers, reduced platform conversion rates and market competitiveness, and ultimately affecting the effectiveness of property promotions and the platform's overall revenue.

[0006] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide a distributed storage and management system for real estate data to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a distributed storage management system for real estate data, including a high-concurrency identification and load status acquisition module, a node load analysis and classification module, a hierarchical load adjustment mechanism construction module, a dynamic feedback and optimization evaluation module, and a system performance optimization report generation module;

[0009] The high concurrency identification and load status acquisition module monitors the status of each node in the distributed storage system in real time to identify whether there are high-concurrency users accessing the platform's video viewing function. When high-concurrency users access the platform's video viewing function, it acquires the load status information of each node in real time.

[0010] The node load analysis and classification module analyzes the load status information of each node acquired in real time, determines whether there is an imbalance in node load, and classifies each node into high-load nodes, normal-load nodes, and low-load nodes based on the judgment results.

[0011] The tiered load adjustment mechanism construction module constructs a load adjustment mechanism based on the division results of each node, which is used to make different adjustments to high-load nodes, normal-load nodes and low-load nodes respectively.

[0012] The dynamic feedback and optimization evaluation module acquires feedback information from each node in real time during the load scheduling mechanism's adjustment of each node, analyzes the acquired information, evaluates whether the load adjustment mechanism's adjustment effect on each node can meet expectations, and optimizes the load adjustment mechanism based on the evaluation results.

[0013] The system performance optimization report generation module comprehensively analyzes the load status information, feedback information, and adjustment records of each node to generate a system performance optimization report, so as to provide data support and decision-making basis for subsequent load management strategies and resource configuration optimization.

[0014] Preferably, in the node load analysis and classification module, after acquiring the load status information of each node in real time, the load status information of each node is preprocessed, and the resource usage information and response performance information in the preprocessed load status information of each node are extracted and analyzed to generate the resource consumption coefficient and response latency index of each node.

[0015] Preferably, the logic for obtaining the resource consumption coefficient and response latency index of each node is as follows:

[0016] Extract resource usage information from the preprocessed load status information of each node. Specifically, this includes the percentage of total bandwidth used by each node at different times within a given period when processing video viewing requests, the percentage of total CPU time used within the overall time, and the rate at which data is read from storage. These are then labeled as follows: , and , Indicates the first Each node processes video viewing requests within a certain period of time. The percentage of total bandwidth used at any given moment. Indicates the first Each node processes video viewing requests within a certain period of time. The percentage of total CPU usage time at any given moment out of the total total time. Indicates the first Each node processes video viewing requests within a certain period of time. The rate at which data is read from storage at any given time. , , and All are positive integers;

[0017] The resource consumption coefficient of each node is calculated using the following formula:

[0018] ;

[0019] In the formula, For the first Resource consumption coefficient of each node;

[0020] Extract response performance information from the preprocessed load status information of each node. Specifically, this includes the average time each node takes to process a single request at different times within a given period, the maximum length of the request queue, and the failure rate of each node during request processing within that period. These are then labeled as follows: , and , Indicates the first A node within a certain period of time The average time required to process a single request at any given moment. Indicates the first A node within a certain period of time The maximum length of the request queue processed at any given time. Indicates the number of times within that time period The rate at which each node fails to process a request;

[0021] The response latency exponent for each node is calculated using the following formula:

[0022] ;

[0023] In the formula, For the first The response latency index of each node.

[0024] Preferably, the resource consumption coefficients of each generated node are... and response latency index Construct a load balancing evaluation model and generate the load coefficient of each node through weighted summation. Adjust the load factor of all nodes A comprehensive analysis is performed to generate the standard deviation of the load factor. According to the formula: and the standard deviation of the generated load factor Compared with the pre-set standard deviation threshold of the load factor A comparison is performed, and the results are used to determine whether there is any unbalanced node load. The specific comparison analysis is as follows:

[0025] like There is no issue of uneven node load.

[0026] like There is a situation where the node load is unbalanced.

[0027] Preferably, if the determination result indicates that there is an imbalance in node load, a pre-set load factor threshold range is determined. The threshold range Load factor of each node A comparison is performed, and based on the comparison results, each node is divided into high-load nodes, normal-load nodes, and low-load nodes. The specific comparison and division are as follows:

[0028] like This node is classified as a low-load node.

[0029] like This node is classified as a normal load node.

[0030] like This node is then classified as a high-load node.

[0031] Preferably, in the tiered load adjustment mechanism construction module, based on the division results of each node, a load adjustment mechanism is constructed to perform different adjustments on high-load nodes, normal-load nodes, and low-load nodes, specifically including:

[0032] Based on the division of high-load nodes, normal-load nodes, and low-load nodes, different load adjustment parameters are set to form a hierarchical load adjustment mechanism. This load adjustment mechanism automatically determines the priority and adjustment range of load adjustment based on the load coefficient of each node and the current request processing status, through pre-set rules.

[0033] Different load adjustment operations are performed on high-load nodes, normal-load nodes, and low-load nodes respectively. Specifically: in high-load nodes, the high load adjustment parameters in the load adjustment mechanism are used to prioritize request redirection and reduce resource consumption; in normal-load nodes, the standard load adjustment parameters in the load adjustment mechanism are maintained to maintain the current load distribution and balance resource usage; in low-load nodes, the low load adjustment parameters in the load adjustment mechanism are used to reduce load adjustment operations and maintain the existing load level.

[0034] Preferably, in the dynamic feedback and optimization evaluation module, after acquiring the feedback information of each node in real time, it is preprocessed, and the load adaptation information and request recovery information in the preprocessed feedback information of each node are extracted and analyzed to generate the load distribution adaptation coefficient and request processing recovery index of each node respectively.

[0035] Preferably, the logic for obtaining the load distribution fitness coefficient and request processing recovery index of each node is as follows:

[0036] Extract load adaptation information from the preprocessed feedback information of each node. Specifically, this includes the proportion of requests allocated to each node to the total number of system requests at different times during a period of time as the load scheduling mechanism adjusts each node, the change in the number of requests allocated to each node, and the average response latency for each node to process requests transferred from other nodes. Then, apply these parameters according to the time series using functions... , and To express, For a point in time, This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The proportion of requests allocated to each node relative to the total number of requests in the system. This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The change in the number of requests allocated to each node This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The average response latency for each node to process requests transferred from other nodes, defined as the time period. , , It is a positive integer;

[0037] The load distribution fitness coefficient for each node is calculated using the following formula:

[0038] ;

[0039] In the formula, For the first The load distribution adaptation factor for each node;

[0040] Extract request recovery information from the preprocessed feedback information of each node. Specifically, this includes the increase in request processing success rate, request latency recovery rate, and reduction in request processing failure rate of each node at different times during a period of time as the load scheduling mechanism adjusts each node. Then, apply these parameters according to the time series using a function. , and To express, For a point in time, This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The improvement in the success rate of request processing per node. This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The request latency recovery rate of each node, This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The reduction in the failure rate of request processing per node, defined over a time period of [time period missing]. ;

[0041] The request processing recovery index for each node is calculated using the following formula:

[0042] ;

[0043] In the formula, For the first The request processing recovery index of each node.

[0044] Preferably, the load distribution adaptation coefficients for each node are generated. and request processing recovery index Then, an adjustment degree assessment model is constructed, and the adjustment coefficients for each node are generated by weighted summation. And the adjustment coefficients of each generated node Adjustment coefficient thresholds for each pre-set node A comparison was conducted, and the load adjustment mechanism's effectiveness at each node was evaluated based on the comparison results. The load adjustment mechanism was then optimized based on the evaluation results. The specific comparison analysis is as follows:

[0045] like The load balancing mechanism did not achieve the expected results at this node, and needs to be optimized. Specifically, this includes: adjusting node load allocation parameters to balance resource distribution; optimizing request redirection strategies to reduce node load; improving resource scheduling strategies to increase node resource allocation; increasing the frequency of dynamic monitoring and feedback to capture load changes in a timely manner; and resetting the priority and magnitude of load balancing to adapt to high load conditions.

[0046] like The load adjustment mechanism at this node achieves the expected adjustment effect, and there is no need to optimize the load adjustment mechanism.

[0047] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0048] 1. This invention collects and analyzes the load status information and request processing status of each node in real time, enabling timely identification of load imbalances and automatically constructing and adjusting load distribution mechanisms based on the load status of different nodes. This real-time monitoring and feedback mechanism effectively avoids overload or idleness of single nodes, ensuring the rational utilization of system resources, improving the stability and response speed of the platform during peak access periods, and guaranteeing a smooth viewing experience for users.

[0049] 2. This invention utilizes an evaluation model based on load distribution adaptability coefficients and request processing recovery indices. This model allows the system to quantify the load adaptability of each node and automatically determine and classify node states. Based on these indicators, this invention can apply different adjustment strategies, such as request redirection and resource allocation, to nodes with high, normal, and low loads, thereby achieving tiered optimization. Compared to traditional static load management methods, this invention offers stronger dynamic adaptability, enabling timely responses to fluctuations in user request volume, significantly reducing video loading latency and stuttering, and improving the platform's user experience and service quality.

[0050] 3. The system performance optimization report generation module of this invention provides data support for further optimization of load management. By comprehensively analyzing the load status, feedback information, and adjustment records of each node, it generates a detailed performance report. This not only helps identify load bottlenecks and nodes with insufficient resource allocation, but also provides reliable data for subsequent resource scheduling and strategy adjustments. Overall, this invention not only improves the system's load handling capability in high-concurrency scenarios, but also achieves adaptive optimization and performance improvement, providing the platform with a technological advantage in a highly competitive market. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0052] Figure 1 This is a schematic diagram of the modules of the distributed storage management system for real estate data of the present invention. Detailed Implementation

[0053] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0054] This invention provides, for example Figure 1 The real estate data distributed storage management system shown includes a high concurrency identification and load status acquisition module, a node load analysis and classification module, a hierarchical load adjustment mechanism construction module, a dynamic feedback and optimization evaluation module, and a system performance optimization report generation module.

[0055] The high concurrency identification and load status acquisition module monitors the status of each node in the distributed storage system in real time to identify whether there are high-concurrency users accessing the platform's video viewing function. When high-concurrency users access the platform's video viewing function, it acquires the load status information of each node in real time.

[0056] To identify high-concurrency user access to the platform's video viewing function, this can be achieved by monitoring the number and rate of system requests. Specifically, the system records key data such as the frequency, time interval, and peak request values ​​of video requests in real time, using preset thresholds to determine high concurrency. When the request frequency exceeds the preset threshold, the system can determine that it has entered a high-concurrency state. Furthermore, through access log analysis and user behavior analysis, specific access paths for the video viewing function and the video request volume for popular properties can be further identified, allowing for accurate identification of whether the platform has entered a high-concurrency state and triggering subsequent load status collection and optimization adjustments.

[0057] When high-concurrency users access the platform's video viewing function, to obtain real-time load status information for each node, the system can set up monitoring scripts or use load monitoring software to periodically poll or event-drivenly collect load information from each node. Specifically, the system collects parameters such as bandwidth utilization, CPU load, storage capacity, and request frequency from each node and transmits this information to the central load monitoring module. This monitoring module updates the collected data in real time, ensuring that the load status of each node can be dynamically monitored under high concurrency. In this way, the system can globally track the node load status, ensuring that timely adjustment measures can be taken when load imbalances occur.

[0058] Real-time monitoring of high-concurrency user access and the load status of each node is a crucial step in resolving the issue of uneven node load in distributed storage systems. When high-concurrency users access the video viewing function, some nodes may experience resource exhaustion and slow response due to exceeding their load thresholds, resulting in video loading delays or stuttering on the user's end. By identifying high-concurrency situations and acquiring node load status information in real time, the system can proactively assess and classify node load, providing data support for subsequent load adjustment mechanisms. This solution not only effectively prevents the resource exhaustion of a single node but also improves the system's stability and responsiveness under high-concurrency conditions through dynamic adjustment of node load balancing, thereby enhancing the user's viewing experience and meeting the core requirements of the technical solution and problem-solving.

[0059] The node load analysis and classification module analyzes the load status information of each node acquired in real time, determines whether there is an imbalance in node load, and classifies each node into high-load nodes, normal-load nodes, and low-load nodes based on the judgment results.

[0060] In this embodiment, in the node load analysis and classification module, after the load status information of each node is acquired in real time, the load status information of each node is preprocessed, and the resource usage information and response performance information in the preprocessed load status information of each node are extracted and analyzed to generate the resource consumption coefficient and response latency index of each node.

[0061] In the node load analysis and classification module, preprocessing the real-time acquired load status information of each node is crucial to ensure data accuracy, consistency, and comparability. This allows for a more precise reflection of the actual load on nodes when calculating resource consumption coefficients and response latency indices. First, preprocessing includes denoising, using statistical methods or filtering algorithms to remove outliers or noisy data, preventing sudden anomalies from misleading the results. Second, data standardization transforms the load status information of different nodes into a unified metric, such as normalizing bandwidth, CPU, and storage data according to a specific range to ensure comparability on the same scale. Finally, time window smoothing is implemented, smoothing the collected real-time data using moving averages or exponentially weighted averages to reduce the impact of data fluctuations on subsequent analysis, making the extracted information more representative. These preprocessing steps can be monitored and processed in real-time by software, automatically optimizing and standardizing the data flow to provide a reliable data foundation for subsequent node classification and load analysis.

[0062] In this embodiment, the logic for obtaining the resource consumption coefficient and response latency index of each node is as follows:

[0063] Extract resource usage information from the preprocessed load status information of each node. Specifically, this includes the percentage of total bandwidth used by each node at different times within a given period when processing video viewing requests, the percentage of total CPU time used within the overall time, and the rate at which data is read from storage. These are then labeled as follows: , and , Indicates the first Each node processes video viewing requests within a certain period of time. The percentage of total bandwidth used at any given moment. Indicates the first Each node processes video viewing requests within a certain period of time. The percentage of total CPU usage time at any given moment out of the total total time. Indicates the first Each node processes video viewing requests within a certain period of time. The rate at which data is read from storage at any given time. , , and All are positive integers;

[0064] Resource usage information for each node can be extracted through real-time monitoring and data collection. The specific steps are as follows: First, the system sets the collection frequency for bandwidth, CPU, and storage usage on each node and collects this information periodically within a specified time window. Bandwidth usage can be obtained in real time using network monitoring tools such as SNMP; CPU usage can be obtained using system resource monitoring tools (such as Linux's `top` or Windows' performance counters); and storage read rates can be obtained using file system monitoring (such as I / O statistics tools). All collected data is periodically transmitted to the central analysis module, where it is automatically categorized and preprocessed to extract the bandwidth, CPU, and storage usage data for each node, enabling subsequent calculations of the node's resource consumption coefficient.

[0065] In the specific data collection process, various quantitative data are acquired in the following ways: Total bandwidth percentage is calculated by dividing the actual bandwidth usage within each time slice of each node's processing of video viewing requests by the node's total bandwidth. Higher bandwidth usage indicates a heavier load. Total CPU usage time percentage is calculated by collecting the CPU usage time within each time slice and dividing it by the total time, yielding the CPU utilization under high concurrency load. This data accurately reflects the computational resource pressure on the node when processing requests. Storage read rate is measured by monitoring the I / O access rate of the node to the storage device during request processing (expressed in units such as MB / s). A higher storage read rate indicates that the node reads data frequently and is more dependent on storage resources. All of this data can be collected in real-time using operating system and network monitoring tools, and unified management is achieved through periodic transmission and integration, providing a reliable quantitative data foundation for subsequent load assessment and adjustments.

[0066] The resource consumption coefficient of each node is calculated using the following formula:

[0067] ;

[0068] In the formula, For the first Resource consumption coefficient of each node;

[0069] This formula calculates the resource consumption coefficient of each node in detail through component calculations and weighted averaging, ensuring accurate measurement of node load. First, the three metrics—bandwidth utilization ratio, CPU usage time percentage, and storage read rate—each reflect the node's resource consumption in terms of network, computing, and storage. The formula uses fractional forms for each term. , , This is to emphasize the nonlinear impact of nodes on the load when they are near saturation (i.e., values ​​close to 1). The closer these index values ​​are to 1, the more nonlinearly the fractional values ​​amplify, highlighting the significant impact of nodes nearing resource saturation on the overall system load. The squaring operation unifies positive and negative deviations and increases the impact of high values ​​on total consumption; the square root operation further ensures the combined effect of different resource items. Finally, through the time window... The summation of the mean over all time points within the time frame smooths out instantaneous fluctuations in the calculation results, generating a stable resource consumption coefficient. This helps the system to fully understand the resource load of nodes, enabling more accurate load balancing and resource management.

[0070] No. Resource consumption coefficient of each node The resource consumption coefficient directly reflects the resource usage and load pressure of a node, and is closely related to judging node load imbalance. When a node has a high resource consumption coefficient, it indicates that the node is close to saturation in the use of network, CPU, and storage resources, and is under heavy load; while a node with a low resource consumption coefficient means that the resource utilization is low and the load is light. By comparing the resource consumption coefficients of various nodes, nodes with high load, normal load, and low load can be identified. If some nodes in the system have significantly high resource consumption coefficients while other nodes have low coefficients, this difference indicates an imbalance in the load among nodes, thus requiring appropriate adjustments to the high-load nodes to achieve a balanced allocation of resources. Therefore, the magnitude and difference of the resource consumption coefficient are important criteria for judging node load imbalance.

[0071] Extract response performance information from the preprocessed load status information of each node. Specifically, this includes the average time each node takes to process a single request at different times within a given period, the maximum length of the request queue, and the failure rate of each node during request processing within that period. These are then labeled as follows: , and , Indicates the first A node within a certain period of time The average time required to process a single request at any given moment. Indicates the first A node within a certain period of time The maximum length of the request queue processed at any given time. Indicates the number of times within that time period The rate at which each node fails to process a request;

[0072] Extracting response performance information from each node can be achieved through real-time monitoring software and log analysis tools. First, the system uses request processing monitoring tools (such as APM – Application Performance Management) to record the performance data of each node in real time while processing requests. To extract the average request processing time of each node at different times, the system analyzes the start and end times of each request, calculates the processing duration of each request, and then statistically analyzes these durations within a preset time window to extract the average processing time. The maximum length of the request queue is obtained by analyzing the request queue data of nodes during high concurrency periods. The system records the number of backlogged requests at each time point and selects the highest queue length within the time window as the peak. The request failure rate is extracted through statistical analysis of the number of successful and failed requests. The system can directly count the number of failed requests from the logs and calculate the ratio to the total number of requests. All this data, collected by automated monitoring tools, is transmitted to the central data processing module for further analysis and use within a preset time period.

[0073] The average time required to process a single request is obtained by recording the duration of each request processed by the node and then calculating the average processing time of all requests over a period of time. For example, the start and end times of each request processed by the node can be obtained from log records. Averaging these durations over a time window yields the average response time of the node to requests within that time period. The maximum length of the request queue is determined by periodically sampling the length of the request queue during node operation and recording the queue length changes within each time slice to find the highest peak within the time period. This peak reflects the request backlog of the node under high concurrency, thus indicating the system's performance when its processing capacity is close to its limit. The request processing failure rate is calculated by statistically analyzing the ratio of the number of failed requests processed by the node to the total number of requests within a certain period of time. The system can analyze the request status codes (such as HTTP 500 errors) in the processing logs to determine which requests failed and calculate the proportion of these failed requests to the total number of requests, thereby quantifying the node's processing stability under high load. Through these automated monitoring and analysis tools, the system can accurately and promptly obtain the response performance information of each node, providing data support for subsequent load analysis and adjustments.

[0074] The response latency exponent for each node is calculated using the following formula:

[0075] ;

[0076] In the formula, For the first The response latency index of each node.

[0077] This formula accurately reflects the node's response latency through nonlinear and combinatorial operations, ensuring a comprehensive measurement of load pressure across different dimensions. Specifically, the first term... The average processing time is multiplied by the request queue length to amplify the latency pressure on the node under high concurrency. Simultaneously, the effect of a request failure rate approaching 1 is amplified by adjusting for the inverse square root of the failure rate, reflecting the increase in non-linear response latency when the node is near its maximum load. (Second term) Using a logarithmic function, processing time and failure rate are combined, with additional amplification applied to nodes with high failure rates. This highlights the impact of high failure rates on latency, resulting in a significant increase in this term when the failure rate is high. (Third term) The peak request queue length is fractionally cubed to amplify the latency pressure caused by peak request backlog, making the response lag more significant during high concurrency periods. Finally, by averaging across all points within the time window, short-term fluctuations are smoothed out, resulting in a more favorable response latency index. It is more stable and representative. This calculation method accurately characterizes the response latency of each node under high load conditions through complex formulas, providing solid data support for judging load imbalance.

[0078] No. Response latency index of each node The response latency index directly reflects the node's response efficiency and latency under high concurrency and high load conditions, and is an important indicator for judging node load imbalance. When a node's response latency index is significantly higher than other nodes, it indicates that the node may experience delays, request backlogs, or a high failure rate when processing requests, indicating that its load pressure is high; conversely, nodes with lower response latency indices indicate good response performance and low resource consumption. By comparing the response latency indices of each node, it is possible to identify whether there is an imbalance in load distribution. If some nodes in the system... A significantly higher load than other nodes, while the load on other nodes is lower, indicates an unbalanced load. Resource allocation needs to be adjusted to optimize the overall system performance and ensure that each node operates efficiently under a reasonable load level.

[0079] In this embodiment, the resource consumption coefficients of each node are generated. and response latency index Construct a load balancing evaluation model and generate the load coefficient of each node through weighted summation. Adjust the load factor of all nodes A comprehensive analysis is performed to generate the standard deviation of the load factor. According to the formula: and the standard deviation of the generated load factor Compared with the pre-set standard deviation threshold of the load factor A comparison is performed, and the results are used to determine whether there is any unbalanced node load. The specific comparison analysis is as follows:

[0080] like There is no issue of uneven node load.

[0081] like There is a situation where the node load is unbalanced.

[0082] These two judgments are based on the standard deviation of the load factor. With preset threshold The comparison results determine whether the node load is balanced. When When the load distribution is normal, it indicates that the load differences among the nodes in the system are small, the load balance is good, and there is no load imbalance. This means that the node resource allocation is relatively reasonable, and the resources of all nodes can be effectively utilized, thereby improving the overall performance and stability of the system. Conversely, when the load distribution is normal, it indicates that the load differences among the nodes in the system are small, the load balance is good, and there is no load imbalance. This indicates a significant load difference among nodes, suggesting a load imbalance. This result indicates that some nodes are under excessive load, while other nodes may be underutilizing resources. This could lead to decreased response speed, increased latency, or even resource exhaustion on high-load nodes. Therefore, when the load is unbalanced, timely load redistribution or adjustment is necessary to ensure stable system operation and improved user experience.

[0083] The "pre-set standard deviation threshold for the load factor" can be determined through statistical analysis. The specific steps are as follows: First, the system can collect the distribution of the load factor (a comprehensive load assessment value combining resource consumption factor and response latency index) of each node across different time periods based on historical operating data. Next, standard deviation analysis is performed on this historical data to calculate the average standard deviation of the load factor for each time period, which serves as the benchmark value for load balancing. To address system load fluctuations, a certain tolerance range can be added based on the system's tolerance (e.g., increasing or decreasing the average standard deviation by a certain percentage), thereby determining the threshold for the standard deviation of the load factor. This method can automatically analyze historical data and dynamically adjust the standard deviation threshold through software, ensuring that the threshold meets the actual load conditions and the system's balancing requirements.

[0084] To build a load balancing evaluation model, the resource consumption coefficient of each node must first be determined. and response latency index Combined, a weighted summation is used to generate the overall load factor for each node. Specifically, the formula for calculating the load factor can be expressed as follows: ,in and These are weighting coefficients used to reflect the importance of resource consumption and response latency to load assessment. In practical settings, these weights can be adjusted according to the system's performance requirements. For example, if the system has higher requirements for response speed, The value can be set larger, allowing response latency to account for a larger proportion of the load factor; if the system focuses more on the balanced allocation of resource utilization, it can be increased. The value of these two weights is used to improve the impact of resource consumption. By reasonably adjusting these two weights, the load factor can be made more in line with the actual application requirements, achieving the goal of accurately assessing and effectively balancing the load of each node.

[0085] In this embodiment, if the determination result indicates that there is an imbalance in node load, a pre-set load factor threshold range is determined. The threshold range Load factor of each node A comparison is performed, and based on the comparison results, each node is divided into high-load nodes, normal-load nodes, and low-load nodes. The specific comparison and division are as follows:

[0086] like This node is classified as a low-load node.

[0087] like This node is classified as a normal load node.

[0088] like This node is then classified as a high-load node.

[0089] These three scenarios represent different states of node load, each impacting the overall system load distribution and resource utilization differently. A low-load node indicates insufficient resource utilization, with redundant resources not being effectively used, potentially leading to wasted system resources. A normal-load node indicates that the node's load is within an ideal range, with moderate resource utilization and stable operation, contributing to system efficiency and stability. A high-load node indicates excessive load, potentially indicating resource exhaustion or overload operation, easily causing increased response latency, performance degradation, or even system crashes, impacting user experience and system stability. Overall, the system needs to adjust the load based on these classifications, distributing overloaded tasks to low-load nodes to achieve reasonable resource allocation and balanced overall system load, ensuring efficient and stable operation.

[0090] Determine the pre-set load factor threshold range This can be achieved through historical data analysis and statistical methods. Specifically, the system can calculate the average and standard deviation of these load factors by analyzing the resource consumption coefficients and response latency indices of each node over a historical period. First, the distribution of load factors is obtained through historical load data, using the average as a benchmark. Then, a deviation range is set based on the actual load requirements of the system. For example, the average minus a standard deviation can be used as the baseline. This indicates the lower limit of a moderate load, while the average value plus a standard deviation is used as the lower limit. This indicates the upper limit of a moderate load. Nodes meeting normal load conditions will primarily be distributed within this range, while nodes deviating from this range represent abnormal loads. Through this method based on historical data and statistical analysis, the system can automatically and dynamically adjust the load threshold to more accurately adapt to the actual load situation of the system.

[0091] The tiered load adjustment mechanism construction module constructs a load adjustment mechanism based on the division results of each node, which is used to make different adjustments to high-load nodes, normal-load nodes and low-load nodes respectively.

[0092] In this embodiment, in the hierarchical load adjustment mechanism construction module, based on the division results of each node, a load adjustment mechanism is constructed to perform different adjustments on high-load nodes, normal-load nodes, and low-load nodes, specifically including:

[0093] Based on the division of high-load nodes, normal-load nodes, and low-load nodes, different load adjustment parameters are set to form a hierarchical load adjustment mechanism. This load adjustment mechanism automatically determines the priority and adjustment range of load adjustment based on the load coefficient of each node and the current request processing status, through pre-set rules.

[0094] To achieve a tiered load adjustment mechanism by setting different load adjustment parameters based on the classification of high-load, normal-load, and low-load nodes, an automated load monitoring and intelligent scheduling system can be used. First, the system needs to monitor the load factor and current request processing status of each node in real time and transmit this data to the load management module. This module classifies nodes (high-load, normal-load, and low-load) using preset rules and thresholds and applies different adjustment parameters based on the classification results. For example, for high-load nodes, the system can set a higher load transfer priority, automatically triggering request redirection to transfer some requests to lower-load nodes, thus alleviating the pressure on high-load nodes. For normal-load nodes, the system maintains a balancing strategy, preserving existing resource allocation as much as possible while satisfying current requests. For low-load nodes, the system may reduce the frequency of resource allocation adjustments to reduce the scheduling burden.

[0095] This tiered load balancing mechanism can be implemented through machine learning models or rule-based scheduling algorithms. Specifically, it involves dynamically setting the priority and adjustment range for load balancing for each node. For example, the system can preset different parameter weights, and when the node classification results are updated, the corresponding weight is automatically selected and applied to the load balancing decision. The core of this approach is to adapt the adjustment strategy in real time based on the node's load status, ensuring that high-load nodes do not affect system response speed due to overload, while ensuring that the resource utilization of normal-load and low-load nodes is not idle and wasted. This flexible tiered load balancing mechanism can optimize overall system performance, ensure stability in high-concurrency environments, improve user experience, and reduce resource waste. This approach can not only be implemented automatically through software but also adapt to the load requirements of different scenarios, ensuring the system's elasticity and dynamic adaptability.

[0096] Different load adjustment operations are performed on high-load nodes, normal-load nodes, and low-load nodes respectively. Specifically: in high-load nodes, the high load adjustment parameters in the load adjustment mechanism are used to prioritize request redirection and reduce resource consumption; in normal-load nodes, the standard load adjustment parameters in the load adjustment mechanism are maintained to maintain the current load distribution and balance resource usage; in low-load nodes, the low load adjustment parameters in the load adjustment mechanism are used to reduce load adjustment operations and maintain the existing load level.

[0097] To implement different load adjustment operations for high-load, normal-load, and low-load nodes, an automated load management system and tiered scheduling strategies can be used. First, for high-load nodes, the system will automatically prioritize request redirection and resource reduction strategies by setting high-load adjustment parameters. Specifically, some requests can be dynamically transferred to other lower-load nodes. This can be achieved through an algorithm-based request distributor, such as a load balancing algorithm (e.g., round-robin or response-time-based algorithms), ensuring that high-load nodes do not experience performance degradation due to excessive requests. Simultaneously, the system can reduce resource consumption on high-load nodes, such as limiting cache space or reducing resource allocation for background tasks, to alleviate current load pressure and improve response speed.

[0098] For nodes with normal load, the system uses standard load adjustment parameters to achieve relatively stable resource allocation and maintain the current load level. This means the system will maintain its existing request allocation strategy and ensure balanced resource usage through resource monitoring and scheduling algorithms. The core of nodes with normal load is to meet current request demands while preventing load fluctuations. Therefore, the system will maintain a uniform request flow on these nodes and adjust resources appropriately to maintain efficiency. This approach ensures that nodes with normal load can achieve their optimal performance without interfering with system stability.

[0099] For low-load nodes, the system will reduce frequent load adjustment operations and maintain the existing load level by adjusting parameters during low load periods. This can be achieved by reducing the frequency of request allocation to nodes, avoiding unnecessary resource waste. For example, the system can reduce resource calls to low-load nodes or appropriately reduce the number of active tasks on them, allowing resources to be released or reallocated when demand is insufficient. By maintaining the stable state of low-load nodes, the system can avoid excessive resource dispersion and improve the overall resource utilization efficiency of the system.

[0100] This tiered load balancing approach can be automated through software. Dynamic monitoring and intelligent scheduling allow the system to adapt to the needs of nodes under varying load conditions. This ensures that high-load nodes avoid overload, normally loaded nodes operate stably, and low-load nodes avoid resource waste, thereby achieving system load balancing, improving overall performance and stability, and maximizing resource utilization.

[0101] The dynamic feedback and optimization evaluation module acquires feedback information from each node in real time during the load scheduling mechanism's adjustment of each node, analyzes the acquired information, evaluates whether the load adjustment mechanism's adjustment effect on each node can meet expectations, and optimizes the load adjustment mechanism based on the evaluation results.

[0102] In this embodiment, in the dynamic feedback and optimization evaluation module, after obtaining the feedback information of each node in real time, it is preprocessed, and the load adaptation information and request recovery information in the preprocessed feedback information of each node are extracted and analyzed to generate the load distribution adaptation coefficient and request processing recovery index of each node.

[0103] Real-time feedback from each node can be obtained through an automated monitoring system and data acquisition tools. First, the system sets up a data acquisition agent on each node to monitor key performance indicators during load adjustment, including load adaptation information (such as average load distribution rate and load adjustment stability) and request recovery information (such as request processing success rate and latency recovery rate). Each agent uploads the real-time collected data to the central monitoring platform, using polling or event-triggered methods to ensure real-time data acquisition. After receiving node feedback, the central monitoring platform automatically processes this data for subsequent preprocessing and analysis. This automated data acquisition mechanism ensures that feedback information is uploaded to the analysis module in real-time and continuously, enabling the system to dynamically monitor the adjustment effects of the nodes.

[0104] The purpose of preprocessing feedback information is to ensure data consistency, accuracy, and stability, providing reliable input for subsequent load balancing adaptation coefficients and request processing recovery indices. The primary task of preprocessing is noise reduction, which uses filtering algorithms to remove outliers (such as short-term spikes in request latency) to avoid bias. Secondly, data standardization is performed, transforming feedback information from different nodes into a unified metric, such as standardizing request processing success rates as percentages, making the data more comparable across different nodes. Finally, smoothing is implemented using moving averages or exponentially weighted smoothing algorithms to reduce data fluctuations caused by sudden requests or adjustments. Preprocessed data is more stable, reflecting the true node feedback situation and providing higher-quality input for subsequent analysis and evaluation. Automating the preprocessing process through software helps improve system reliability and evaluation accuracy.

[0105] In this embodiment, the logic for obtaining the load distribution fitness coefficient and request processing recovery index of each node is as follows:

[0106] Extract load adaptation information from the preprocessed feedback information of each node. Specifically, this includes the proportion of requests allocated to each node to the total number of system requests at different times during a period of time as the load scheduling mechanism adjusts each node, the change in the number of requests allocated to each node, and the average response latency for each node to process requests transferred from other nodes. Then, apply these parameters according to the time series using functions... , and To express, For a point in time, This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The proportion of requests allocated to each node relative to the total number of requests in the system. This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The change in the number of requests allocated to each node This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The average response latency for each node to process requests transferred from other nodes, defined as the time period. , , It is a positive integer;

[0107] To extract load adaptation information, a real-time monitoring and data acquisition system can be set up on each node. During load scheduling, the system collects request processing information, load changes, and response times for cross-node requests from each node. First, the system can record the number of requests received by each node at different times using request allocation logs or a load balancer, and calculate its proportion of the total number of requests in the system. Second, load changes can be extracted by analyzing the increase or decrease in the number of requests at each node at different times. Finally, the system monitors the average response latency for processing requests transferred from other nodes, which can be obtained by measuring the difference between the arrival time and completion time of each transferred request. All this data is collected in real time and transmitted to a central data analysis module for further analysis.

[0108] The number of requests received by each node at each time point can be recorded in real time by the request allocator. By comparing the number of requests received by each node with the total number of requests in the system, the proportion of requests allocated can be calculated. This proportion reflects the distribution of the node in the total load and can quantify the load weight it bears. The change in the number of requests received by a node can be obtained by analyzing the difference in the number of requests at consecutive time points. Specifically, the system collects the number of requests received by each node within a set time interval and calculates the change in the number of requests at adjacent time points. This change can reflect the fluctuations during the load adjustment process and reveal the stability or volatility of the load distribution. The response latency of each node in processing transfer requests from other nodes is obtained by calculating the difference between the arrival time and completion time of each transfer request. The system can analyze the request latency by recording the timestamps of transfer requests, thereby quantifying the speed and efficiency of the node in processing cross-node requests during the load adjustment process. This data directly reflects the node's adaptability to load transfer.

[0109] The load distribution fitness coefficient for each node is calculated using the following formula:

[0110] ;

[0111] In the formula, For the first The load distribution adaptation factor for each node;

[0112] This formula accurately measures the adaptability of each node to request allocation during load adjustment through a combination of integration and nonlinear operations. The specific explanation is as follows: First, the integral term in the formula represents the time interval... The load distribution data within the time frame is averaged to ensure that the calculation results can smooth out instantaneous fluctuations and reflect the load adaptability of nodes over the entire time period. Part Two It involves non-linear processing of the request allocation ratio per unit time, gradually reducing the impact of high load conditions through a fractional approach to prevent an excessively large proportion of high-load nodes from causing imbalance. Part Three This approach uses a fractional-squared method to combine load distribution fluctuations with request transfer response latency, balancing their impact. Lower fluctuations and faster response latency make nodes more adaptable during load adjustments. Overall, this formula accurately and comprehensively reflects a node's adaptability to request distribution during load adjustments. Load Distribution Adaptability Coefficient The higher the value, the more adaptable the node is to load adjustments, thus helping the system to better evaluate and optimize the load distribution mechanism.

[0113] No. Load distribution fitness coefficient of each node The size of this value directly reflects the node's adaptability to request allocation during load balancing and is an important indicator for evaluating whether the load balancing mechanism has achieved its expected results. A higher value indicates that the node can handle the allocated requests well after load balancing, exhibiting a stable request distribution ratio, low load fluctuation, and fast request transfer response speed. This means that the load balancing mechanism has achieved its expected effect on this node, and its load adaptability is good. Conversely, a lower value indicates... The value indicates that the node is struggling to adapt to the current load distribution after adjustment, potentially resulting in uneven distribution, long response times, or significant fluctuations, suggesting that the load adjustment is not effective. Therefore, comparing the values ​​of each node... The value can be used to determine the effectiveness of the load adjustment mechanism on each node and identify nodes that need further optimization.

[0114] Extract request recovery information from the preprocessed feedback information of each node. Specifically, this includes the increase in request processing success rate, request latency recovery rate, and reduction in request processing failure rate of each node at different times during a period of time as the load scheduling mechanism adjusts each node. Then, apply these parameters according to the time series using a function. , and To express, For a point in time, This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The improvement in the success rate of request processing per node. This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The request latency recovery rate of each node, This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The reduction in the failure rate of request processing per node, defined over a time period of [time period missing]. ;

[0115] Real-time monitoring and data acquisition modules can be deployed on each node to extract request recovery information. During load balancing, the system monitors the changes in request processing success rate, request latency, and failure rate on each node in real time. First, the system analyzes the request processing logs of each node to obtain the request processing success rate at different time points and compares it with the success rate before load balancing to extract the improvement in success rate. Second, the system monitors the changes in request processing latency after load balancing using request response time logs to calculate the latency recovery rate. Finally, the system records the frequency of request failures to calculate the reduction in the request failure rate. This data is uploaded to the central data analysis module for unified processing at each time interval to obtain real-time feedback.

[0116] Improvement in request processing success rate: The system can determine the improvement by calculating the request processing success rate at each time point before and after the adjustment. Specifically, the improvement in request processing success rate is the percentage increase in the success rate after the adjustment relative to the success rate before the adjustment. For example, if the success rate before the adjustment was 80% and after the adjustment it is 90%, then the improvement is 12.5%{(90−80) / 80}. This reflects the improvement in request processing by the nodes after load adjustment.

[0117] Request latency recovery rate: The request latency recovery rate is obtained by calculating the rate at which the node's request latency decreases after load adjustment. Specifically, the average request latency before and after adjustment is recorded, the latency before adjustment is subtracted, and then divided by the latency after adjustment and the recovery time (i.e., the time required for latency to stabilize). This quantifies the node's latency recovery after load adjustment. A higher recovery rate indicates that the node can recover to a low-latency state more quickly.

[0118] Reduction in request processing failure rate: The reduction in request processing failure rate refers to the difference in failure rate before and after load adjustment. This is calculated by monitoring the request failure rate at each time point before and after the adjustment. For example, if the failure rate was 5% before adjustment and 2% after adjustment, the reduction is 3%. This data reflects the degree to which a node improves its handling of failed requests after load adjustment and is a key indicator for evaluating stability.

[0119] The request processing recovery index for each node is calculated using the following formula:

[0120] ;

[0121] In the formula, For the first The request processing recovery index of each node.

[0122] This formula comprehensively evaluates the recovery performance of each node in request processing after load adjustment through integration and nonlinear processing. The specific explanation is as follows: First, the integral term represents the time interval... The data is averaged to smooth out short-term fluctuations in node performance during the recovery process, resulting in a more stable request processing recovery index. Part Two Increase the success rate of requests by multiplying the product. and request latency recovery rate The combined effect of these factors. This product term accurately reflects the node's overall recovery performance in terms of improved success rate and delayed recovery. Part Three Using fractions and squares to reduce the request failure rate. The relationship with the other two terms. In the denominator This design balances the impact of success rate and latency recovery on failure rate reduction, ensuring the rationality of all data in the overall assessment. It accurately measures the request processing recovery of nodes after load adjustment by comprehensively considering request success rate, latency recovery speed, and failure rate reduction. A high request processing recovery index is achieved. The value indicates that the node's recovery effect is significant, and the load adjustment mechanism has achieved the expected results.

[0123] No. Request processing recovery index per node The magnitude of this value directly reflects the recovery of the node's request processing capacity after load adjustment, and is a key indicator for evaluating whether the load adjustment mechanism is effective on each node. A higher value indicates that the node has achieved significant improvements in request success rate, request latency recovery speed, and failure rate reduction. This demonstrates that the load balancing mechanism has effectively helped the node recover quickly and improved request processing performance, and the adjustment effect can be considered to have met expectations. Conversely, when... A lower value indicates that the node's request processing recovery after load adjustment is not ideal, potentially showing insufficient performance in success rate, latency recovery, or failure rate, suggesting that the adjustment mechanism has failed to adequately meet the node's needs. Therefore, comparing the values ​​of each node... The value can accurately determine the effectiveness of the load adjustment mechanism and identify nodes that may require further optimization.

[0124] In this embodiment, the load distribution adaptation coefficients for each node are generated. and request processing recovery index Then, an adjustment degree assessment model is constructed, and the adjustment coefficients for each node are generated by weighted summation. And the adjustment coefficients of each generated node Adjustment coefficient thresholds for each pre-set node A comparison was conducted, and the load adjustment mechanism's effectiveness at each node was evaluated based on the comparison results. The load adjustment mechanism was then optimized based on the evaluation results. The specific comparison analysis is as follows:

[0125] like The load balancing mechanism did not achieve the expected results at this node, and needs to be optimized. Specifically, this includes: adjusting node load allocation parameters to balance resource distribution; optimizing request redirection strategies to reduce node load; improving resource scheduling strategies to increase node resource allocation; increasing the frequency of dynamic monitoring and feedback to capture load changes in a timely manner; and resetting the priority and magnitude of load balancing to adapt to high load conditions.

[0126] This situation indicates that the load balancing mechanism is not performing well at this node and has failed to meet the system's performance expectations. This suggests that the node's improvements in request processing success rate, load balancing, or latency recovery are insufficient, and it cannot adequately adapt to the current load demands. In this case, without optimization of the load balancing mechanism, the node may experience performance bottlenecks under high load scenarios, thereby affecting the overall stability and response speed of the system and negatively impacting user experience and service quality.

[0127] Optimizing load balancing mechanisms can be achieved through software configuration and real-time data analysis. First, the system can adjust node load allocation parameters to make resource allocation more adaptive. This can be done by setting different weight values ​​for different nodes; for example, high-load nodes can be assigned lower weights, thus receiving fewer requests in the load allocation algorithm. Second, optimizing request redirection strategies can be achieved through a dynamic load balancing module that analyzes node load in real time. When a node's load exceeds a threshold, its requests are redirected to other low-load nodes to alleviate its burden. Furthermore, improving resource scheduling strategies can be achieved through a dynamic resource management module, such as allocating more memory or computing resources to high-load nodes to enhance their processing capabilities and better handle sudden increases in requests.

[0128] The aforementioned optimization measures aim to improve the system's adaptability and response speed under high load conditions. Adjusting load balancing parameters ensures balanced resource allocation and prevents resource overload; optimizing request redirection strategies can avoid performance degradation caused by overload of individual nodes; and improving resource scheduling strategies can effectively alleviate the pressure on high-load nodes, thereby improving overall service stability. These optimization measures are dynamically adjusted through software, enabling the system to maintain efficient and stable operation even under environments with large load fluctuations, ultimately improving user experience and system reliability.

[0129] like The load adjustment mechanism at this node achieves the expected adjustment effect, and there is no need to optimize the load adjustment mechanism.

[0130] This indicates that the load balancing mechanism is performing as expected on this node. This means the node is able to allocate and handle the load effectively, successfully improving load adaptability and request processing recovery. In this case, no further optimization of the load balancing mechanism is needed, which helps save system resources and allows more computing power to be used for other nodes or tasks, thereby improving the overall resource utilization efficiency of the system.

[0131] In generating load distribution adaptation coefficients for each node. and request processing recovery index Then, the adjustment effect of each node can be quantified by constructing an adjustment degree evaluation model. Specifically, this involves... and Perform a weighted summation to generate the adjustment coefficients for each node. The formula is as follows: ,in and This refers to the weighting coefficient. The weighting coefficient is set based on the system's relative emphasis on load adaptability and request processing recovery. If the system prioritizes load balancing across nodes, it can assign a weighting coefficient. Higher weight Conversely, if the system focuses more on the node's recovery performance in request processing, it can increase... weight This weighted summation method integrates the two indicators into a comprehensive index. This allows for a more comprehensive reflection of the load adjustment effect of each node, enabling the system to adjust accordingly. Evaluate whether the node needs further optimization.

[0132] The system performance optimization report generation module comprehensively analyzes the load status information, feedback information, and adjustment records of each node to generate a system performance optimization report, so as to provide data support and decision-making basis for subsequent load management strategies and resource configuration optimization.

[0133] The system performance optimization report generation module utilizes data acquisition, statistical analysis, and automated report generation tools. First, the module extracts load status information, feedback information, and adjustment records for each node from the database and inputs this data into the analysis engine for comprehensive analysis. Using statistical analysis algorithms, the system calculates key indicators such as performance trends, load adaptability, and recovery effectiveness for each node. Next, the analysis engine compares and summarizes these indicators, automatically generating reports summarizing the performance of each node, load distribution, and resource utilization. Finally, the report generation tool synthesizes the analysis results into visual charts and text descriptions, forming a complete system performance optimization report for administrators to review and reference.

[0134] The primary purpose of generating system performance optimization reports is to provide data support and decision-making basis for subsequent load management strategies and resource allocation optimization. By systematically analyzing the load and adjustment effects of each node, administrators can identify which nodes perform well under high loads and which nodes require further optimization. Such reports not only help system administrators quickly locate performance bottlenecks and resource allocation issues but also provide accurate data support, making subsequent load management and resource optimization decisions more scientific and efficient. Utilizing software to generate these reports improves the efficiency and accuracy of data analysis, reduces errors from human analysis, and thus enables continuous system optimization and efficient operation.

[0135] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0136] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0137] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0138] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0142] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A distributed storage management system for real estate data, characterized in that, It includes a high concurrency identification and load status acquisition module, a node load analysis and classification module, a hierarchical load adjustment mechanism construction module, a dynamic feedback and optimization evaluation module, and a system performance optimization report generation module; The high concurrency identification and load status acquisition module monitors the status of each node in the distributed storage system in real time to identify whether there are high-concurrency users accessing the platform's video viewing function. When high-concurrency users access the platform's video viewing function, it acquires the load status information of each node in real time. The node load analysis and classification module analyzes the load status information of each node acquired in real time, determines whether there is an imbalance in node load, and classifies each node into high-load nodes, normal-load nodes, and low-load nodes based on the judgment results. In the node load analysis and classification module, after acquiring the load status information of each node in real time, the load status information of each node is preprocessed, and the resource usage information and response performance information in the preprocessed load status information of each node are extracted and analyzed to generate the resource consumption coefficient and response latency index of each node. The logic for obtaining the resource consumption coefficient and response latency index of each node is as follows: Resource usage information is extracted from the preprocessed load status information of each node. Specifically, this includes the percentage of total bandwidth used by each node at different times within a given period when processing video viewing requests, the percentage of total CPU time used within the overall time, and the rate at which data is read from storage. These are then labeled as follows: , and , Indicates the first Each node processes video viewing requests within a certain period of time. The percentage of total bandwidth used at any given time. Indicates the first Each node processes video viewing requests within a certain period of time. The percentage of total CPU usage time at any given moment out of the total total time. Indicates the first Each node processes video viewing requests within a certain period of time. The rate at which data is read from storage at any given time. , , and All are positive integers; The resource consumption coefficient of each node is calculated using the following formula: ； In the formula, For the first Resource consumption coefficient of each node; Extract response performance information from the preprocessed load status information of each node. Specifically, this includes the average time each node takes to process a single request at different times within a given period, the maximum length of the request queue, and the failure rate of each node during request processing within that period. These are then labeled as follows: , and , Indicates the first A node within a certain period of time The average time required to process a single request at any given moment. Indicates the first A node within a certain period of time The maximum length of the request queue processed at any given time. Indicates the number of times within that time period The rate at which each node fails to process a request; The response latency exponent for each node is calculated using the following formula: ； In the formula, For the first The response latency index of each node; The tiered load adjustment mechanism construction module constructs a load adjustment mechanism based on the division results of each node, which is used to make different adjustments to high-load nodes, normal-load nodes and low-load nodes respectively. The dynamic feedback and optimization evaluation module acquires feedback information from each node in real time during the load scheduling mechanism's adjustment of each node, analyzes the acquired information, evaluates whether the load adjustment mechanism's adjustment effect on each node can meet expectations, and optimizes the load adjustment mechanism based on the evaluation results. The system performance optimization report generation module comprehensively analyzes the load status information, feedback information, and adjustment records of each node to generate a system performance optimization report, so as to provide data support and decision-making basis for subsequent load management strategies and resource configuration optimization.

2. The distributed storage management system for real estate data according to claim 1, characterized in that, The resource consumption coefficients of each generated node and response latency index Construct a load balancing evaluation model and generate the load coefficient of each node through weighted summation. Calculate the load factor of all nodes A comprehensive analysis was conducted to generate the standard deviation of the load factor. According to the formula: and the standard deviation of the generated load factor Compared with the pre-set standard deviation threshold of the load factor A comparison is performed, and the results are used to determine whether there is any unbalanced node load. The specific comparison analysis is as follows: like There is no issue of uneven node load. like There is a situation where the node load is unbalanced.

3. The distributed storage management system for real estate data according to claim 2, characterized in that, If the judgment result indicates that there is an imbalance in node load, a pre-set load factor threshold range is determined. The threshold range With the load factor of each node A comparison is performed, and based on the comparison results, each node is divided into high-load nodes, normal-load nodes, and low-load nodes. The specific comparison and division are as follows: like This node is classified as a low-load node. like This node is classified as a normal load node. like This node is then classified as a high-load node.

4. The distributed storage management system for real estate data according to claim 3, characterized in that, In the tiered load balancing mechanism construction module, based on the partitioning results of each node, a load balancing mechanism is constructed to perform different adjustments on high-load nodes, normal-load nodes, and low-load nodes, specifically including: Based on the division of high-load nodes, normal-load nodes, and low-load nodes, different load adjustment parameters are set to form a hierarchical load adjustment mechanism. This load adjustment mechanism automatically determines the priority and adjustment range of load adjustment based on the load coefficient of each node and the current request processing status, through pre-set rules. Different load adjustment operations are performed on high-load nodes, normal-load nodes, and low-load nodes respectively. Specifically: in high-load nodes, the high load adjustment parameters in the load adjustment mechanism are used to prioritize request redirection and reduce resource consumption; in normal-load nodes, the standard load adjustment parameters in the load adjustment mechanism are maintained to maintain the current load distribution and balance resource usage; in low-load nodes, the low load adjustment parameters in the load adjustment mechanism are used to reduce load adjustment operations and maintain the existing load level.

5. The distributed storage management system for real estate data according to claim 4, characterized in that, In the dynamic feedback and optimization evaluation module, after acquiring the feedback information of each node in real time, it is preprocessed, and the load adaptation information and request recovery information of each node are extracted from the preprocessed feedback information. After extraction, the information is analyzed to generate the load distribution adaptation coefficient and request processing recovery index of each node.

6. The distributed storage management system for real estate data according to claim 5, characterized in that, The logic for obtaining the load distribution fitness coefficient and request processing recovery index of each node is as follows: Extract load adaptation information from the preprocessed feedback information of each node. Specifically, this includes the proportion of requests allocated to each node to the total number of system requests at different times during a period of time as the load scheduling mechanism adjusts each node, the change in the number of requests allocated to each node, and the average response latency for each node to process requests transferred from other nodes. Then, apply these parameters according to the time series using functions... , and To express, For a point in time, This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The proportion of requests allocated to each node relative to the total number of requests in the system. This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The change in the number of requests allocated to each node This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The average response latency for each node to process requests transferred from other nodes, defined as the time period. , , It is a positive integer; The load distribution fitness coefficient for each node is calculated using the following formula: ； In the formula, For the first The load distribution adaptation factor for each node; Extract request recovery information from the preprocessed feedback information of each node. Specifically, this includes the increase in request processing success rate, request latency recovery rate, and reduction in request processing failure rate of each node at different times during a period of time as the load scheduling mechanism adjusts each node. Then, apply these parameters according to the time series using a function. , and To express, For a point in time, This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The improvement in the success rate of request processing per node. This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The request latency recovery rate of each node, This indicates that the load scheduling mechanism is applied to the first... During the adjustment of each node, for a period of time Time of the first The reduction in the failure rate of request processing per node, defined over a time period of [time period missing]. ; The request processing recovery index for each node is calculated using the following formula: ； In the formula, For the first The request processing recovery index of each node.

7. The distributed storage management system for real estate data according to claim 6, characterized in that, In generating load distribution adaptation coefficients for each node. and request processing recovery index Then, an adjustment degree assessment model is constructed, and the adjustment coefficients for each node are generated by weighted summation. And the adjustment coefficients of each generated node Adjustment coefficient thresholds for each pre-set node A comparison was conducted, and the load adjustment mechanism's effectiveness at each node was evaluated based on the comparison results. The load adjustment mechanism was then optimized based on the evaluation results. The specific comparison analysis is as follows: like The load balancing mechanism did not achieve the expected results at this node, and needs to be optimized. Specifically, this includes: adjusting node load allocation parameters to balance resource distribution; optimizing request redirection strategies to reduce node load; improving resource scheduling strategies to increase node resource allocation; increasing the frequency of dynamic monitoring and feedback to capture load changes in a timely manner; and resetting the priority and magnitude of load balancing to adapt to high load conditions. like The load adjustment mechanism at this node achieves the expected adjustment effect, and there is no need to optimize the load adjustment mechanism.

Images