A large-scale e-commerce order data distributed storage processing method

Abstract

This invention discloses a distributed storage processing method for large-scale e-commerce order data. By collecting data such as disk usage and CPU load, the load rate of each storage node is calculated, providing a basis for subsequent node classification. Storage nodes are classified according to load level, distinguishing between high-load and low-load node groups. Clustering algorithms are used to automatically divide node groups for targeted management. When a high-load node reaches a threshold, available backup nodes are selected from the low-load group to ensure the system can quickly find replacement nodes when some nodes are overloaded. A dynamic filtering algorithm is used to determine the path with the lowest read / write latency between nodes, establishing an efficient data access route. By identifying key data requests and assigning them appropriate storage paths, important requests are ensured to receive priority access to high-quality resources. Finally, high-priority order data is migrated from overloaded nodes to optimized paths, achieving load balancing and optimized data access.

Description

Technical Field

[0001] This invention relates to the field of dynamic scheduling, storage, and processing of network data, and in particular to a distributed storage and processing method for large-scale e-commerce order data. Background Technology

[0002] With the rapid development of large-scale e-commerce platforms and the continuous surge in business volume, order data volume has exploded, especially during peak business periods such as promotional activities and holidays. Distributed storage systems face severe challenges in handling high-concurrency read / write operations and low-latency processing. Traditional solutions typically rely on static path selection methods based on instantaneous state measurements, such as the classic Dijkstra algorithm. This method determines the so-called shortest storage path by analyzing the instantaneous read / write latency between two storage nodes at a given moment. However, this static analysis approach has several inherent technical limitations in dynamic and complex network environments.

[0003] First, this method relies solely on single or short-term instantaneous measurements, failing to effectively capture the continuous dynamic changes in path transmission performance within the network environment. In actual operation, network conditions exhibit significant time-varying characteristics, with read / write latency fluctuating continuously with data traffic, node load, and network congestion levels. When the system evaluates paths, if a path happens to be in a momentarily low-load state, the algorithm mechanically identifies it as the optimal path, completely ignoring the potential performance degradation risk that may occur at subsequent times. This decision-making mechanism based on instantaneous snapshots lacks the ability to predict path performance trends, leading to path selection results that often deviate significantly from reality.

[0004] Secondly, traditional methods lack a comprehensive consideration and analysis mechanism for the stability differences of multiple storage paths between the same node pair. The system only focuses on the instantaneous latency metric of a single path, and cannot effectively identify potentially risky paths that, although their instantaneous latency values ​​are low, actually fluctuate drastically. In e-commerce order processing scenarios, although such paths may exhibit low latency at certain times, their instability can easily cause severe fluctuations in the response time of high-priority businesses such as payment processing and order creation, seriously affecting user experience and business continuity.

[0005] Furthermore, existing solutions fail to establish an effective mechanism to organically integrate historical trend data with real-time measurements during the path selection decision-making process. The system neither establishes an evaluation model for the long-term performance of paths nor considers the collaborative relationships and overall stability characteristics among multiple paths. This one-sided decision-making mechanism leads to frequent problems of biased path selection decisions and uneven load distribution in high-concurrency order processing scenarios, especially during peak business periods such as promotional activities. A large number of data requests are concentrated on certain paths that perform well momentarily but have limited actual carrying capacity. These paths quickly become new performance bottlenecks, ultimately affecting the response speed of critical business processes such as payment processing and the operational stability of the entire distributed storage system. Summary of the Invention

[0006] The purpose of this invention is to provide a distributed storage and processing method for large-scale e-commerce order data, which solves the aforementioned technical problems pointed out in the prior art.

[0007] This invention provides a distributed storage and processing method for large-scale e-commerce order data, comprising the following steps:

[0008] The current load data and capacity indicators of each storage node are collected in real time by the monitoring sensors deployed on the storage nodes. The node load rate of each path is calculated based on the current load data and capacity indicators of each storage path.

[0009] Based on the load rate of the path nodes of each storage path, the K-Means clustering algorithm is applied to cluster all storage nodes to obtain high load rate group and low load rate group.

[0010] When the node load rate of any storage node in the high load rate group exceeds the preset load threshold, multiple backup storage nodes are selected from the low load rate group.

[0011] By using Kalman filtering, the shortest read / write latency path between every two alternative storage nodes is calculated among all candidate storage paths, and an optimized path set is established based on all the shortest read / write latency paths.

[0012] Retrieve all data requests on storage paths in the high-load group where the load rate of path nodes exceeds a preset load threshold; retrieve the data type information of each data request; determine the priority evaluation value of each data request based on the data type information; mark data requests with priority evaluation values ​​greater than or equal to the preset priority threshold as high-priority data requests; for each high-priority data request, select the shortest read / write latency path from the optimized path set as the target redirection path based on the particle swarm optimization algorithm combined with path state transient analysis.

[0013] Migrate high-priority data requests in high-load groups to the corresponding target redirection storage path.

[0014] Preferably, Kalman filtering is used to calculate the shortest read / write latency path between every two storage nodes among all candidate paths, including the following steps:

[0015] An optimized network topology is constructed based on all alternative paths; every two storage nodes in the optimized network topology are combined to obtain a start-destination node group; basic read / write latency test data sequence sets are obtained by collecting data at the same frequency for each start-destination node group;

[0016] Calculate the inter-path read / write latency fluctuation index based on the read / write latency data sequence of multiple storage paths between the same network start node and network destination node; calculate the initial read / write latency estimate and initial read / write latency change rate of each start-destination node group based on the basic read / write latency test data sequence set; and set the state vector, error covariance matrix, process noise matrix and measurement noise parameters of the Kalman filter in combination with the inter-path read / write latency fluctuation index.

[0017] For each start-destination node group, a Kalman filter is applied to predict the read / write delay state: a state prediction step is performed based on the state vector and the process noise matrix to obtain the prior state estimate and prior error covariance; then a measurement update step is performed based on the measurement noise parameters and the actual read / write delay test data to update the posterior state estimate and error covariance.

[0018] By combining the error covariance matrix, a multi-path dynamic comparison is performed on the read / write latency estimate and the read / write latency change rate in the updated posterior state estimation. The comprehensive score of each path is calculated, and the path with the lowest comprehensive score is selected as the shortest read / write latency path.

[0019] Preferably, the state vector is composed of the initial read / write delay estimate and the initial read / write delay change rate; the process noise matrix is ​​set according to the read / write delay sequence variance and the change rate sequence variance, and the inter-path read / write delay fluctuation index is introduced as an adjustment factor; the measurement noise parameter is set based on the measurement equipment error variance, and the inter-path read / write delay fluctuation index is introduced as an adjustment factor; the error covariance matrix is ​​set according to the initial measurement statistical variance and the trend estimate confidence, and the inter-path read / write delay fluctuation index is introduced as an adjustment factor.

[0020] Preferably, the inter-path read / write latency fluctuation index is calculated based on the read / write latency data sequence of multiple storage paths between the same network origin node and network destination node, including the following steps:

[0021] For all storage paths between the same network start node and destination node, obtain the complete read / write latency data sequence for each storage path within the current statistical period; calculate the average read / write latency of each storage path to form a path performance baseline vector; calculate the standard deviation of the read / write latency data for each storage path to form a path stability vector.

[0022] Set read / write latency stability level thresholds; divide each storage path into stability levels based on the path stability vector and the read / write latency stability level thresholds; and statistically analyze the distribution of the number of storage paths in each stability level.

[0023] Calculate the performance baseline dispersion of paths within each stability level, i.e., the standard deviation of the average read / write latency of each path within the same stability level; calculate the path coordination index of the performance of each storage path.

[0024] Based on the stability level distribution and path coordination index, the inter-path read / write latency fluctuation index is calculated.

[0025] Preferably, for each high-priority data request, a shortest read / write latency path is selected from the optimized path set as the target redirection path based on particle swarm optimization combined with path state transient analysis, including the following steps:

[0026] Multiple redirection combinations are randomly generated based on all high-priority data requests and the shortest read / write latency paths in the optimized path set; the particle position vector corresponding to each redirection combination is obtained; the maximum number of iterations and the iteration counter are set, with the iteration counter initially set to 0; the learning rate, search step size, and redirection combination search speed vector are set.

[0027] The remaining capacity data is calculated based on the current load data and capacity indicators of each shortest read / write latency path; and the instantaneous read / write latency value of each shortest read / write latency path is obtained at the same time; based on the remaining capacity data and instantaneous read / write latency value, the read / write latency jitter trend coefficient of each shortest read / write latency path is obtained through time series-based fluctuation and read / write latency analysis.

[0028] The overall fitness of each redirection combination is calculated based on the data volume of each high-priority data request in the redirection combination and the post-migration node load rate of each shortest read / write latency path; the overall fitness of each redirection combination is calculated based on the post-migration node load rate and the read / write latency jitter trend coefficient of each shortest read / write latency path.

[0029] The target redirection path is obtained by performing multi-level judgments based on the comprehensive fitness and preset fitness threshold, combined with the iteration count and maximum iteration count of the iteration counter, and iterative processing of the directional combined search velocity vector based on the learning rate and search step size.

[0030] Preferably, the target redirection path is obtained by performing multi-level judgments based on the comprehensive fitness and a preset fitness threshold, combined with the iteration count and maximum iteration count of the iteration counter, and by iteratively processing the directional combined search velocity vector based on the learning rate and search step size. This includes the following steps:

[0031] Determine if the overall fitness of each redirection combination is less than or equal to a preset fitness threshold. If yes, determine the corresponding redirection combination as the target redirection combination, and determine the shortest read / write latency path corresponding to each high-priority data request in the target redirection combination as the target redirection path. If no, increment the iteration count by 1 to obtain the current iteration count, and determine if the current iteration count is greater than or equal to the maximum iteration count. If the current iteration count is greater than or equal to the maximum iteration count, output the redirection combination with the minimum overall fitness under the current iteration count as the target redirection combination, and determine the shortest read / write latency path corresponding to each high-priority data request in the target redirection combination as the target redirection path.

[0032] If the current iteration number is less than the maximum iteration number, then determine whether the difference between the minimum comprehensive fitness and the fitness threshold under the current iteration number is greater than or equal to the preset fitness difference threshold; if so, then output the target redirection path based on the iterative processing of the directional combined search velocity vector according to the learning rate and search step size.

[0033] Preferably, the target redirection path is obtained by iteratively processing the directional combined search velocity vector based on the learning rate and search step size, including the following steps:

[0034] The search step size is updated using the learning rate to obtain a new search step size; the redirection combination search velocity vector is updated using the new search step size to obtain a new redirection combination search velocity vector; the particle position vector of each redirection combination is updated using the new redirection combination search velocity vector to obtain a new particle position vector; a new redirection combination is obtained based on the new particle position vector; the new redirection combination is returned to the above steps, and the new search step size and the new redirection combination velocity vector are returned to the above steps for re-execution until the target redirection path is output.

[0035] Preferably, based on the remaining capacity data and instantaneous read / write latency values, the read / write latency jitter trend coefficients for each shortest read / write latency path are obtained through time-series-based fluctuation and read / write latency analysis, including the following steps:

[0036] For each shortest read / write latency path, collect multiple historical read / write latency values ​​within a preset time period prior to the current time point;

[0037] The sliding average read / write latency value at each time point within the time period is calculated using a preset sliding time window; the read / write latency volatility within the sliding window is calculated by calculating the standard deviation of the sliding average read / write latency value at each time point within the preset time period; and the average read / write latency volatility within the time period is calculated based on the read / write latency volatility.

[0038] The moving average read / write latency values ​​at each time point within the time period are used to establish a moving average sequence based on the time series of each time point within the time period. Linear regression is performed on the moving average sequence to obtain the path read / write latency fluctuation line of the shortest read / write latency path. The slope of the path read / write latency fluctuation line is calculated using the least squares method.

[0039] The read / write latency jitter trend coefficient of the shortest read / write latency path is calculated based on the average read / write latency volatility and slope.

[0040] A preferred method for calculating the read / write latency jitter trend coefficient of the shortest read / write latency path is as follows: the ratio of the slope of the path read / write latency fluctuation line to the trend threshold is added to the ratio of the average read / write latency volatility to the benchmark volatility. This sum is then multiplied by a scaling factor and substituted into an S-shaped function for normalization mapping, ultimately outputting an evaluation value between 0 and 1.

[0041] In another aspect, the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described distributed storage and processing method for large-scale e-commerce order data.

[0042] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0043] Analysis of the large-scale e-commerce order data distributed storage processing method provided by this invention reveals that, in practical applications, firstly, real-time performance indicators of storage nodes are collected and node load rates are calculated to quantify the server's operating status and provide a data foundation for subsequent operations. Then, a clustering algorithm is applied to divide the storage nodes into high-load and low-load groups, classifying the node clusters within the server and establishing a target range for subsequent load scheduling. Furthermore, when the node load rate in the high-load group exceeds a preset threshold, a backup node is selected from the low-load group. After determining the backup node, a dynamic filtering algorithm is used to calculate the path with the lowest read / write latency between nodes, constructing an optimized path set and extending load balancing from the node level to the path level. Further, by identifying data requests and assigning priorities, high-priority requests are marked to ensure priority processing. Then, high-priority data requests are migrated from overloaded nodes to low-load backup nodes via the optimal path, reducing data pressure on storage nodes and storage paths, ensuring priority processing of high-priority requests, and improving the server's processing responsiveness. Attached Figure Description

[0044] Figure 1 A schematic diagram of the main process of a distributed storage and processing method for large-scale e-commerce order data;

[0045] Figure 2 This is a schematic diagram of redirection simulation in a distributed storage and processing method for large-scale e-commerce order data;

[0046] Figure 3 This is a schematic diagram simulating the basic read / write latency test data sequence set in a large-scale e-commerce order data distributed storage processing method;

[0047] Figure 4 This is a schematic diagram simulating multiple storage paths between the same network start node and network destination node in a distributed storage processing method for large-scale e-commerce order data.

[0048] Figure 5 A schematic diagram illustrating the baseline performance vector and baseline stability vector in a distributed storage and processing method for large-scale e-commerce order data. Detailed Implementation

[0049] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0051] Example 1

[0052] like Figure 1 As shown, Embodiment 1 of the present invention provides a distributed storage and processing method for large-scale e-commerce order data, including the following steps:

[0053] Step S10: Collect the current load data (such as disk utilization and CPU utilization) and capacity indicators (such as disk capacity and IOPS limit) of each storage node in real time through the monitoring sensors deployed on the storage nodes. Calculate the node load rate (the ratio of load data to capacity indicators, such as the ratio of disk utilization to total disk capacity) of each storage path based on the current load data and capacity indicators of each storage path.

[0054] Step S20: Based on the load rate of each storage path node, apply the K-Means clustering algorithm to cluster all storage nodes to obtain high load rate group and low load rate group (i.e., storage node grouping).

[0055] Step S30: When the node load rate of any storage node in the high load rate group exceeds the preset load threshold (e.g., 90%), select multiple backup storage nodes from the low load rate group.

[0056] It should be noted that in the above embodiments of this application, when the node load rate of a storage node in the high load rate group exceeds a preset load threshold, the data access path of the storage node is selected from the low load rate group. If the data starting node and the target node are on the same path and the path belongs to a storage node in the low load rate group, then it is used as a backup path (i.e., an alternative storage path).

[0057] Step S40: Calculate the shortest read / write latency path between every two alternative storage nodes in all candidate storage paths using Kalman filtering, and establish an optimized path set based on all shortest read / write latency paths;

[0058] Step S50: Obtain all data requests on storage paths where the load rate of path nodes in the high load rate group exceeds a preset load threshold; obtain the data type information of each data request; determine the priority evaluation value of each data request based on the data type information (data type information includes, for example, order creation, payment processing, inventory query, etc.); mark the data requests with priority evaluation values ​​greater than or equal to the preset priority threshold as high-priority data requests; for each high-priority data request, select the shortest read / write latency path from the optimized path set as the target redirection path based on the particle swarm optimization algorithm combined with path state transient analysis.

[0059] It should be noted that, as Figure 2 As shown, Figure 2 The system contains two terminal storage nodes, A and B, as well as three 101 storage nodes and five 201 storage nodes. The three 101 storage nodes are in a high-load group, with their load rates exceeding a preset load threshold (e.g., 90%). The five 201 storage nodes are in a low-load group. In this embodiment, step S40 analyzes the shortest read / write latency path between every two storage nodes in the candidate paths, rather than directly analyzing the shortest latency path from the storage node containing the high-priority data request to the candidate storage nodes. This is designed based on the architectural characteristics of the distributed storage system and the actual needs of e-commerce order data processing. In large-scale e-commerce order processing scenarios, the data request transmission path often involves multiple intermediate storage nodes. These nodes may perform different specialized functions, such as the order sender (front-end node) responsible for data authentication and compression, and the order receiver (core node) responsible for business logic and storage, forming a continuous processing pipeline. Therefore, the target redirection path may be composed of multiple consecutive shortest read / write latency paths, rather than a simple point-to-point path.

[0060] Specifically, by calculating the shortest read / write latency path between every two storage nodes and constructing an optimized path set, the system can flexibly combine these path segments to form an overall optimized path from the order sender to the order receiver. For example, during order payment processing, high-priority data requests may need to migrate from high-load storage nodes to low-load nodes, but the migration path may pass through multiple intermediate nodes (such as security gateway nodes, cache nodes, etc.), and the latency between each node pair will affect the overall performance. The Kalman filtering process in step S40 can dynamically predict and track the read / write latency changes between these node pairs, avoiding misjudgments based on instantaneous states and ensuring the stability and accuracy of path selection.

[0061] Furthermore, in step S50, when selecting the target redirection path based on the particle swarm optimization algorithm, it relies on combining multiple shortest paths from the optimized path set to adapt to the service decoupling architecture of the distributed storage system. For example, order data may first be encrypted at edge nodes before being forwarded to core nodes for persistent storage. Only by optimizing the read / write latency of each path can end-to-end data transmission efficiency be guaranteed. If only the direct path from the node containing the high-priority data request to the alternative node is analyzed, the performance bottleneck of intermediate nodes may be ignored, and global load balancing cannot be achieved. Therefore, this solution analyzes the path between every two storage nodes to construct a comprehensive path network, providing a reliable and low-latency migration solution for high-priority data requests, ultimately improving the overall response speed and stability of the e-commerce order system.

[0062] Step S60: Migrate high-priority data requests in the high-load group to the corresponding target redirection storage path.

[0063] It should be noted that in the embodiments of this application described above, the load rate of each storage node is calculated by collecting data such as disk usage and CPU load, providing a basis for subsequent node classification. For example, during peak e-commerce order periods, storage nodes with disk usage approaching their upper limit can be accurately identified, avoiding order processing delays due to insufficient resources. Furthermore, storage nodes are classified according to load levels, distinguishing between high-load and low-load node groups. Clustering algorithms are used to automatically divide node groups for targeted management. For example, nodes with a load rate exceeding 70% are classified as high-load groups, and those below 30% are classified as low-load groups, providing a clear target range for subsequent load balancing.

[0064] Furthermore, when a high-load node reaches a threshold, available backup nodes are selected from the low-load group to ensure that the system can quickly find replacement nodes when some nodes are overloaded. For example, when a storage node's load rate exceeds 90% due to a surge in order data, the system automatically selects a node with the same data path in the low-load group as a backup to ensure continuous access to order data. Dynamic filtering algorithms are used to determine the path with the lowest read / write latency between nodes, establishing efficient data access routes. For example, in a distributed storage system, the shortest read / write path from the order database node to the cache node is calculated to reduce the latency of order queries and writes, improving the user experience.

[0065] By identifying critical data requests and assigning them appropriate storage paths, we ensure that important requests receive priority access to high-quality resources. For example, payment processing requests are marked as high priority, and low-latency paths are allocated to them using optimization algorithms to prevent payment transactions from timeouts due to path congestion. Finally, high-priority order data is migrated from overloaded nodes to optimized paths, achieving load balancing and optimized data access. For instance, during promotional events, order data for best-selling products is migrated from high-load nodes to backup nodes to ensure the stability and responsiveness of the order processing system.

[0066] In the specific implementation of the above-described embodiments of this application, the traditional method analyzes the read / write latency between two storage nodes to obtain the shortest storage path. This traditional method typically uses Dijkstra's algorithm to analyze the instantaneous read / write latency between two storage nodes to determine the shortest storage path. However, this static analysis method based on instantaneous states has significant limitations. Since the transmission read / write latency between two nodes in a network environment is continuously and dynamically changing, when analyzing three candidate paths, if one path currently has no data request transmission while the other two paths have data request queues, Dijkstra's algorithm will mechanically determine the path without data requests as the optimal choice. This judgment method ignores the time-varying characteristics of network states, leading to an overly one-sided path selection decision that cannot accurately reflect the transmission performance in a real network environment. Therefore, for the shortest read / write latency path between each pair of storage nodes, it is necessary to integrate instantaneous read / write latency measurements with historical trend data to achieve continuous tracking and prediction of the path transmission status.

[0067] Specifically, in step S40, the shortest read / write latency path between every two storage nodes is calculated through Kalman filtering among all candidate paths, including the following steps:

[0068] Step S41: Construct an optimized network topology based on all alternative paths; combine every two storage nodes in the optimized network topology to obtain a start-destination node group; send read / write latency test data to each start-destination node group multiple times at fixed time intervals to obtain a basic read / write latency test data sequence set for each start-destination node group (e.g., ...). Figure 3 The intersections of the dashed lines and straight lines shown are the basic read / write latency test data. The intersections on each straight line constitute a basic read / write latency test data sequence set on the corresponding straight line.

[0069] It should be noted that in the above embodiments of this application, each alternative path is a low read / write latency path, that is, the load rate of its path nodes is lower than the load threshold. Therefore, the data packets (i.e., data requests) processed by each storage node and the data transmission between storage nodes are in a low-load state. Therefore, a network topology map is constructed for all storage nodes of all alternative paths. This network topology map is also in a low-load state. By reconstructing the network topology map and considering that the start-destination node group formed by each pair of storage nodes is also in a low-load state, the read / write latency between each pair of storage nodes is analyzed to determine the shortest read / write latency path between each pair of storage nodes. This provides a migration target path for the subsequent high-priority data request migration processing, facilitating the rapid transmission of high-priority data requests.

[0070] Step S42: Calculate the inter-path read / write latency fluctuation index based on the read / write latency data sequence of multiple storage paths between the same network start node and network destination node; calculate the initial read / write latency estimate and initial read / write latency change rate for each start-destination node group based on the basic read / write latency test data sequence set; set the state vector, error covariance matrix, process noise matrix, and measurement noise parameters of the Kalman filter in combination with the inter-path read / write latency fluctuation index;

[0071] It should be noted that in the above embodiments of this application, linear regression analysis is performed on the basic read / write latency test data sequence, and the trend of read / write latency over time is calculated using the least squares method. Specifically, the calculation involves dividing the covariance of the time series index and the read / write latency value by the variance of the time series index to obtain the initial read / write latency change rate, which reflects the direction and speed of change of path read / write latency. The state vector includes the initial read / write latency estimate and the initial read / write latency change rate. The process noise matrix is ​​adjusted based on the variance of the read / write latency sequence and the variance of the change rate sequence, with the adjustment factor being the inter-path read / write latency fluctuation index plus one. When the fluctuation index is high, the noise value is increased. The measurement noise parameter is adjusted based on the measurement equipment error variance, with the adjustment factor being the inter-path read / write latency fluctuation index plus one, reflecting the influence of network fluctuations on measurement accuracy. The error covariance matrix is ​​adjusted based on the initial measurement statistical variance and the trend estimation confidence, with the adjustment factor being the inter-path read / write latency fluctuation index plus one, representing the uncertainty of the initial estimate.

[0072] In the above embodiments of this application, by introducing the inter-path read / write latency fluctuation index, the performance comparison results of multiple storage paths between the same network start node and destination node are directly incorporated into the Kalman filter parameter settings. This multi-path fluctuation analysis can effectively capture the overall stability of the network. A high fluctuation index indicates a complex network state, requiring more conservative filter parameters; a low fluctuation index indicates a stable network state, allowing for a more aggressive tracking strategy.

[0073] Step S43: Apply a Kalman filter to each start-destination node group to predict the read / write delay state; perform a state prediction step based on the state vector and process noise matrix to obtain the prior state estimate and prior error covariance matrix; perform a measurement update step based on the measured noise parameters and actual read / write delay test data, calculate the Kalman gain in conjunction with real-time multipath fluctuations, and update the posterior state estimate and error covariance matrix.

[0074] It should be noted that in the embodiments of this application described above, the read / write latency and rate of change at the current moment are predicted based on the state estimation and state transition matrix of the previous moment. The state transition matrix reflects the cumulative effect of the rate of change on the read / write latency, that is, the current read / write latency equals the read / write latency of the previous moment plus the rate of change multiplied by the time interval. The error covariance matrix prediction propagates the uncertainty of the previous moment and adds process noise through the state transition matrix. The process noise includes multipath fluctuation information. Based on the measurement data of all paths in the same node group in the current period, the ratio of the standard deviation to the average value of the read / write latency is calculated to obtain the real-time inter-path fluctuation index. This index dynamically reflects the network path... The Kalman gain is determined by the ratio of the prediction error covariance matrix to the adjustment value of the measurement noise parameter. The adjustment factor of the measurement noise parameter is the inter-path read / write delay fluctuation index plus one. When the real-time inter-path fluctuation index increases, the effective measurement noise increases, the Kalman gain decreases, and the system trusts the prediction value more. When the prediction uncertainty increases, the Kalman gain increases, and the system trusts the measurement value more. The state update is achieved by adding the prediction value to the Kalman gain and multiplying it by the measurement residual. The measurement residual is the difference between the actual measurement value and the prediction value. The covariance update is achieved by subtracting the Kalman gain multiplied by the measurement matrix from the identity matrix and then multiplying it by the prediction error covariance matrix.

[0075] In the above embodiments of this application, by incorporating the real-time multipath fluctuation index into the Kalman gain calculation, the weight allocation between model prediction and measured data is dynamically adjusted. When multiple paths in the same node group perform consistently, the measured data is trusted more; when there are large differences between paths, the model prediction is relied upon more to avoid being affected by abnormal measured values.

[0076] Step S44: Perform multi-path dynamic comparison of the posterior state estimates of all paths in each origin-destination node group. Calculate the comprehensive score of each path based on the estimated read / write latency and read / write latency change rate in the posterior state estimates, combined with the error covariance matrix and the inter-path fluctuation index. Sort all paths in the same origin-destination node group based on the comprehensive score, and select the path with the lowest comprehensive score as the shortest read / write latency path.

[0077] It should be noted that in the above embodiments of this application, a confidence score is calculated based on the read / write delay estimation variance in the error covariance matrix. The confidence score is the reciprocal of the square root of the read / write delay estimation variance. The smaller the variance, the higher the confidence score, indicating that the state estimation is more reliable. A path stability score is calculated, which includes the combined effect of the absolute value of the read / write delay change rate and the real-time inter-path fluctuation index. The stability score is the absolute value of the change rate multiplied by one and the inter-path fluctuation index plus one. When the change rate is large and the inter-path fluctuation is large, the stability score is poor. The comprehensive score is calculated by weighting three factors: read / write delay, stability, and confidence. The read / write delay factor is the read / write delay estimate divided by the normalized baseline of fifty milliseconds. The stability factor is one minus the confidence score divided by the maximum confidence level of ten. The trend factor is the stability score divided by the normalized baseline of five multiplied by one and the inter-path fluctuation index plus one. All paths in the same node group are sorted in ascending order of comprehensive score, and the path with the smallest score is selected as the shortest read / write delay path.

[0078] In the above embodiments of this application, the multi-path dynamic comparison not only considers the performance indicators of a single path, but also incorporates the overall status of multiple paths in the same node group into the scoring system through the inter-path fluctuation index. This allows the server to consider the overall stability of the network when selecting a path. When there is a large fluctuation between paths, it tends to select the path with the relatively stable rather than the lowest absolute read / write latency, thereby improving the reliability of the decision.

[0079] Specifically, in step S42, the inter-path read / write latency fluctuation index is calculated based on the read / write latency data sequence of multiple storage paths between the same network starting node and network destination node, including the following steps:

[0080] Step S421: For all storage paths between the starting node and the destination node of the same network, obtain each storage path (the storage path here may be a segment of storage path or a complete storage path, such as...). Figure 4 As shown, there are two paths between the starting node A and the destination node B in the network: a storage path from storage node A to storage node B (storage path A→B) and a complete storage path from storage node A to storage node C, then from storage node C to storage node S, then from storage node S to storage node D, and finally from storage node S back to storage node B (storage path A→C→S→D→B). The complete read / write latency data sequence within the current statistical period is calculated for each storage path. The average read / write latency of each storage path is calculated to form a path performance baseline vector. The standard deviation of the read / write latency data for each storage path is calculated to form a path stability vector.

[0081] It should be noted that, as Figure 5As shown in the above embodiments of this application, the performance baseline vector of the storage path is calculated to reflect the basic performance level of each storage path in the e-commerce order processing scenario. For example, in the order query business, the performance baseline vector reflects the typical response speed of each path from the database node to the application server node. The stability vector of the path is calculated to reflect the performance fluctuation of each storage path in the order processing process. For example, in the payment business, the stability vector reflects the consistency of the response time of each storage path.

[0082] Step S422: Set the read / write latency stability level threshold (the read / write latency stability level threshold is set according to the service level requirements of e-commerce order business); based on the path stability vector and the read / write latency stability level threshold, divide each storage path into stability levels; count the distribution of the number of storage paths in each stability level;

[0083] It should be noted that in the above embodiments of this application, the stability level threshold is set based on the latency sensitivity of different order businesses. For example, the payment business requires a latency standard deviation of no more than 5 milliseconds, and the inventory query business can accept a latency standard deviation of no more than 20 milliseconds. By dividing the stability level, a set of stable paths suitable for high-priority order data is identified. For example, payment orders are preferentially assigned to the path with the highest stability level.

[0084] Step S423: Calculate the performance baseline dispersion of the path within each stability level, i.e., the standard deviation of the average read / write latency of each path within the same stability level; calculate the path coordination index of the performance of each storage path (obtained by the weighted average of the dispersion of each stability level).

[0085] It should be noted that in the above embodiments of this application, the calculation of the path coordination index of each storage path performance is used to evaluate the overall coordination degree of the storage path group. For example, during a promotional event, by analyzing the path performance coordination, it can be determined whether all storage paths can meet the processing requirements of sudden order traffic. The higher the coordination index, the smaller the performance difference between storage paths and the better the system load balancing effect.

[0086] Step S424: Calculate the inter-path read / write latency fluctuation index based on the stability level distribution and path coordination index;

[0087] It should be noted that in the above embodiments of this application, the inter-path read / write latency fluctuation index comprehensively reflects the overall stability of the storage path group. When the fluctuation index is high, it indicates that there is a large performance difference between paths, and the system needs to select the most stable path for high-priority order data. When the fluctuation index is low, it indicates that the path performance is uniform, and the system can adopt a more flexible load balancing strategy.

[0088] Specifically, in step S50, for each high-priority data request, a shortest read / write latency path is selected from the optimized path set as the target redirection path based on the particle swarm optimization algorithm combined with path state transient analysis, including the following steps:

[0089] Step S51: Randomly generate multiple redirection combinations based on all shortest read / write latency paths in all high-priority data requests and optimization path sets; obtain the particle position vector corresponding to each redirection combination; set the maximum number of iterations and the iteration counter, with the iteration counter initially set to 0; set the learning rate, search step size, and redirection combination search speed vector (the particle search speed vector represents the direction and speed of each redirection combination when searching for the optimal redirection combination).

[0090] It should be noted that the above redirection combination matches and combines high-priority data requests into a preliminary shortest read / write latency path. In other words, the above redirection combination is the first set of particles generated during the particle swarm optimization process. The position vector of the above redirection combination is a multi-dimensional vector, represented as X = (p1, p2, ... pm), where pm is an integer representing the index number of the shortest read / write latency path in the optimized path set to which the high-priority data request Fm is assigned. For example, if there are three high-priority data requests (F1, F2, F3) and the optimized path set has five shortest read / write latency paths, the particle (2, 5, 1) means that high-priority data request F1 matches the 2nd shortest read / write latency path, i.e., F1→2; high-priority data request F2 matches the 5th shortest read / write latency path, i.e., F2→5; and high-priority data request F3 matches the 1st shortest read / write latency path, i.e., F3→1.

[0091] Step S52: Calculate the remaining capacity data based on the current load data and capacity indicators of each shortest read / write latency path; and simultaneously obtain the instantaneous read / write latency value of each shortest read / write latency path; based on the remaining capacity data and instantaneous read / write latency value, obtain the read / write latency jitter trend coefficient of each shortest read / write latency path through time series-based fluctuation and read / write latency analysis.

[0092] Step S53: Calculate the overall fitness of each redirection combination based on the data volume of each high-priority data request in the redirection combination and the post-migration node load rate of each shortest read / write latency path; calculate the overall fitness of each redirection combination based on the post-migration node load rate and the read / write latency jitter trend coefficient of each shortest read / write latency path.

[0093] Step S54: Determine whether the overall fitness of each redirection combination is less than or equal to the preset fitness threshold. If yes, determine the corresponding redirection combination as the target redirection combination, and determine the shortest read / write latency path corresponding to each high-priority data request in the target redirection combination as the target redirection path. If no, increment the iteration count by 1 to obtain the current iteration count, and determine whether the current iteration count is greater than or equal to the maximum iteration count. If the current iteration count is greater than or equal to the maximum iteration count, output the redirection combination with the minimum overall fitness under the current iteration count as the target redirection combination, and determine the shortest read / write latency path corresponding to each high-priority data request in the target redirection combination as the target redirection path. If the current iteration count is less than the maximum iteration count, determine whether the difference between the minimum overall fitness under the current iteration count and the fitness threshold (i.e., subtracting the minimum overall fitness from the fitness threshold) is greater than or equal to the maximum iteration count. If the learning rate is equal to or greater than the preset fitness difference threshold, then update the redirection combination using the redirection combination search velocity vector to obtain a new redirection combination, and return the new redirection combination to step S53 for re-execution until the target redirection path is output. If the learning rate is equal to or greater than the preset fitness difference threshold, then update the search step size using the learning rate to obtain a new search step size. Update the redirection combination search velocity vector using the new search step size to obtain a new redirection combination search velocity vector. Update the particle position vector of each redirection combination using the new redirection combination search velocity vector to obtain a new particle position vector. Obtain a new redirection combination based on the new particle position vector. Return the new redirection combination to step S53, and return the new search step size and the new redirection combination velocity vector to step S51 for re-execution until the target redirection path is output.

[0094] Specifically, in step S52, based on the remaining capacity data and instantaneous read / write latency values, the read / write latency jitter trend coefficients of each shortest read / write latency path are obtained through time-series-based fluctuation and read / write latency analysis, including the following steps:

[0095] Step S521: Collect multiple historical read / write latency values ​​for each shortest read / write latency path within a preset time period prior to the current time point;

[0096] By collecting historical data, we can capture the short-term fluctuations and long-term trends of the shortest read / write latency path, avoid misjudgments based on instantaneous values, and ensure the reliability of subsequent analysis.

[0097] Step S522: Calculate the moving average read / write latency value at each time point within the time period using a preset sliding time window; calculate the read / write latency volatility within the sliding window by calculating the standard deviation of the moving average read / write latency value at each time point within the preset time period; calculate the average read / write latency volatility within the time period based on the read / write latency volatility.

[0098] It should be noted that the average read / write latency volatility reflects the dispersion of read / write latency of the shortest read / write latency path within a time period; the above embodiments of this application use a sliding time window to calculate the sliding average read / write latency value to eliminate short-term noise, and the read / write latency volatility measures the consistency of read / write latency of the shortest read / write latency path. A high read / write latency volatility reflects that the shortest read / write latency path has a high risk of read / write latency.

[0099] Step S523: Based on the time series of each time point within the time period, a moving average sequence is established for the moving average read / write latency values ​​at each time point within the time period; linear regression is performed on the moving average sequence to obtain the path read / write latency fluctuation line of the shortest read / write latency path; the slope of the path read / write latency fluctuation line is calculated using the least squares method.

[0100] It should be noted that the above embodiments of this application obtain a straight line of path read / write latency fluctuation through linear regression, capturing the main read / write latency fluctuation trend of the shortest read / write latency path.

[0101] Step S524: Calculate the read / write latency jitter trend coefficient of the shortest read / write latency path based on the average read / write latency volatility and slope. ;

[0102] The calculation method for the read / write latency jitter trend coefficient of the shortest read / write latency path is as follows:

[0103] ;

[0104] In the formula, The slope This represents the average read / write latency volatility. This is the trend threshold (i.e., the acceptable rate of increase in read / write latency). The benchmark volatility (i.e., the standard threshold for typical read / write latency fluctuations in a network under normal conditions, which is a fixed value and will not be elaborated further) It is the scaling factor;

[0105] It should be noted that, in the above embodiments of this application, the trend threshold... It is set based on business type information. For example, if video streaming requires stable read / write latency, then its trend threshold is lower. It is derived from the network baseline performance, ensuring volatility standardization, through Standardization trend risks, utilizing Standardize the volatility risk, then sum it and map it to the range (0, 1). The closer the value is to 1, the higher the risk. High slope and high average read / write latency volatility both indicate that the shortest read / write latency path is unstable and should be avoided when selecting a redirection path.

[0106] The above-described embodiments of this application first obtain multiple historical read / write latency values ​​within a preset time period to form a data sequence. In e-commerce order storage scenarios, for example, by collecting read / write latency data of the order database over the past 30 minutes, misjudgments caused by instantaneous network jitter or temporary load peaks are avoided, providing reliable data support for trend analysis. Then, a sliding time window is used to calculate the average read / write latency value and volatility. For example, in order query business, a 5-minute sliding window smooths out instantaneous peaks, accurately reflecting the continuous performance of the path. The average read / write latency volatility can identify storage paths that, although having a low average latency, experience drastic fluctuations. Path types may pose a timeout risk when processing payment orders. By performing linear regression on the smoothed latency sequence to obtain the slope of change, in the inventory query scenario, a positive slope indicates that the path performance is deteriorating. Even if the current latency is average, it is not suitable to allocate new order query requests, serving as an early warning. Furthermore, by combining volatility and trend, a standardized risk coefficient is output. For example, by setting a low trend threshold for payment business, it is ensured that payment orders are always allocated to the most stable path. When the coefficient is close to 1, it indicates that the storage path has a serious instability risk, and the system will automatically avoid such paths when allocating important order data.

[0107] In summary, the distributed storage processing method for large-scale e-commerce order data proposed in this invention calculates the load rate of each storage node by collecting data such as disk usage and CPU load, providing a basis for subsequent node classification; furthermore, the storage nodes are classified according to load level, distinguishing between high-load and low-load node groups, and the node groups are automatically divided through clustering algorithms, which facilitates targeted management.

[0108] Furthermore, when a high-load node reaches a threshold, available backup nodes are selected from the low-load group to ensure that the system can quickly find replacement nodes when some nodes are overloaded; a dynamic filtering algorithm is used to determine the path with the lowest read / write latency between nodes and establish an efficient data access route.

[0109] By identifying critical data requests and assigning them appropriate storage paths, we ensure that important requests receive priority access to high-quality resources. Finally, we migrate high-priority order data from overloaded nodes to optimized paths, achieving load balancing and optimized data access.

[0110] In the specific execution process, the preprocessing through Kalman filtering not only considers the performance indicators of a single path, but also incorporates the overall situation of multiple paths in the same node group into the scoring system through the inter-path fluctuation index, and comprehensively analyzes and selects the shortest delay path.

[0111] In addition, a stable target redirection path is selected through particle swarm optimization and analysis of time series fluctuations and read / write latency.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; those skilled in the art can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large-scale e-commerce order data distributed storage processing method, characterized in that, The following steps are included: The current load data and capacity indicators of each storage node are collected in real time by the monitoring sensors deployed on the storage nodes. The node load rate of each path is calculated based on the current load data and capacity indicators of each storage path. The path node load rate of each storage path is then calculated. Based on the load rate of the path nodes of each storage path, the K-Means clustering algorithm is applied to cluster all storage nodes to obtain high load rate group and low load rate group. When the node load rate of any storage node in the high load rate group exceeds the preset load threshold, multiple backup storage nodes are selected from the low load rate group. By using Kalman filtering, the shortest read / write latency path between every two alternative storage nodes is calculated among all candidate storage paths, and an optimized path set is established based on all the shortest read / write latency paths. Retrieve all data requests on storage paths where the load rate of path nodes in the high load rate group exceeds a preset load threshold; retrieve the data type information of each data request; The priority evaluation value of each data request is determined based on the data type information; data requests with priority evaluation values ​​greater than or equal to the preset priority threshold are marked as high-priority data requests; for each high-priority data request, the shortest read / write latency path is selected from the optimized path set as the target redirection path based on the particle swarm optimization algorithm combined with path state transient analysis. Migrate high-priority data requests in high-load groups to the corresponding target redirection storage path; The shortest read / write latency path between any two storage nodes is calculated using Kalman filtering, including the following steps: An optimized network topology is constructed based on all alternative paths; every two storage nodes in the optimized network topology are combined to obtain a start-destination node group; basic read / write latency test data sequence sets are collected at the same frequency for each start-destination node group; Calculate the inter-path read / write latency fluctuation index based on the read / write latency data sequence of multiple storage paths between the same network start node and network destination node; calculate the initial read / write latency estimate and initial read / write latency change rate of each start-destination node group based on the basic read / write latency test data sequence set; and set the state vector, error covariance matrix, process noise matrix and measurement noise parameters of the Kalman filter in combination with the inter-path read / write latency fluctuation index. For each start-destination node group, a Kalman filter is applied to predict the read / write delay state: the state prediction step is performed based on the state vector and the process noise matrix to obtain the prior state estimate and the prior error covariance; Then, based on the measured noise parameters and the actual read / write latency test data, a measurement update step is performed to update the posterior state estimate and error covariance; By combining the error covariance matrix, a multi-path dynamic comparison is performed on the read / write latency estimate and the read / write latency change rate in the updated posterior state estimation. The comprehensive score of each path is calculated, and the path with the lowest comprehensive score is selected as the shortest read / write latency path. The state vector consists of the initial read / write delay estimate and the initial read / write delay change rate; the process noise matrix is ​​set based on the read / write delay sequence variance and the change rate sequence variance, and the inter-path read / write delay fluctuation index is introduced as an adjustment factor; the measurement noise parameter is set based on the measurement equipment error variance, and the inter-path read / write delay fluctuation index is introduced as an adjustment factor; the error covariance matrix is ​​set based on the initial measurement statistical variance and the trend estimate confidence level, and the inter-path read / write delay fluctuation index is introduced as an adjustment factor. The inter-path read / write latency fluctuation index is calculated based on the read / write latency data sequence of multiple storage paths between the same network origin node and network destination node, including the following steps: For all storage paths between the same network start node and destination node, obtain the complete read / write latency data sequence for each storage path within the current statistical period; calculate the average read / write latency of each storage path to form a path performance baseline vector; Calculate the standard deviation of read / write latency data for each storage path to form a path stability vector; Set read / write latency stability level thresholds; divide each storage path into stability levels based on the path stability vector and the read / write latency stability level thresholds; and statistically analyze the distribution of the number of storage paths in each stability level. Calculate the performance baseline dispersion of the path within each stability level, that is, the standard deviation of the average read / write latency of each path within the same stability level; Calculate the path coordination metrics for the performance of each storage path; Based on the distribution of the number of storage paths and the path coordination index in the stability level, the read / write latency fluctuation index between paths is calculated.

2. The method of claim 1, wherein, For each high-priority data request, a shortest read / write latency path is selected from the optimized path set as the target redirection path based on the particle swarm optimization algorithm combined with path state transient analysis. The process includes the following steps: Randomly generate multiple redirection combinations based on all shortest read / write latency paths in the set of all high-priority data requests and optimized paths; obtain the particle position vector corresponding to each redirection combination; Set the maximum number of iterations and the iteration counter, with the iteration counter initially set to 0. Set the learning rate, search step size, and retargeting combined search speed vector; The remaining capacity data is calculated based on the current load data and capacity indicators of each shortest read / write latency path; and the instantaneous read / write latency value of each shortest read / write latency path is obtained at the same time; based on the remaining capacity data and instantaneous read / write latency value, the read / write latency jitter trend coefficient of each shortest read / write latency path is obtained through time series-based fluctuation and read / write latency analysis. The overall fitness of each redirection combination is calculated based on the data volume of each high-priority data request in the redirection combination and the post-migration node load rate of each shortest read / write latency path; the overall fitness of each redirection combination is calculated based on the post-migration node load rate and the read / write latency jitter trend coefficient of each shortest read / write latency path. The target redirection path is obtained by performing multi-level judgments based on the comprehensive fitness and preset fitness threshold, combined with the iteration count and maximum iteration count of the iteration counter, and iterative processing of the directional combined search velocity vector based on the learning rate and search step size. 3.The method of claim 2, wherein, Based on the comprehensive fitness and a preset fitness threshold, combined with the iteration count and maximum iteration count of the iteration counter, multi-level judgments are performed, and iterative processing of the directional combined search velocity vector is carried out based on the learning rate and search step size. The output is the target redirection path, including the following operation steps: Determine whether the overall fitness of each redirection combination is less than or equal to the preset fitness threshold. If so, determine the corresponding redirection combination as the target redirection combination, and determine the shortest read / write latency path corresponding to each high-priority data request in the target redirection combination as the target redirection path. If not, increment the iteration count by 1 to obtain the current iteration count. Determine if the current iteration count is greater than or equal to the maximum iteration count. If the current iteration count is greater than or equal to the maximum iteration count, output the redirection combination with the minimum overall fitness under the current iteration count as the target redirection combination. Determine the shortest read / write latency path corresponding to each high-priority data request in the target redirection combination as the target redirection path. If the current iteration number is less than the maximum iteration number, then determine whether the difference between the minimum comprehensive fitness and the fitness threshold under the current iteration number is greater than or equal to the preset fitness difference threshold; if so, then output the target redirection path based on the iterative processing of the directional combined search velocity vector according to the learning rate and search step size.

4. The method of claim 3, wherein, The target redirection path is obtained by iteratively processing the directional combined search velocity vector based on the learning rate and search step size, including the following steps: The search step size is updated using the learning rate to obtain a new search step size; the redirection combination search velocity vector is updated using the new search step size to obtain a new redirection combination search velocity vector; and the particle position vector of each redirection combination is updated using the new redirection combination search velocity vector to obtain a new particle position vector. A new redirection combination is obtained based on the new particle position vector; the new redirection combination is returned to the above steps, and the new search step size and the new redirection combination search velocity vector are returned to the above steps for re-execution until the target redirection path is obtained as output.

5. The method of claim 4, wherein, Based on the remaining capacity data and instantaneous read / write latency values, and through time-series-based fluctuation and read / write latency analysis, the read / write latency jitter trend coefficients for each shortest read / write latency path are obtained, including the following steps: For each shortest read / write latency path, collect multiple historical read / write latency values ​​within a preset time period prior to the current time point; The sliding average read / write latency value at each time point within the time period is calculated using a preset sliding time window; the read / write latency volatility within the sliding window is calculated by calculating the standard deviation of the sliding average read / write latency value at each time point within the preset time period; and the average read / write latency volatility within the time period is calculated based on the read / write latency volatility. The moving average read / write latency values ​​at each time point within the time period are used to establish a moving average sequence based on the time series of each time point within the time period. Linear regression is performed on the moving average sequence to obtain the path read / write latency fluctuation line of the shortest read / write latency path. The slope of the path read / write latency fluctuation line is calculated using the least squares method. The read / write latency jitter trend coefficient of the shortest read / write latency path is calculated based on the average read / write latency volatility and slope.

6. The method of claim 5, wherein, The calculation method for the read / write latency jitter trend coefficient of the shortest read / write latency path is as follows: add the ratio of the slope of the path read / write latency fluctuation line to the trend threshold to the ratio of the average read / write latency volatility to the benchmark volatility, multiply the sum by the scaling factor, and then input it into the S-shaped function for normalization mapping, finally outputting an evaluation value between 0 and 1.

7. A storage medium, characterized by The storage medium stores a computer program, which, when executed by a processor, implements the steps of the distributed storage and processing method for large-scale e-commerce order data as described in any one of claims 1-6.

Images