Separated memory pooling method and system based on adaptive redundancy strategy

Abstract

The invention relates to the field of distributed storage, and particularly discloses a separated memory pooling method based on a self-adaptive redundancy strategy, which comprises the following steps: when a central coordinator receives a memory application request of a client, analyzing a delay constraint parameter and a cost constraint parameter contained in the request, dynamically selecting an optimal redundancy strategy through a multi-dimensional quantitative scoring model in combination with a node health score calculated based on a dynamic weighting factor, a fault probability predicted value, a network topology and a bandwidth utilization rate, and generating a composite global memory handle; and when the node fails or the strategy is switched, the coordinator pushes the new storage position and the strategy information to the client in real time through a handle hot update mechanism. According to the method, the data availability and the access continuity of the separated memory pool system in a multi-type fault scene can be remarkably improved, and meanwhile, dynamic balance among performance, cost and reliability is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage, in particular to a separated memory pooling method based on an adaptive redundancy strategy, mainly applied to large-scale data centers to improve memory resource utilization, system expansion flexibility and fault tolerance capability. BACKGROUND

[0002] With the rapid development of cloud computing, big data and artificial intelligence technologies, distributed storage systems play a crucial role in modern data centers. Especially in memory computing, real-time analysis and other scenarios, efficient management of memory resources and high availability of data are required. However, existing distributed memory management technologies still face many challenges and limitations.

[0003] Firstly, the current mainstream distributed storage systems usually adopt fixed data redundancy strategies, such as multi-copy replication (Replication) or erasure coding (Erasure Coding). The multi-copy strategy stores multiple copies of data on different nodes to ensure reliability, although it provides low-latency data access, but the storage overhead is huge, resulting in low resource utilization, and the erasure coding technology can significantly reduce the storage overhead, but the calculation overhead and access delay introduced by the coding and decoding process are high, which is difficult to meet the real-time requirement of application scenarios. More importantly, these fixed strategies lack flexibility and cannot be dynamically adjusted according to different business needs. Secondly, in traditional distributed memory systems, clients access remote memory data through memory handles (Memory Handle), which are usually statically allocated and remain unchanged once established. When the underlying data needs to be changed due to node failure, load balancing or data migration, the client must reacquire a new handle, which often requires interrupting the current session, resulting in business suspension, I / O request loss or access delay significantly increased. With the expansion of system size and the increase of business complexity, node failure becomes the norm rather than the exception. The static handle management method is difficult to cope with frequent node state changes, which seriously affects the availability and access continuity of the system.

[0004] In summary, the existing distributed memory management technology has obvious shortcomings in multi-dimensional demand balance, fault recovery efficiency, resource utilization and system scalability, and an innovative solution is urgently needed to cope with increasingly complex application scenarios and strict business requirements. SUMMARY

[0005] The purpose of the present application is to solve the problems of low memory utilization, poor scalability and poor fault tolerance in the prior art, and to propose a separated memory pooling method based on an adaptive redundancy strategy, which aims to solve the limitations of the prior art through an adaptive redundancy strategy selection and a complex handle hot update mechanism, and to improve the overall performance of the distributed memory system.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme: A separated memory pooling method based on an adaptive redundancy strategy, comprising: S1: receiving and analyzing client requests: the central coordinator receives the memory application request sent by the client, the request at least containing multi-dimensional operation requirement parameters such as acceptable delay threshold, cost budget and data priority, and analyzes the parameters to extract the constraint conditions and priority information required for strategy calculation, so as to obtain the request analysis result as the input basis for subsequent redundancy strategy calculation and resource allocation; S2: collecting global running state information: through the central coordinator, real-time running state is obtained from multiple distributed memory nodes, and topological information such as rack position and data center identifier of the nodes is obtained, so as to obtain the global running state information, which provides a comprehensive environment perception basis for redundancy strategy selection; S3: executing adaptive redundancy strategy selection: based on the request analysis result and the global running state information, the redundancy strategy selection engine is called to calculate the optimal redundancy strategy; S4: generating a complex global memory handle: according to the optimal redundancy strategy, a complex global memory handle with a clear binary structure is generated, which contains the following fields: redundancy strategy identifier, detailed data block position list, version number, verification and encryption information, and hot update flag bit; S5: real-time detection and handle hot update: the running state of the distributed memory nodes is continuously monitored, and when node failure, network anomaly or redundancy strategy switching is detected, the central coordinator generates a new complex global memory handle, and pushes the new handle to the client through the hot update channel without closing the client session, so as to realize seamless switching of the data access path, improve the continuity and stability of the system; S6: access and data recovery: the client directly accesses the relevant memory nodes for read-write operation according to the latest complex global memory handle; during the node recovery or data migration process, the recovered data blocks are immediately available, and the unrecovered part is decoded through the optimal redundancy strategy (specifically, error correction code) to maintain business continuity, minimize business interruption and improve system fault tolerance.

[0007] As a further technical solution of the present application, in the S2, the real-time running state includes CPU utilization, memory usage, I / O load, node health score, failure probability prediction value, network path delay and bandwidth utilization, etc., wherein the node health score and the failure probability prediction value are calculated by modeling the real-time and historical performance data of the node.

[0008] As a further technical solution of the present application, the node health score is calculated by a weighted linear model, and the specific formula is: , wherein: , , represent the normalized utilization rate (0 to 1) of the CPU, memory and I / O of the node , , , are the corresponding weight coefficients, and ; The value range of is , and the higher the score is, the healthier the node is.

[0009] As a further technical solution of the present application, the failure probability prediction value is predicted based on the historical health score time series by using an LSTM model to evaluate the possibility of node failure in a future time window.

[0010] As a further technical solution of the present application, in the S3, the optimal redundancy strategy at least includes multiple copy mirroring, erasure code, differential incremental redundancy or a combination thereof, and the delay constraint, cost constraint, fault domain isolation requirement and data access mode are comprehensively considered in the strategy selection process to ensure that the redundancy scheme is both efficient and reliable.

[0011] As a further technical solution of the present application, in the S3, the redundancy strategy selection engine adopts a multi-dimensional weighted scoring-based decision model when performing strategy calculation, wherein the decision model takes the delay constraint, cost constraint, node health score and data priority as input factors, and adjusts the weight coefficients of each factor dynamically to adapt to different business scenarios (i.e. the decision model quantitatively evaluates the matching degree score of each redundancy strategy under the current constraints and system state, combines the client requirements analyzed in step S1 and the running state data collected in step S2, and calculates the optimal redundancy strategy), so as to realize the balance between real-time performance and resource utilization of the redundancy strategy.

[0012] As a further technical solution of the present invention, in S3, the redundancy strategy selection engine supports the combined use of multi-replica redundancy and erasure coding redundancy, and adopts a low-latency multi-replica strategy (switching from erasure coding to multi-replica) in the part of data with high data popularity (exceeding the preset threshold), and adopts an erasure coding strategy with lower storage overhead in the part of data with low data popularity, so as to significantly reduce the overall storage cost while meeting the latency requirements.

[0013] As a further technical solution of the present invention, in S4, the data block location list includes node ID, physical location, access endpoint, bandwidth, fault domain identifier, and version number increments when the redundancy strategy or data block location changes. The composite global memory handle is used to guide the client to directly access the corresponding memory node through Remote Direct Memory Access (RDMA) technology to achieve accurate data location and secure access.

[0014] As a further technical solution of the present invention, in S4, the composite global memory handle further includes a handle validity period field and a handle signature field. The validity period field is used to automatically trigger the hot update process before the handle expires, ensuring timely updates of the access path. The signature field is generated using the HMAC-SHA256 algorithm and is used by the client to verify the integrity and source credibility of the handle before use, thereby improving system security.

[0015] A split memory pooling system based on an adaptive redundancy strategy is provided to implement a split memory pooling method based on an adaptive redundancy strategy. The system includes a client layer, a central coordinator layer, and a distributed memory node layer. The central coordinator layer integrates a monitoring module, a redundancy strategy selection engine, a handle generator, and a hot update push module. It is used to dynamically select a redundancy strategy according to business needs and issue composite global memory handles to the client to achieve hot updates of handles and uninterrupted business operations. The client obtains the optimal access path through the central coordinator layer and directly accesses multiple distributed memory nodes in the distributed memory node layer to support high availability and low latency memory data access.

[0016] The beneficial effects of this invention are as follows: 1. Significantly improve system availability and fault recovery capabilities: Through the composite handle hot update mechanism, millisecond-level access path switching is achieved in the event of node failure, ensuring business continuity; combined with intelligent redundancy strategy and erasure coding technology, data is quickly recovered, significantly reducing service interruption time.

[0017] 2. Achieve intelligent dynamic balance between performance and cost: Based on multi-dimensional constraints, adaptively select the optimal redundancy strategy, break through the limitations of fixed strategies, and reduce storage costs by more than 35% and optimize access latency by about 40% while ensuring reliability.

[0018] 3. Greatly improve resource utilization and system scalability: adopt data heat driven hybrid redundancy mechanism, improve storage efficiency by about 30%; separated pool architecture supports elastic expansion of memory nodes, meets large-scale application demand.

[0019] 4. Reduce operation and maintenance complexity and enhance security guarantee: automatic policy selection and fault handling reduce manual intervention; handle signature verification and integrity check to improve security, realize efficient and reliable operation and maintenance in large-scale distributed environment. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A flowchart of a separated memory pooling method based on an adaptive redundancy strategy is proposed for the present application. Figure 2 A system architecture diagram of a separated memory pooling system based on an adaptive redundancy strategy is proposed for the present application. DETAILED DESCRIPTION

[0021] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the present application will be further described below in combination with specific embodiments.

[0022] Please refer to the accompanying Figure 1 A separated memory pooling method based on an adaptive redundancy strategy, comprising: S1: receiving and analyzing client request: the central coordinator receives the memory application request sent by the client (in JSON format), which at least contains the acceptable delay threshold (unit: ms), cost budget (unit: yuan / GB), data priority (such as high, medium and low, which can be mapped to numerical values 3, 2 and 1), and other multi-dimensional operation demand parameters, and analyzes the parameters to extract the constraint conditions and priority information required for strategy calculation, so as to obtain the request analysis result as the input basis for subsequent redundancy strategy calculation and resource allocation, which specifically includes: The central coordinator parses the request through the FastAPI (https: / / fastapi.tiangolo.com) server side to extract the constraint conditions of policy selection; for example, the client can specify that the delay does not exceed 10 ms, the budget cost is 0.5 yuan per GB of memory, and the data priority is "high"; the analysis module will convert these information into constraint conditions and pass them to the subsequent policy selection module.

[0023] S2: Collecting global running status information: Collecting real-time running status from multiple distributed memory nodes through the central coordinator, including CPU utilization, memory usage, I / O load, node health score, failure probability prediction value, network path delay and bandwidth utilization, etc., as well as the rack position of the node, data center identifier and other topology information, so as to obtain the global running status information, which provides a comprehensive environment perception basis for the selection of redundancy strategy, including: using Prometheus (https: / / prometheus.io / ) to collect system running related performance indicators (CPU utilization, memory usage, I / O load), combining a simple topology management database to maintain node rack position and data center identifier; network delay and bandwidth information is obtained by periodic detection or node self-reporting; among them, the node health score and failure probability prediction value are calculated by modeling the real-time and historical performance data of the node, that is, for each memory node Calculate real-time node health score and failure probability prediction value , specifically: the node health score is calculated by a weighted linear model, and the specific formula is: , wherein: 、 、 represent the normalized utilization of CPU, memory and I / O of the node , the CPU utilization normalization is: , wherein is the actual CPU utilization (percentage); the memory usage normalization is: , wherein is the actual memory usage, is the total memory capacity; the I / O load normalization is: , wherein is the actual I / O load, is the maximum processing capacity of I / O; 、 、 are the corresponding weight coefficients, and ; The value range of , the higher the score, the healthier the node; the failure probability prediction value Based on the historical health score time series, an LSTM model is used for prediction to evaluate the possibility of node failure in a future time window, specifically including: first, define the time window length T, and take the health score data of the past T time points as the input sequence; then build an LSTM network, including an input layer, an LSTM layer and an output layer; the LSTM layer controls the information flow through the forget gate, the input gate and the output gate, and captures the dynamic characteristics of the time series; finally, the output layer outputs a probability value representing the possibility of failure in the future time window; when training the LSTM model, a binary cross-entropy loss function is used to quantify the error between the predicted value and the true value.

[0024] S3: Perform adaptive redundancy strategy selection: based on the request analysis result and the global running state information, call the redundancy strategy selection engine to calculate the optimal redundancy strategy (including at least multiple copies of mirrors, erasure code, differential incremental redundancy or their combination), and comprehensively consider the delay constraint, the cost constraint, the fault domain isolation requirement and the data access mode in the strategy selection process, to ensure that the redundancy scheme is both efficient and reliable; the redundancy strategy selection engine adopts a decision model based on multi-dimensional weighted scoring when performing strategy calculation, and takes the delay constraint, the cost constraint, the node health score and the data priority as input factors, and adjusts the weight coefficients of each factor dynamically to adapt to different business scenarios (i.e. the decision model quantitatively evaluates the matching degree score of each redundancy strategy under the current constraints and system state, combines the client requirements analyzed in step S1 and the running state data collected in step S2, and calculates the optimal redundancy strategy), so as to realize the balance between real-time performance and resource utilization of the redundancy strategy; the redundancy strategy selection engine supports the combination of multiple copy redundancy and erasure code redundancy, and uses a low-delay multiple copy strategy (switches from erasure code to multiple copy) in the part with high data heat (exceeding a preset threshold), and uses an erasure code strategy with lower storage overhead in the part with low data heat, so as to significantly reduce the overall storage cost while meeting the delay requirement; S3 specifically includes: for a data block that needs to allocate memory, its data heat is calculated by the following formula , wherein: is the heat value at the current time, is the access frequency in the current statistical period, is the heat value at the last time, is a smoothing factor (for example, 0.3), and the heat value is used for decision-making in the hybrid strategy; The redundancy strategy selection engine scores each candidate strategy (such as three copies and erasure code), and for a specific redundancy strategy , the comprehensive score is calculated as follows: ​ , wherein: , , are the weights of performance, cost and reliability respectively, determined dynamically by data priority , for example, high priority data is given and higher values; and are the estimated delay and cost of the strategy respectively; is a normalization function representing the degree of satisfaction of the constraints; is the average health score of the nodes selected for this strategy; represents the stability of the least reliable node in the selected node group; The engine calculates the of all available strategies and selects the strategy with the highest score as the optimal redundancy strategy.

[0025] S4: Generate a composite global memory handle: according to the optimal redundancy strategy, generate a composite global memory handle with a clear binary structure, which contains the following fields: redundancy strategy identifier, detailed data block location list (including node ID, physical location, access endpoint, bandwidth, fault domain identifier), version number (incremented when the redundancy strategy or data block location changes), verification and encryption information, hot update flag (used to identify whether the client supports the hot update function of the handle), valid period field and handle signature field, the valid period field is used to automatically trigger the hot update process before the handle expires, to ensure timely update of the access path, the signature field is generated using the HMAC-SHA256 algorithm, used by the client to verify the integrity and source credibility of the handle before use, to improve system security; Specifically including: The composite global memory handle is used to guide the client to directly access the corresponding memory node through the Remote Direct Memory Access technology (RDMA), to realize accurate data positioning and secure access; The generated composite global memory handle is serialized in Protocol Buffers (https: / / protobuf.dev / ) format; The binary structure of the handle used is shown in Table 1 below: Table 1: Binary structure of composite global memory handle

[0026] wherein, BlockInfo is a nested structure containing the fields shown in Table 2 below: Table 2: BlockInfo fields

[0027] S5: Real-time Detection and Handle Hot Update: Continuously monitor the running status of the distributed memory nodes. When a node failure, network anomaly, or redundancy strategy switching is detected, the central coordinator generates a new composite global memory handle and pushes it to the client without closing the client session using a lightweight message push channel based on an event-driven asynchronous communication mechanism, such as a bidirectional stream based on gRPC (https: / / grpc.io). This achieves seamless switching of data access paths and improves system continuity and stability. Unfinished I / O requests are stored in a local queue and rescheduled for execution after switching, ensuring that requests are not interrupted. S6: Access and Data Recovery: The client directly accesses the relevant memory nodes for read and write operations based on the latest composite global memory handle. During node recovery or data migration, recovered data blocks are immediately available, while unrecovered portions are decoded using the optimal redundancy strategy (specific erasure coding) to maintain business continuity, minimize business interruption, and improve system fault tolerance. Specifically, the client uses Remote Memory Access (RDMA) technology (https: / / github.com / linux-rdma / rdma-core) to directly access memory nodes based on the latest composite handle, enabling high-speed data read and write. During node recovery, recovered data blocks are immediately accessible, while unrecovered data is asynchronously reconstructed through the erasure coding module. A segmented identification mechanism is used to manage data recovery status, supporting gradual business recovery and reducing service interruption risks. The specific steps for erasure coding decoding to recover data are as follows: Fault detection and information collection: A client fails to access a data block, or the central coordinator detects a node failure via heartbeat. When offline, the coordinator determines the data blocks stored on that node. Lost; Location recovery set: Based on the information in the handle, the coordinator identifies the set related to... All other data blocks and check blocks belonging to the same band; for example, for Encoding requires finding at least One surviving block (data block or check block); Constructing the decoding matrix: Assuming the original data block is The verification block is Generate matrix is ​​used during encoding. (Using a Cauchy matrix), making When a block (such as After being lost, from Select with The rows corresponding to each surviving block form a new... matrix ; Solving linear equations: let the survived blocks form a vector , and the original data block vector is , then ; by solving the linear equations, i.e. calculating , all original data blocks, including the missing , can be recovered; this calculation is usually performed on a finite field; Data restoration and handle update: the recovered data blocks are written to a new healthy node, and a new composite handle is generated and sent to the client.

[0028] Please refer to the attached Figure 2 A separated memory pooling system based on adaptive redundancy strategy is used to realize a separated memory pooling method based on adaptive redundancy strategy, which comprises a client layer, a central coordinator layer and a distributed memory node layer; the central coordinator layer integrates a monitoring module, a redundancy strategy selection engine, a handle generator and a hot update pushing module, which is used to dynamically select a redundancy strategy according to business requirements and send a composite global memory handle to the client, so as to realize the hot update of the handle and the uninterrupted business; The client obtains the optimal access path through the central coordinator layer and directly accesses a plurality of distributed memory nodes in the distributed memory node layer, so as to support high-availability and low-latency memory data access; through the above-mentioned accurate decoding steps, the accuracy and efficiency of data recovery in the case of partial node failure are ensured. Through the above-mentioned steps, an intelligent and reliable separated memory pool management scheme is realized. The scheme not only improves the redundancy strategy selection efficiency of the system, but also guarantees the continuity of data access and the high availability of the system, and is suitable for cloud computing and large-scale distributed memory management scenarios.

[0029] Embodiment 1 for verifying the technical advantages and application value of the separated memory pooling method based on adaptive redundancy strategy provided by the present application.

[0030] I. Verification environment building 1.1 Hardware and software configuration A test cluster containing 4 nodes, including 1 central coordinator node and 3 distributed memory nodes, is built in this verification example. Each node is equipped with an 8-core Intel Xeon processor, 32GB DDR4 memory, 10Gbps Ethernet interface and 512GB NVMe SSD storage. The software environment uses Ubuntu 20.04 LTS operating system, the monitoring tool uses Prometheus 2.40 combined with Node Exporter 1.6, the coordinator is implemented based on FastAPI and gRPC framework, and the data storage uses Redis cluster to simulate distributed memory environment.

[0031] 1.2 Test data preparation To comprehensively verify the system function, two typical business load scenarios are designed: the high performance scenario requires a delay of no more than 10 milliseconds, the cost budget is 0.8 yuan / GB, and the data priority is high; the low cost scenario allows a delay of up to 50 milliseconds, the cost budget is strictly controlled at 0.3 yuan / GB, and the data priority is low. These two scenarios simulate the actual needs of real-time analysis business and archival storage business respectively.

[0032] II. Core function verification 2.1 Verification of node health score calculation By simulating the running indicators of the three memory nodes under different load states, the accuracy of the health score calculation model is verified. Node 1 is in a light load state, with CPU utilization of 25%, memory utilization of 40%, and I / O load of 15%; Node 2 is in a medium load state, with utilization rates of 65%, 70% and 45% respectively; Node 3 is in a heavy load state, with utilization rates reaching 90%, 85% and 75% respectively.

[0033] According to the weighted linear model calculation, the health score of node 1 is 0.795, that of node 2 is 0.535, and that of node 3 is 0.235. The actual system calculation result shows that the health scores of the three nodes are completely consistent with the expected values, with an error of less than 0.01, proving that the health score model can accurately reflect the actual running state of the nodes.

[0034] 2.2 Verification of redundancy strategy selection engine For different business needs, the intelligent decision-making ability of the strategy selection engine is verified. In the high performance scenario test, the system correctly selects the multi-replica redundancy strategy to meet the low delay requirement, with an estimated delay of 8.2 milliseconds and a cost of 0.75 yuan / GB. In the low cost scenario test, the system selects the erasure code strategy to prioritize storage efficiency, with an estimated delay of 42.1 milliseconds and a cost of 0.28 yuan / GB. In the balanced scenario test, the system adopts a hybrid strategy, achieving a good balance between a delay of 23.5 milliseconds and a cost of 0.45 yuan / GB.

[0035] All test cases show that the policy selection engine can intelligently select the optimal redundancy strategy according to business constraints, and the decision result meets expectations.

[0036] 2.3 Compound handle generation and verification The complete generation process of the compound global memory handle was verified. The system successfully generated a handle structure containing version number, policy identifier, data block location list, check block information, valid period timestamp, and digital signature, etc. The initial handle version number is 1, using RS(6,3) erasure code strategy, containing 2 data blocks and 1 check block, and the valid period is 24 hours.

[0037] The handle signature is generated using the HMAC-SHA256 algorithm, and the verification pass rate is 100%. Serialization and deserialization tests show that the handle maintains integrity and consistency during binary format conversion, laying a solid foundation for the subsequent hot update mechanism.

[0038] 2.4 Hot update mechanism verification Simulate node failure scenarios to verify the effectiveness of the handle hot update mechanism. When node 2 is detected to have failed, the system completes the generation and pushing of the new handle within 156 milliseconds. The updated handle version number is incremented to 2, and all 3 affected client sessions achieve seamless switching without business interruption or data loss.

[0039] During the hot update process, unfinished I / O requests are temporarily stored in the local queue, and are rescheduled for execution after the handle is updated, ensuring the continuity of request processing. This mechanism significantly improves the availability of the system in failure scenarios.

[0040] 2.5 Data recovery function verification Test the erasure code data recovery capability, simulate the scenario of 2 data block loss in RS(6,3) encoding environment. The system successfully recovers all 6 original data blocks from the remaining 5 surviving blocks (including data blocks and check blocks), with a recovery completeness rate of 100%.

[0041] The data recovery process takes 45 milliseconds, and the recovery throughput reaches 142.67 MB / s. The recovered data is verified to be completely consistent with the original data, proving the effective data protection capability of the erasure code mechanism in node failure.

[0042] Three, performance and effect verification 3.1 System performance benchmark test The performance benchmark test is conducted on the key operations of the system. The average delay of the strategy selection operation is 12.5 milliseconds, and the 95% quantile value is 18.3 milliseconds, indicating that the engine can quickly respond to the strategy calculation demand. The average delay of the handle generation operation is only 3.2 milliseconds, and the 95% quantile value is 4.8 milliseconds, meeting the performance requirements in high-concurrency scenarios.

[0043] The average delay of the hot update operation is 145.6 milliseconds, and the 95% quantile value is 203.2 milliseconds, ensuring business continuity while achieving fast fault recovery. All performance indicators meet or exceed the design expectations.

[0044] 3.2 Resource utilization optimization effect Through adaptive redundancy strategy selection, the system significantly optimizes resource utilization while ensuring data reliability. Compared with the fixed multiple copy strategy, the hybrid redundancy strategy saves about 35% of storage space; compared with the pure erasure code strategy, the access delay is reduced by about 40%. This dynamic balancing mechanism enables the system to intelligently allocate resources according to different business needs.

[0045] Four, verification conclusion This verification example comprehensively tests the functions of the separation memory pooling method based on adaptive redundancy strategy selection and composite handle hot update. The results show that the system performs well in node health assessment, redundancy strategy selection, handle management, hot update, and data recovery, etc., and achieves the design goal.

[0046] 4.1 Technical advantage verification Compared with the traditional fixed redundancy strategy system, this method has significant improvements in data availability, access continuity, and resource utilization. Especially in the multi-node failure scenario, the system can quickly switch the access path through the hot update mechanism, combined with the erasure code data recovery capability, to ensure the high availability of the business.

[0047] 4.2 Actual application value This method is suitable for memory-intensive application scenarios such as cloud computing and big data analysis that require high reliability and low latency. Through intelligent resource management and fault tolerance mechanisms, it can provide stable and efficient memory services for enterprise-level applications, reduce operation and maintenance costs, and improve business continuity.

[0048] The verification results fully prove the advancement, practicality, and reliability of the invention technology, providing an effective solution for large-scale distributed memory management.

[0049] From the above description, it can be seen that the above-mentioned embodiments of the present application realize the following technical effects: significantly improving system availability and fault recovery capability: through the composite handle hot update mechanism, millisecond-level access path switching is realized when the node fails, and business continuity is guaranteed; combined with intelligent redundancy strategy and erasure code technology, data is quickly recovered, and service interruption time is greatly reduced.

[0050] Implementing intelligent dynamic balance of performance and cost: based on multi-dimensional constraints (delay, cost, priority) to adaptively select the optimal redundancy strategy, breaking through the limitations of fixed strategies, while ensuring reliability, the storage cost is reduced by more than 35%, and the access delay is optimized by about 40%.

[0051] Greatly improve resource utilization and system scalability: using a hybrid redundancy mechanism driven by data hotness (hot data multiple copies, cold data erasure code), improve storage efficiency by about 30%; the separation type pooling architecture supports elastic expansion of memory nodes, and meets the demand of large-scale applications.

[0052] Reduce the complexity of operation and maintenance and enhance security: automated strategy selection and fault handling reduce human intervention; handle signature verification and integrity check to improve security, and realize efficient and reliable operation and maintenance in a large-scale distributed environment.

[0053] Those skilled in the art should understand that the above discussion of any embodiment is only exemplary and is not intended to imply that the scope of the present application is limited to these examples; under the idea of the present application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the present application as described above. In order to be brief, they are not provided in details.

[0054] The present application is intended to cover all such alternatives, modifications and variations as fall within the broad scope of the specification. Therefore, any omission, modification, equivalent replacement, improvement, etc. made in the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for adaptive redundancy based segregated memory pooling, comprising: Comprise: S1: receiving and analyzing client request: the central coordinator receives the memory application request sent by the client, the request at least contains multi-dimensional running demand parameters, and the parameters are analyzed to extract the constraint conditions and priority information required for strategy calculation, so as to obtain request analysis result; S2: collecting global running state information: through the central coordinator, real-time running state is obtained from multiple distributed memory nodes, and the topology information of the nodes is obtained, so as to obtain global running state information; S3: executing adaptive redundancy strategy selection: based on the request analysis result and the global running state information, the redundancy strategy selection engine is called to calculate the optimal redundancy strategy; S4: generating composite global memory handle: according to the optimal redundancy strategy, a composite global memory handle with clear binary structure is generated, which contains the following fields: redundancy strategy identifier, data block position list, version number, verification and encryption information, hot update flag; S5: real-time detection and handle hot update: the running state of the distributed memory node is continuously monitored, when the node failure, network exception or redundancy strategy switching condition is detected, the central coordinator generates a new composite global memory handle, and pushes the new handle to the client through the hot update channel without closing the client session; S6: access and data recovery: the client directly accesses the related memory node for read-write operation according to the latest composite global memory handle; during the node recovery or data migration process, the recovered data block is immediately available, and the unrecovered part is decoded through the optimal redundancy strategy.

2. The method of claim 1, wherein the method is based on an adaptive redundancy strategy. In S2, the real-time running state includes CPU utilization, memory usage, I / O load, node health score, failure probability prediction value, network path delay and bandwidth utilization, wherein the node health score and failure probability prediction value are calculated by modeling the real-time and historical performance data of the node.

3. The method of claim 2, wherein the method is based on an adaptive redundancy strategy. The node health score The node health score is calculated by a weighted linear model, specifically: ， wherein: , , represent the normalized utilization of CPU, memory and I / O of the node , , , are the corresponding weight coefficients, and .

4. The method of claim 3, wherein the method is characterized by, The failure probability prediction value Based on the historical health score time series, an LSTM model is used for prediction to assess the likelihood of a node failing within a future time window.

5. The method of claim 1, wherein the method is based on an adaptive redundancy strategy. In S3, the optimal redundancy strategy at least includes multi-copy mirroring, erasure code, differential incremental redundancy or combination thereof, and the delay constraint, cost constraint, fault domain isolation requirement and data access mode are considered in the strategy selection process.

6. The method of claim 5, wherein the method is characterized by, In S3, the redundancy strategy selection engine adopts a multi-dimensional weighted scoring based decision model when executing strategy calculation, which takes delay constraint, cost constraint, node health score and data priority as input factors, and adjusts the weight coefficient of each factor to adapt to different business scenarios.

7. The method of claim 6, wherein the method further comprises: In S3, the redundancy strategy selection engine supports the combination of multi-copy redundancy and erasure code redundancy, and adopts low-delay multi-copy strategy in the part with high data heat, and adopts erasure code strategy with lower storage overhead in the part with low data heat.

8. The method of claim 1, wherein the method is based on an adaptive redundancy strategy. In S4, the data block position list contains node ID, physical location, access endpoint, bandwidth and fault domain identifier, the version number is incremented when the redundancy strategy or data block position changes, and the composite global memory handle is used to guide the client to directly access the corresponding memory node through remote direct memory access technology.

9. The method of claim 8, wherein the method further comprises: In the S4, the composite global memory handle further comprises a handle validity period field and a handle signature field, the validity period field is used to automatically trigger a hot update process before the handle expires, and the signature field is generated by using an HMAC-SHA256 algorithm and is used for the client to verify the integrity and source credibility of the handle before use.

10. A split memory pooling system based on adaptive redundancy strategy, characterized in that, A separation memory pooling method based on an adaptive redundancy strategy, for realizing the method of claims 1-9, comprising a client layer, a central coordinator layer, and a distributed memory node layer; the central coordinator layer integrates a monitoring module, a redundancy strategy selection engine, a handle generator, and a hot update pushing module, and is used to dynamically select a redundancy strategy according to a business requirement and issue a composite global memory handle to a client; The client obtains an optimal access path through the central coordinator layer and directly accesses a plurality of distributed memory nodes in the distributed memory node layer.

Images