Memory management system, method and device, medium, equipment and program product

Abstract

The invention discloses a memory management system, method and device, a medium, equipment and a program product, and relates to the technical field of computers. A data processing module is connected with a target computing node and a storage module, so that corresponding data processing modes in the storage module can be called according to different data processing requests; and dynamically managing the memory nodes in the memory pool based on the data processing mode. The limitation that only capacity expansion can be achieved in the past is changed, and therefore effective management on the memories in the distributed memory pool is achieved. Therefore, the technical problem of how to manage the memory can be solved, and the technical effect of effectively managing the memory in the distributed memory pool is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a memory management system, method, apparatus, medium, device, and program product. Background Art

[0002] Distributed memory pools support large-scale distributed computing applications by integrating the memory resources of multiple computing nodes. To ensure the normal operation of the memory in the distributed memory pool, memory management is required.

[0003] Memory management technologies typically involve adding memory expansion cards to compute nodes, which only expands memory capacity but doesn't manage the memory in the distributed memory pool. Therefore, managing memory is an urgent issue. Summary of the Invention

[0004] The present application provides a memory management system, method, apparatus, medium, device and program product to at least solve the problem of how to manage memory in the related art.

[0005] The present application provides a memory management system, comprising: a data processing module, a storage module; The data processing module is connected to the target computing node and the storage module respectively, and the memory pool is connected to the data processing module; the memory pool contains at least two memory nodes; In response to a data processing request inputted by the target computing node into the data processing module, the data processing module calls a data processing method corresponding to the data processing request stored in the storage module; different data processing requests correspond to different data processing methods; The data processing module manages at least one memory node in the memory pool based on the data processing method.

[0006] This application provides a memory management method, including: In response to a data processing request inputted into the data processing module by the target computing node, a pre-stored data processing method corresponding to the data processing request is called; different data processing requests correspond to different data processing methods; At least one memory node in the memory pool is managed based on the data processing method.

[0007] The present application also provides a memory management device, comprising: A calling unit, configured to call a pre-stored data processing method corresponding to a data processing request input to a data processing module by a target computing node; different data processing requests correspond to different data processing methods; The management unit is used to manage at least one memory node in the memory pool based on the data processing method.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned memory management methods when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned memory management methods are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned memory management methods when executed by a processor.

[0011] Through this application, since the data processing module is connected to the target computing node and storage module, it is possible to call the corresponding data processing method in the storage module according to different data processing requests, and dynamically manage the memory nodes in the memory pool based on the data processing method. This overcomes the previous limitation of only being able to expand capacity, thereby achieving effective management of memory in the distributed memory pool. Therefore, it can solve the technical problem of how to manage memory, achieving the technical effect of realizing effective management of memory in the distributed memory pool. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 A schematic diagram of the structure of a memory management system provided in an embodiment of the present application; Figure 2 A flowchart of a memory management method provided in an embodiment of the present application; Figure 3 A schematic diagram of the hardware architecture of a memory management system provided in an embodiment of the present application; Figure 4 A flowchart of a data writing operation provided in an embodiment of the present application; Figure 5 A flowchart of a data reading operation provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a memory management device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0015] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0016] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0017] Figure 1 A schematic diagram of the structure of a memory management system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the memory management system includes: a data processing module and a storage module; The data processing module is connected to the target computing node and the storage module respectively, and the memory pool is connected to the data processing module; the memory pool contains at least two memory nodes.

[0018] The data processing module is the core control unit of the entire memory management system. Its function is similar to the "brain" of a computer. It is responsible for receiving data processing requests from target compute nodes, invoking the corresponding data processing method from the storage module based on the request type, and performing management operations on the memory nodes in the memory pool. The data processing module can be implemented by hardware chips such as a central processing unit, a field-programmable gate array, or an application-specific integrated circuit, combined with appropriate control software.

[0019] The target compute node is the device that initiates data processing requests to the memory management system. For example, in a distributed computing cluster, this could be a server that needs to store, read, or process data, or a workstation running a specific computing task. The target compute node is the source of the data processing request and communicates with the data processing module via a network or data bus. A storage module is a storage device used to store various data processing methods. Similar to a hard drive or solid-state drive in a computer, it can use storage media such as flash memory and disk arrays and run storage management software capable of storing, retrieving, and updating data processing methods. A memory pool is a collection of memory resources integrated from multiple memory nodes that provides memory services to the target compute node. It is equivalent to a "memory warehouse," with memory nodes being the "storage units" within the warehouse. Each memory node can be an independent memory chip, a memory bar, or a memory area partitioned across different compute nodes in a distributed system.

[0020] The data processing module connects to the target compute node and storage module via a high-speed network interface to ensure fast and stable data transmission. Upon receiving a data processing request from the target compute node, the data processing module immediately parses and processes the request, interacts with the storage module via a network communication protocol, and retrieves the data processing method pre-stored in the storage module that corresponds to the request.

[0021] The memory pool and data processing module are connected via a dedicated memory management bus. This bus provides a high-bandwidth, low-latency data transmission channel, ensuring that the data processing module can quickly and accurately read, write, and split memory nodes in the memory pool. Each memory node in the memory pool has a unique address identifier, which the data processing module uses to accurately locate and manage memory nodes. When the memory pool needs to be managed, the data processing module sends control instructions to the corresponding memory node via the memory management bus based on the requested data processing method. This allows for operations such as allocation, reading, writing, splitting, and redundancy protection of the memory node. At the same time, the data processing module also monitors the status of memory nodes in real time, such as memory utilization and data integrity, so that management policies can be adjusted promptly to ensure the stable operation of the memory pool.

[0022] Through centralized management and scheduling of data processing modules, unified management and efficient utilization of multiple memory nodes in the memory pool are achieved. According to the types and characteristics of different data processing requests, appropriate memory nodes can be quickly and accurately allocated for data storage and processing, avoiding waste and conflicts of memory resources and improving the efficiency and performance of memory management.

[0023] In response to a data processing request inputted by the target computing node into the data processing module, the data processing module calls a data processing method corresponding to the data processing request stored in the storage module; different data processing requests correspond to different data processing methods.

[0024] A data processing request is an instruction sent by the target compute node to the data processing module, instructing the system to perform a specific task on the data in the memory pool. For example, in a big data analysis scenario, if the target compute node needs to store analysis results in the memory pool, it will send a data write request. If it needs to retrieve historical data for secondary analysis, it will send a data read request.

[0025] A data processing method is a set of rules and policies corresponding to a specific data processing request. It provides the data processing module with the specific methods and steps for performing the corresponding operation. Each data processing request has its own unique requirements and characteristics, and therefore requires a matching processing method to ensure the accuracy and efficiency of the operation. For example, for a data write request, the data processing method may include steps such as verifying the data, allocating the appropriate memory node, writing the data to the memory node, and updating the relevant data index. For a data read request, the processing method may include searching the storage location based on the data's identification information, reading the data from the memory node, and transferring the data to the target compute node.

[0026] The data processing module connects to the target compute node via a high-speed network interface and receives various data processing requests from the target compute node in real time. Upon receiving a data processing request, the module first parses the request, extracting key information such as the request type (write, read, etc.), the characteristics of the target data (data type, size, identifier, etc.), and information about the target compute node (identity, network address, etc.). Based on the parsed request type, the module searches the storage module for the corresponding data processing method. To improve search efficiency, the storage module employs an efficient data index structure that categorizes and stores different processing methods according to the type of data processing request.

[0027] Once the corresponding data processing method is found, the data processing module begins to manage the memory nodes in the memory pool according to the steps pre-set in that method. For example, when processing a data write request, the data processing module first performs an integrity check on the data to be written to ensure its accuracy, as specified in the data processing method. It then selects the appropriate memory node from the memory pool based on the data type and size, and writes the data to the selected memory node. After the data is written, the data processing module also updates the data index in the memory pool, allowing for quick locating of the data later.

[0028] By storing data processing methods in independent storage modules and aligning them with data processing requests, adding new data processing functions or optimizing existing ones requires simply updating or adding the corresponding data processing methods in the storage modules, eliminating the need for large-scale modifications to the overall data processing module architecture. This design significantly enhances the system's scalability, enabling it to adapt to ever-changing distributed computing application scenarios and requirements.

[0029] The data processing module manages at least one memory node in the memory pool based on the data processing method.

[0030] After receiving a data processing request from a target compute node, the data processing module first parses the request, extracting key information such as the request type, target data characteristics, and target compute node information. Based on the parsed request type, the data processing module searches the storage module for a matching data processing method. The storage module utilizes an efficient data index structure to categorize and store different processing methods according to the data processing request type, enabling the data processing module to quickly and accurately retrieve the required processing method. After obtaining the corresponding data processing method, the data processing module begins managing at least one memory node in the memory pool according to the predefined steps in the method. For example, in the case of a data write request, the data processing module first performs an integrity check on the data to be written to ensure data accuracy. Then, based on the data type and size, it selects a suitable memory node from the memory pool. When selecting a memory node, the data processing module comprehensively considers factors such as the memory node's capacity, data type compatibility, and current load conditions to ensure that the data is stored in the most appropriate node. Once the target memory node is determined, the data processing module writes the data to the selected memory node and, after writing, updates the memory pool's data index to enable rapid subsequent retrieval of the data.

[0031] For data read requests, the data processing module searches the memory pool for a storage location based on the data's identification information, as specified in the data processing method. After finding the target storage location, it reads the data from the corresponding memory node and transmits it to the target compute node. Throughout the data read process, the data processing module also verifies the integrity and accuracy of the data to ensure that the data received by the target compute node is consistent with the data stored in the memory node.

[0032] When processing a data splitting request, the data processing module divides the data to be split into multiple data blocks of predetermined sizes according to the splitting strategy specified in the data processing method. Each data block is assigned to a different memory node for storage, enabling distributed data storage and efficient management. This splitting method improves data read and write efficiency and is particularly suitable for large-scale data processing and analysis scenarios.

[0033] In response to data redundancy protection requests, the data processing module backs up and redundantly stores the target data based on the redundancy strategy specified in the data processing method. It selects multiple suitable memory nodes from the memory pool and stores data copies there. When selecting redundant storage nodes, the data processing module considers factors such as the node's geographic location, network connectivity, and reliability to ensure that redundant copies of the data are distributed across different failure domains, thereby improving data fault tolerance and reliability.

[0034] The data processing module considers multiple factors when selecting memory nodes, allocating data to the most appropriate and efficient memory nodes, thus avoiding wasted memory resources. Furthermore, through the rational allocation and utilization of memory nodes, each node in the memory pool can fully utilize its performance advantages, improving overall memory resource utilization.

[0035] Through this application, since the data processing module is connected to the target computing node and storage module, it is possible to call the corresponding data processing method in the storage module according to different data processing requests, and dynamically manage the memory nodes in the memory pool based on the data processing method. This overcomes the previous limitation of only being able to expand capacity, thereby achieving effective management of memory in the distributed memory pool. Therefore, it can solve the technical problem of how to manage memory, achieving the technical effect of realizing effective management of memory in the distributed memory pool.

[0036] In some embodiments, the data processing request includes any one of a data write request, a data read request, a data split request, and a data redundancy protection request.

[0037] A data write request is an instruction from the target computing node to store data in the memory pool, which is used to add or update data resources in the memory pool, such as saving newly generated calculation results or files uploaded by users. A data read request is an instruction from the target computing node to request the memory pool to retrieve stored data, which is commonly seen in data query, analysis or call scenarios, such as obtaining historical transaction data from the memory pool for report generation. A data splitting request is an instruction from the target computing node to split large-capacity data into multiple data blocks to facilitate distributed storage, parallel processing or transmission of data, such as cutting large video files into segments suitable for network transmission. A data redundancy protection request is an instruction from the target computing node to perform redundant backup or add a verification mechanism for the data in the memory pool to ensure data reliability, such as preventing data loss or damage in financial transaction data storage.

[0038] The data processing module configures a data writing mode corresponding to the data writing request in response to the configuration instruction of data writing, and stores the data writing mode in the storage module; the data processing mode includes the data writing mode.

[0039] In the actual operation of the memory management system, when a target compute node needs to write data to the memory pool, it first sends a data write request containing rich parameter information to the data processing module. Upon receiving this request, the data processing module immediately initiates parsing to accurately extract key features of the data to be written, such as data type and size. The data processing module then fine-tunes the configuration for this data write operation based on pre-set rules or system administrator configuration instructions. It specifies detailed validation rules for data writes, defines memory node selection criteria, and plans index update strategies after data is written, among other operational details, thereby forming a complete and highly targeted data write method. The configured data write method does not exist in isolation; instead, it is stably stored in the storage module using a pre-defined data storage format and high-speed data transmission channels. The storage module then categorizes this data write method and, along with other data processing methods, builds a system knowledge base, providing a comprehensive reference for subsequent data processing operations. Upon receiving a similar data write request, the data processing module does not need to reconfigure itself; instead, it quickly retrieves the pre-configured data write method from the storage module to efficiently execute the write operation on the memory nodes in the memory pool. This efficient configuration and calling mechanism greatly improves the system's operating efficiency and response speed.

[0040] By allowing data processing modules to dynamically adjust data writing methods based on data writing configuration instructions, the system can quickly adapt to the individual needs of different target computing nodes and diverse distributed computing scenarios. Whether processing massive amounts of intermediate results in scientific computing or responding to the high-frequency data writing needs of financial transactions, the system can flexibly configure the most appropriate writing strategy to ensure efficient data storage, greatly enhancing the system's practicality and broad applicability.

[0041] The data processing module configures a data reading mode corresponding to the data reading request in response to a configuration instruction for data reading, and stores the data reading mode in the storage module; the data processing mode includes the data reading mode.

[0042] During the operation of the memory management system, when a target compute node needs to read data from the memory pool, it first sends a data read request containing key information to the data processing module. Upon receiving this request, the data processing module immediately initiates a parsing process to accurately extract characteristic information such as the identifier and type of the data to be read. The data processing module then fine-tunes the configuration for this data read operation based on pre-set rules or system administrator configuration instructions. It specifies a series of operational details, including data location rules, specific data verification methods, and efficient transmission optimization strategies, thereby forming a complete and highly targeted data read method. The configured data read method is stably stored in the storage module using a high-speed data transmission channel in a pre-defined data storage format. The storage module then categorizes this data read method and, along with other data processing methods, builds a system knowledge base, providing a comprehensive reference for subsequent data processing operations. Upon receiving a similar data read request, the data processing module does not need to reconfigure itself; instead, it quickly retrieves the pre-configured data read method from the storage module to efficiently execute read operations on the memory nodes in the memory pool.

[0043] Storing data read methods in the storage module enables configuration information reuse and sharing. When encountering repeated or similar data read requests, the data processing module can quickly retrieve pre-stored read methods without requiring repeated configuration, significantly reducing system resource consumption and response time, thereby improving system efficiency. Furthermore, a rational data transmission optimization strategy helps increase data transmission speed, further enhancing resource utilization and enabling the system to maintain stable and efficient operation even under high loads.

[0044] The data processing module configures a data cutting method corresponding to the data cutting request in response to the configuration instruction of the data cutting, and stores the data cutting method in the storage module; the data processing method includes the data cutting method.

[0045] During the operation of the memory management system, when a target compute node needs to split data in the memory pool, it first sends a data splitting request containing key information to the data processing module. Upon receiving this request, the data processing module immediately initiates a parsing process to accurately extract characteristic information, such as the data range to be split and the size of the split block. The data processing module then fine-tunes the configuration for this data splitting operation based on pre-set rules or system administrator configuration instructions. It specifies a series of operational details, such as the data segmentation rules and the memory node allocation strategy, to develop a complete and highly targeted data splitting method. This configured data splitting method is stably stored in the storage module using a high-speed data transmission channel in a pre-defined data storage format. The storage module then categorizes this data splitting method and, along with other data processing methods, builds a system knowledge base, providing a comprehensive reference for subsequent data processing operations. Upon receiving a similar data splitting request, the data processing module eliminates the need for repeated configuration and simply retrieves the pre-configured data splitting method from the storage module to efficiently execute the splitting operation on the memory nodes in the memory pool.

[0046] Customized data segmentation employs optimal segmentation and processing strategies for different data types. For example, high-resolution videos are segmented and converted to different formats, reducing the size of individual data blocks and alleviating network transmission pressure. Storing segmented data on nodes close to users significantly shortens data loading times and improves user experience.

[0047] The data processing module configures a data redundancy protection mode corresponding to the data redundancy protection request in response to the data redundancy protection configuration instruction, and stores the data redundancy protection mode in the storage module; the data processing mode includes the data redundancy protection mode.

[0048] During the operation of the memory management system, when a target compute node requires redundant protection for critical data in the memory pool, it first sends a data redundancy protection request containing the critical information to the data processing module. Upon receiving this request, the data processing module immediately initiates a parsing process to accurately extract characteristic information, such as the identity of the data to be protected and the desired redundancy level. The data processing module then fine-tunes the configuration of this data redundancy protection operation based on pre-set rules or configuration instructions from the system administrator. It specifies a series of operational details, such as the redundant backup strategy, the selection of data replica storage locations, and the appropriate number of replicas, thereby forming a comprehensive and highly targeted data redundancy protection scheme. The configured data redundancy protection scheme is stably stored in the storage module using a high-speed data transmission channel in accordance with the established data storage format. The storage module then categorizes this data and, together with other data processing schemes, builds a systematic knowledge base, providing a comprehensive reference for subsequent data processing operations. When a similar data redundancy protection request is encountered again in the future, the data processing module does not need to be reconfigured. It only needs to quickly call this pre-configured data redundancy protection method from the storage module to efficiently perform redundancy protection operations on the memory nodes in the memory pool.

[0049] The clear configuration and storage of data redundancy protection methods ensures that every data redundancy protection operation is well-defined. The data processing module strictly follows established rules to execute key steps such as redundant backup and data consistency maintenance. This effectively avoids errors and anomalies during the data redundancy protection process, lays a solid foundation for subsequent data recovery operations, ensures data integrity and availability, and thus improves the reliability and security of the entire system.

[0050] In some embodiments, if the data processing request is a data write request, the data processing module calls the data write method stored in the storage module; the data processing module executes the data write method to manage at least one memory node in the memory pool.

[0051] When the target compute node needs to write data to the memory pool, it sends a data write request to the data processing module. Upon receiving the request, the data processing module first parses the request and extracts key information about the data to be written, such as data type and size. The data processing module then invokes the data write method pre-stored in the storage module. Guided by this data write method, the data processing module begins the specific write operation. The data processing module first performs an integrity check on the data to be written to ensure its accuracy. Next, based on the data type and size, it selects a suitable memory node from the memory pool. When selecting a memory node, the data processing module comprehensively considers factors such as the memory node's capacity, data type compatibility, and current load conditions to ensure that the data is stored in the most appropriate node. After determining the target memory node, the data processing module writes the data to the selected memory node and, after the write is complete, updates the data index in the memory pool to enable quick locating of the data later.

[0052] By invoking the data write method in the storage module through the data processing module, it is possible to achieve precise management and efficient operation of the memory nodes in the memory pool. This enables the memory management system to quickly respond to data write requests from the target computing node, improving the operating efficiency of the entire distributed computing application.

[0053] In some embodiments, when the data processing method to be executed is a data writing method, the data processing module obtains the target data type of the data to be stored in the data writing request; the data processing module determines the target memory node type corresponding to the target data type based on the pre-established mapping relationship between the data type and the memory node type; the data processing module stores the data to be stored in the idle memory node corresponding to the target memory node type.

[0054] The target data type refers to the type of data to be stored. The target data type determines the data's structure, format, and subsequent processing. Common data types include text, binary data, images, video, audio, etc. A memory node type refers to the node category used to store different types of data in the memory space. Each memory node type is optimized for a specific data type. Different memory nodes may have different capacities, access speeds, data structures, and other characteristics. A mapping relationship refers to the predefined association between the target data type and the memory node type. The mapping relationship determines which memory node a certain data type should be stored in. Mapping relationships are typically established during system initialization based on the characteristics of the memory and data type requirements. A free memory node refers to memory space that is currently unused or has been released. Free memory nodes can store new data and are allocated according to the mapping relationship.

[0055] When the data processing module executes a data write request, it first obtains the target data type of the data to be stored from the data write request. This process is typically accomplished by parsing the relevant fields in the data write request, where the data type of the data to be stored is clearly identified in the request, allowing the data processing module to accurately identify it. After determining the target data type, the data processing module determines the target memory node type corresponding to the target data type based on a pre-established mapping relationship between data types and memory node types. This mapping relationship can be stored in a memory management system in the form of a table, configuration file, or database. The data processing module searches this mapping relationship table to quickly locate the memory node type suitable for storing the data type. Next, the data processing module searches the memory pool for a currently idle memory node of the target memory node type. The memory pool maintains a memory node status table that records information such as the type and status (idle or occupied) of each memory node. The data processing module queries this status table to select eligible idle memory nodes. Finally, the data processing module writes the data to the selected idle memory node. After the data write is complete, the data processing module updates the memory pool's data index and memory node status table to reflect the data storage location and the latest status of the memory node.

[0056] By selecting the appropriate memory node type based on the target data type, you can ensure that data is stored in the most suitable memory node, fully leverage the performance advantages of each type of memory node, increase the speed of data storage and reading, and thus improve the efficiency of the entire memory management system.

[0057] In some embodiments, if the data processing request is a data read request, the data processing module calls the data reading method stored in the storage module; the data processing module executes the data reading method to manage at least one memory node in the memory pool.

[0058] When a target compute node needs to read data from the memory pool, it sends a data read request to the data processing module. Upon receiving the request, the data processing module first parses it, extracting key information such as the request type, target data identification information, and information about the target compute node. Based on the parsed request type, the data processing module searches the storage module for a matching data read method. The storage module utilizes an efficient data index structure that categorizes and stores different processing methods according to the type of data processing request, enabling the data processing module to quickly and accurately retrieve the required processing method. After obtaining the corresponding data read method, the data processing module begins managing at least one memory node in the memory pool according to the steps predefined in the method. Based on the data read method, the data processing module first searches the memory pool for the data's identification information. The memory pool maintains a data index table that records each data item's storage location, data type, and memory node location. The data processing module quickly locates the memory node where the target data is located by querying this data index table. After finding the target storage location, the data processing module reads the data from the corresponding memory node. During the reading process, the data processing module verifies the integrity and consistency of the data, ensuring that the read data is consistent with the data stored in the memory node. Verification methods may include checksum verification and data version verification. Finally, the data processing module transmits the read data to the target compute node via the network. During the data transmission process, the data processing module uses an efficient data transmission protocol to ensure that the data reaches the target compute node quickly and accurately. At the same time, the data processing module also updates the relevant data access records to facilitate subsequent data access statistics and analysis.

[0059] The data processing module precisely manages memory nodes based on data access methods, enabling efficient access to data in the memory pool. This enables the memory management system to quickly respond to data read requests from target compute nodes, improving the efficiency and performance of the entire distributed computing application.

[0060] In some embodiments, when the data processing method executed is a data reading method, the data processing module obtains the target identification information of the data to be read in the data reading request; the data processing module determines the target storage location corresponding to the target identification based on the mapping relationship between the pre-established identification information and the storage location; the data processing module reads the data to be read corresponding to the target storage location based on the target storage location, and transmits the data to be read to the target computing node.

[0061] Target identification information is information used in a data read request to uniquely identify the data to be read. It can be a file name, data block number, unique identifier, etc., used to help the data processing module quickly locate the specific location of the target data in the memory pool. A mapping relationship is a predefined correspondence that associates the identification information of the data with the storage location of the data in the memory pool. This mapping relationship is usually stored in the memory management system in the form of a table, configuration file, or database, which facilitates quick query by the data processing module. The target storage location refers to the physical location where the target data is actually stored in the memory pool, including information such as the address of the memory node and the offset of the data in the memory node. The data processing module can accurately locate the memory node and specific location of the data through the target storage location.

[0062] When the data processing module executes a data read, it first obtains the target identification information of the data to be read from the data read request. This process is typically accomplished by parsing the relevant fields in the data read request. The request clearly identifies the identification information of the data to be read, allowing the data processing module to accurately identify it. After determining the target identification information, the data processing module determines the target storage location corresponding to the target identification information based on a pre-established mapping relationship between identification information and storage locations. This mapping relationship can be stored in a memory management system in the form of a table, configuration file, or database. The data processing module quickly locates the storage location of the target data by searching this mapping relationship table. Next, the data processing module sends a read instruction to the memory node corresponding to the target storage location. After receiving the instruction, the memory node reads the data from the specified storage location and returns the data to the data processing module. After receiving the data, the data processing module verifies the integrity and consistency of the data to ensure that the read data is consistent with the data stored in the memory pool. Finally, the data processing module transmits the read data to the target compute node. During the data transmission process, the data processing module uses an efficient data transmission protocol to ensure that the data reaches the target compute node quickly and accurately. At the same time, the data processing module will also update the relevant data access records to facilitate subsequent data access statistics and analysis.

[0063] By quickly locating the data storage location based on the target identification information, the data processing module can efficiently read data from the memory pool. This greatly shortens the data reading time, improves the response speed and operating efficiency of the entire system, and ensures that the target computing node can obtain the required data in a timely manner.

[0064] In some embodiments, if the data processing request is a data cutting request, the data processing module calls the data cutting method stored in the storage module; the data processing module executes the data cutting method and manages at least one memory node in the memory pool.

[0065] When a target compute node needs to split data in the memory pool, it sends a data split request to the data processing module. Upon receiving the request, the data processing module first parses it, extracting key information such as the request type, the range of the data to be split, and the size of the resulting data chunks. Based on the parsed request type, the data processing module searches the storage module for a matching data split method. The storage module utilizes an efficient data index structure that categorizes and stores different processing methods according to the type of data processing request, enabling the data processing module to quickly and accurately retrieve the required processing method. After obtaining the corresponding data split method, the data processing module begins managing at least one memory node in the memory pool according to the steps pre-defined in the split method. Based on the data split method, the data processing module first searches the memory pool for the corresponding data based on the range of the data to be split. The memory pool maintains a data index table that records information such as the storage location, data type, and memory node of each data item. The data processing module quickly locates the memory node where the data to be split resides by querying this data index table. After locating the data to be split, the data processing module splits the data into multiple data chunks according to the specified size. Each data block is assigned a new unique identifier, and the data processing module updates the data index, recording information such as the storage location of each data block. The segmented data blocks can be assigned to different computing nodes for parallel processing, thereby improving data processing efficiency. Finally, the data processing module stores the segmented data blocks in the corresponding memory nodes in the memory pool and updates the memory pool's data index and memory node status table to reflect the data storage location and the latest status of the memory nodes.

[0066] The data processing module precisely manages memory nodes based on data segmentation, enabling efficient segmentation and allocation of data within the memory pool. This enables the memory management system to quickly respond to data segmentation requests from target compute nodes, improving the efficiency and performance of the entire distributed computing application, making it particularly suitable for large-scale parallel data processing scenarios.

[0067] In some embodiments, when the data processing method executed is the data cutting method, the data processing module obtains the target cutting size of the data to be cut in the data cutting request; the data processing module cuts the data to be cut into at least two cutting data according to the target cutting size; the data size of each cutting data is the target cutting size.

[0068] The target size for each data block is defined in the data splitting request. The target size is a requirement that the data processing module ensures each data block meets when splitting the data. The split data refers to the small data blocks obtained through the data splitting operation. Each split data block must meet the target size.

[0069] The data processing module receives a data splitting request from a user or other system component. This request contains necessary parameters, such as the source information of the data to be split, the target split size, and the splitting rules. The data processing module extracts the target split size from the data splitting request. The target split size indicates the ideal size of each split data item and is typically defined by the user based on requirements. For example, the size of each data block might be 1MB, 10MB, or so. The data processing module determines the total size of the data to be split. This size is the number of bytes or file size of the data to be split. This information is crucial for determining how to execute the data splitting operation. The data processing module splits the data to be split into multiple data blocks based on the target split size. The size of each data block is close to the target split size (if the data to be split cannot be split evenly, the size of the last data block may be smaller than the target split size). In actual execution, the data processing module splits the data to be split into at least two data blocks. The size of each data block should meet the target split size as much as possible, ensuring that the data is distributed as evenly as possible. The data processing module executes the splitting operation, splitting the data to be split into several small data blocks based on the target split size, with each data block close to the target split size. The split data blocks are then used as split data for subsequent processing. After the segmentation is complete, the data processing module stores the generated segmented data in a designated location or memory pool. Each data block is allocated appropriate storage resources based on its size and storage requirements. The data processing module also manages the segmented data blocks to ensure their integrity and validity.

[0070] By splitting a large data block into multiple smaller blocks, data storage and transmission can be made more efficient. Small data blocks are easier to store, distribute, and transmit than large data blocks, especially in distributed systems.

[0071] In some embodiments, if the data processing request is a data redundancy protection request, the data processing module calls the data redundancy protection method stored in the storage module; the data processing module executes the data redundancy protection method and manages at least one memory node in the memory pool.

[0072] When a target compute node requires redundant protection for critical data in the memory pool, it sends a data redundancy protection request to the data processing module. Upon receiving the request, the data processing module first parses it, extracting key information such as the request type, the identifier of the data to be protected, and the desired redundancy level. Based on the parsed request type, the data processing module searches the storage module for a matching data redundancy protection method. The storage module utilizes an efficient data index structure that categorizes and stores different processing methods according to the type of data processing request, enabling the data processing module to quickly and accurately retrieve the required method. After obtaining the corresponding data redundancy protection method, the data processing module begins managing at least one memory node in the memory pool according to the steps specified in the method. Based on the data redundancy protection method, the data processing module first searches the memory pool for the current storage location and status of the target data based on the data identifier. The memory pool maintains a data index table that records information such as the storage location, data type, and memory node of each data item. By querying this data index table, the data processing module quickly locates the memory node where the data requiring redundant protection resides. Next, the data processing module will back up the target data according to the preset redundancy strategy. This may include creating multiple copies of the data and storing these copies on memory nodes in different locations in the memory pool. When selecting redundant storage nodes, the data processing module will consider factors such as the node's geographical location, network connectivity, and reliability to ensure that redundant copies of the data can be distributed across different fault domains, thereby improving the data's fault tolerance. During the data redundancy protection process, the data processing module will also monitor the status of each memory node in real time to ensure the consistency and integrity of the redundant data. Once a memory node failure or data corruption is detected, the data processing module will automatically restore the data from other normal redundant copies and update the data index table to ensure the continued stable operation of the system.

[0073] The data processing module precisely manages memory nodes based on data redundancy protection methods, achieving efficient redundant protection for critical data in the memory pool. This enables the memory management system to quickly respond to data redundancy protection requests from target compute nodes, improving data reliability across the entire distributed computing application.

[0074] In some embodiments, when the data processing method executed is a data redundancy protection method, the data processing module obtains the target redundancy level of the data to be protected in the data redundancy protection; the data processing module determines the target redundancy protection strategy corresponding to the target redundancy level based on the mapping relationship between the pre-established redundancy level and the redundancy protection strategy; different redundancy levels correspond to different redundancy protection strategies; the data processing module performs redundant protection on the data to be protected based on the redundant protection strategy.

[0075] Target redundancy level of the data to be protected: A pre-set standard for data redundancy protection strength used to measure the required level of redundancy for the data. Common redundancy levels include RAID 1 (mirror redundancy), RAID 5 (distributed parity redundancy), and a three-copy strategy. Different levels correspond to different redundancy protection strategies and resource consumption, and are the core basis for the data processing module to select a protection strategy. The mapping relationship between redundancy level and redundancy protection strategy refers to the pre-established correspondence rules that associate different redundancy levels with specific redundancy protection strategies. This mapping relationship is stored in the storage module in the form of a configuration file, database table, or hash table, facilitating the data processing module to quickly query and invoke the corresponding strategy. The target redundancy protection strategy refers to the specific redundancy protection scheme determined by the mapping relationship based on the target redundancy level of the data to be protected, such as the number of data copies, storage node selection rules, and verification algorithm. It directly guides the data processing module in performing redundancy protection operations.

[0076] When the data processing module executes a data redundancy protection method, it first obtains the target redundancy level of the data to be protected from the data redundancy protection request. This process is typically accomplished by parsing the relevant fields in the data redundancy protection request. The request clearly identifies the redundancy level of the data to be protected, allowing the data processing module to accurately identify it. After determining the target redundancy level, the data processing module determines the target redundancy protection policy corresponding to the target redundancy level based on a pre-established mapping relationship between redundancy levels and redundancy protection policies. This mapping relationship can be stored in a memory management system in a table, configuration file, or database. The data processing module quickly locates the appropriate redundancy protection policy by searching the mapping relationship table. Next, the data processing module performs redundant backup operations on the data to be protected according to the requirements of the target redundancy protection policy. The specific steps are as follows: Determine the number of data replicas: Based on the target redundancy protection policy, determine the number of data replicas to create. For example, the policy may require creating two replicas to achieve three-copy storage. Select storage locations: Based on the redundancy protection policy, select appropriate memory nodes to store the data replicas. This may involve distributing the replicas across different physical servers, racks, or data centers to improve fault tolerance. Create and store replicas: The data processing module copies the protected data to the selected memory nodes and ensures the integrity and consistency of each replica. Maintain data consistency: When data is updated, all replicas are updated synchronously according to the redundancy protection strategy to ensure data consistency. Finally, the data processing module updates the memory pool's data index and memory node status table to reflect the latest status of data redundancy protection.

[0077] By selecting the appropriate redundancy protection strategy based on the target redundancy level, data availability can be ensured in various failure scenarios. Even if some memory nodes fail, the system can recover data from other replicas, thereby improving data reliability and fault tolerance.

[0078] According to the embodiment of the present application, the present application also proposes a memory management method, such as Figure 2 As shown, Figure 2 This is a flow chart of a memory management method provided in an embodiment of the present application. The method is applied to a memory management system and includes the following steps: Step 101 : In response to a data processing request inputted into a data processing module of a target computing node, a pre-stored data processing method corresponding to the data processing request is called; different data processing requests correspond to different data processing methods.

[0079] Step 102: Manage at least one memory node in the memory pool based on the data processing method.

[0080] Through this application, since the data processing module is connected to the target computing node and storage module, it is possible to call the corresponding data processing method in the storage module according to different data processing requests, and dynamically manage the memory nodes in the memory pool based on the data processing method. This overcomes the previous limitation of only being able to expand capacity, thereby achieving effective management of memory in the distributed memory pool. Therefore, it can solve the technical problem of how to manage memory, achieving the technical effect of realizing effective management of memory in the distributed memory pool.

[0081] As a refinement of the above embodiment, the data processing request includes any one of a data writing request, a data reading request, a data cutting request, and a data redundancy protection request; In response to a configuration instruction for data writing, configuring a data writing mode corresponding to the data writing request and storing the data writing mode; the data processing mode includes the data writing mode; In response to a data reading configuration instruction, configuring a data reading mode corresponding to the data reading request and storing the data reading mode; the data processing mode includes the data reading mode; In response to a configuration instruction for data cutting, configuring a data cutting method corresponding to the data cutting request and storing the data cutting method; the data processing method includes the data cutting method; In response to the data redundancy protection configuration instruction, a data redundancy protection mode corresponding to the data redundancy protection request is configured, and the data redundancy protection mode is stored; the data processing mode includes the data redundancy protection mode.

[0082] As a refinement of the above embodiment, when executing management of at least one memory node in a memory pool based on a data processing method, it can be implemented by but not limited to the following methods, including: if the data processing request is a data write request, calling a pre-stored data write method; when the executed data processing method is a data write method, obtaining the target data type of the data to be stored in the data write request; determining the target memory node type corresponding to the target data type based on a pre-established mapping relationship between the data type and the memory node type; and storing the data to be stored in an idle memory node corresponding to the target memory node type.

[0083] As a refinement of the above embodiment, when executing a data processing method to manage at least one memory node in a memory pool, it can be implemented by but not limited to the following methods, including: if the data processing request is a data read request, calling a pre-stored data read method; when the executed data processing method is a data read method, obtaining the target identification information of the data to be read in the data read request; determining the target storage location corresponding to the target identification based on the mapping relationship between the pre-established identification information and the storage location; based on the target storage location, reading the data to be read corresponding to the target storage location, and transmitting the data to be read to the target computing node.

[0084] As a refinement of the above embodiment, when executing management of at least one memory node in the memory pool based on the data processing method, it can be implemented by but not limited to the following methods, including: if the data processing request is a data cutting request, calling a pre-stored data cutting method; when the executed data processing method is the data cutting method, obtaining the target cutting size of the data to be cut in the data cutting request; according to the target cutting size, cutting the data to be cut into at least two cutting data; the data size of each cutting data is the target cutting size.

[0085] As a refinement of the above embodiment, when executing management of at least one memory node in a memory pool based on a data processing method, it can be implemented in but not limited to the following methods, including: if the data processing request is a data redundancy protection request, calling a pre-stored data redundancy protection method; when the executed data processing method is the data redundancy protection method, obtaining the target redundancy level of the data to be protected in the data redundancy protection; determining the target redundancy protection policy corresponding to the target redundancy level based on the mapping relationship between the pre-established redundancy level and the redundancy protection policy; different redundancy levels correspond to different redundancy protection policies; and based on the redundancy protection policy, performing redundant protection on the data to be protected.

[0086] Since the embodiments of the memory management method and the memory management system correspond to each other, the embodiments of the memory management method can be found in the description of the embodiments of the memory management system, and will not be repeated here. The embodiments of the memory management method have the same beneficial effects as the aforementioned memory management system.

[0087] In order to better understand the hardware architecture of the memory management system of the embodiment of the present application, Figure 3 As shown, Figure 3This is a schematic diagram of the hardware architecture of a memory management system provided in an embodiment of the present application. This includes the hardware architecture of the memory array management card (i.e., the memory management system). The high-performance data processing chip (i.e., the data processing module) is responsible for handling all data read and write requests and complex data conversion operations. This chip possesses powerful computing capabilities, enabling rapid processing of massive amounts of data and supporting low-latency and high-throughput data access. The storage control unit (i.e., the storage module) is closely connected to the data processing chip and is responsible for storing the management card's configuration information, including data storage policies (such as data distribution rules and priority policies), striping standards (such as stripe size and splitting rules), and redundancy checking methods (such as Redundant Arrays of Independent Memory (RAIM) levels and parity checking rules). The local cache unit, located between the data processing chip and the storage control unit, is used to temporarily store frequently accessed data and pre-read data. High-speed caching technology significantly reduces repeated accesses to the local memory module, improving data access efficiency. The communication interface module connects to upper-layer applications and the management card and is responsible for receiving and sending data requests. It supports multiple communication protocols to ensure efficient interaction with upper-layer applications. A standard high-speed serial peripheral component interconnect express (PCIE) interface: The PCIE interface is a key channel connecting the memory management card to the distributed memory pool nodes. Through this interface, the memory management card's high-performance data processing chip can connect to and exchange data with the memory pool's memory, achieving fast and stable data transmission. 2. Data Read and Write Function Implementation: When receiving a data write request from an upper-layer application, the management card's data processing chip receives the data through the communication interface module and analyzes the data characteristics based on write rules pre-stored in the local storage control unit. These rules can be formulated based on factors such as data type, size, and application priority, ensuring that data is stored in an organized manner in the appropriate location in the local memory module, achieving orderly data organization. When reading data, the management card's data processing chip, based on the request information provided by the compute node, quickly queries the data storage location index in the local storage control unit. It then quickly locates and extracts the corresponding data via high-speed data channels and returns it to the requesting node. The entire process is completed extremely quickly, meeting the distributed application's demand for fast memory data access. 3. Striping Function: The memory management card can stripe data entering the node based on instructions or pre-defined policies. The data processing chip cuts the data into segments (strips) of fixed size according to unified standards, and records the relevant information of the strips in the local storage control unit, including the stripe number, the data block to which it belongs, the storage location, etc., providing basic support for subsequent distributed storage and data scheduling, ensuring the orderly organization of data at the local level, and facilitating collaborative management in a distributed environment.4. Redundancy Check: The management card's data processing chip regularly initiates a redundancy check on data stored in the local memory array. This checks data accuracy using parity checks, parity bit generation, or more advanced error correction coding techniques. Upon detecting a local data error, the chip's built-in error correction unit immediately attempts to correct the error using redundant information. If the error can't be corrected immediately, the chip flags the error and promptly reports it to the global distributed memory controller via the communication interface module for further processing, ensuring data reliability and reducing the risk of data loss. 5. Data Redundancy Protection: The memory management card supports multiple levels of memory redundancy: RAIM 1: Data is written simultaneously to two memory modules, mirroring each other. If one memory module fails, the other module immediately provides the data, ensuring data integrity. This mode is suitable for scenarios requiring extremely high data security, such as financial transaction data storage. RAIM 5: Using parity checking, data and checksum information are distributed across multiple memory modules, allowing for single-module failure without data loss. This mode is suitable for scenarios requiring high data reliability. RAIM 0: Stores data in blocks across multiple memory modules, significantly improving data read and write speeds through parallel reading and writing. It is suitable for scenarios with high performance requirements but low redundancy requirements, such as video editing and temporary storage areas for large databases. RAIM 10: Combining the advantages of RAIM 1 and RAIM 0, it ensures data security and improves read and write performance. It is often used in server scenarios with high requirements for both performance and data security. RAIM 50: Based on RAIM 5, it introduces a multi-layer parity check mechanism to support data recovery when multiple memory modules fail at the same time. It is suitable for scenarios with extremely high data reliability requirements, such as medical data storage and scientific research. RAIM 60: Combining the advantages of RAIM 10 and RAIM 50, it ensures data security and improves read and write performance through a combination of mirroring and multiple parity checks. It also supports data recovery when multiple memory modules fail. It is suitable for mission-critical scenarios with extremely high performance and data security requirements. In summary, the memory array management card for distributed memory pools provided by the present invention provides support for the management and reliability of distributed memory systems.

[0088] This embodiment of the present application describes in detail a memory array management card for a distributed memory pool, which supports the management and reliability of distributed memory systems. The following are the detailed steps of this embodiment: 1. Initialization Phase: 1.1 Self-Check and Connection with Local Memory Modules: After powering on, the management card first performs a self-check on the local memory modules, checking their health (e.g., presence of bad blocks, proper connection), etc. It then establishes a stable connection with the local memory modules to ensure smooth high-speed data access. If an anomaly is detected (e.g., a memory module failure), the management card records the error information and reports it to the corresponding compute node via the communication interface module. 1.2 Configuration Information Initialization: Initial configuration information is checked, including data storage policies (e.g., data distribution rules, priority policies), striping criteria (e.g., stripe size, splitting rules), and redundancy check methods (e.g., RAIM levels, parity check rules, etc.). This configuration information is stored in the local storage control unit and serves as the basis for subsequent data processing. 1.3 Local Cache Unit Initialization: The management card initializes the local cache units, preloading frequently accessed data or metadata (e.g., index information, stripe mapping tables, etc.) to improve subsequent data access efficiency. The cache unit uses high-speed cache technology to significantly reduce repeated access to the local memory module. 2. Data processing flow: During normal operation, the management card performs data writing operations based on the requests of upper-layer applications: In order to better understand the data writing operations, such as Figure 4 As shown, Figure 4 A flow chart of a data writing operation provided in an embodiment of the present application, 2.1 Data writing process: Receiving a write request: The management card receives a data write request (such as a large file or database transaction data) from a computing node through a communication interface module. Data classification and processing: The data processing chip classifies the received data (such as by data type, application priority, etc.). For example, for database transaction data, the management card selects a suitable memory address segment for storage based on the transaction priority and the distribution of free areas in the local memory. Data storage and index recording: The data is stored in the specified location of the local memory module. At the same time, the management card records the index information of the data block (such as data block ID, storage address, timestamp, etc.) in the local storage control unit for subsequent quick retrieval. Confirmation information is returned after successful writing. In order to facilitate a better understanding of the data reading operation, such as Figure 5 As shown, Figure 5This is a schematic diagram of a data read operation process provided by an embodiment of the present application. 2.2 Data Read Process: Receiving a Read Request: The management card receives a data read request from a compute node via the communication interface module. Data Locating and Extracting: Based on the read request information, the data processing chip queries the local storage control unit for the data storage location index. The corresponding data is quickly located and extracted via a high-speed data channel. Data Return: The data processing chip returns the extracted data to the compute node via the communication interface.

[0089] To better understand the striping operation (i.e., data splitting), an example is provided. 3. When new data flows into a node, the management card performs the striping operation: 3.1 Receiving Instructions and Data Splitting: The management card stripes the incoming data based on real-time instructions from the global distributed memory controller. For example, a 1GB video stream can be split into 1MB stripes. 3.2 Stripe Information Recording: Each stripe is labeled with relevant information, including the stripe number, the data block to which it belongs (such as the video frame number), the storage timestamp, and the storage location (such as the local memory module address). This information is recorded in the local storage control unit for subsequent distributed storage and scheduling. 4. Regarding redundancy checking, the management card periodically performs redundancy checking to ensure data reliability and integrity: 4.1 Starting the Verification Process: At regular intervals (such as one hour), the data processing chip initiates a redundancy check on the local memory array. 4.2 Verification Method: Advanced technologies such as parity checking and error correction coding (such as ECC) are used to verify data accuracy. 4.3 Error handling: If it is found that a certain stripe data verification fails (such as parity check failure), the management card immediately starts the error correction program and uses redundant information to restore the data. If the error correction fails, the management card will mark the area as abnormal and notify the central processor through the communication interface module to arrange data reconstruction or other remedial measures. 5. Performance optimization and expansion functions: 5.1 Dynamic load balancing: The management card dynamically adjusts data storage and access strategies according to the instructions of the memory node controller to achieve load balancing. For example, when the load on a node is too high, part of the data will be migrated to other nodes. 5.2 Multi-level cache mechanism: The management card supports multi-level cache (such as L1 and L2 cache) to further optimize data access efficiency. 5.3 Disaster recovery and fault recovery: When a local memory module failure is detected, the management card will automatically start the disaster recovery mechanism, migrate the data to other nodes, and notify the global controller to perform fault recovery.

[0090] Based on the above, this memory array management card for distributed memory pools can greatly improve the efficiency and reliability of distributed memory management, providing strong memory support for distributed computing applications. This application provides a memory array management card for distributed memory pools, which is designed to support the management and reliability of distributed memory systems. The beneficial effects of the present invention are mainly reflected in the following aspects: 1. Improved data read and write efficiency: Through the high-performance data processing chip and storage control unit, combined with high-speed channels and memory module interconnection, rapid data location, retrieval, and cache optimization are achieved. In particular, the application of RAIM 0 technology significantly improves data read and write speed by storing data in blocks on multiple disks and enabling parallel read and write. It is suitable for scenarios with high read and write performance requirements, such as video editing and temporary storage areas for large databases. 2. Enhanced data reliability: Through redundancy check functions and multiple RAIM-level data redundancy protection mechanisms, such as RAIM 1 and RAIM 5, data integrity and security are effectively guaranteed. RAIM 1 uses mirrored storage to ensure data accessibility even in the event of a single memory failure, while RAIM 5 uses parity checking and distributed storage to tolerate single memory failures without data loss, significantly reducing the risk of data loss. 3. Improve the intelligence level of memory management: The striping function cuts data into strips and records relevant information according to instructions or preset strategies, providing basic functions for distributed storage and scheduling. This intelligent management method not only improves the utilization rate of memory, but also enhances the synergy of distributed memory, making memory management more efficient and flexible. 4. Wide application value: The present invention is applicable to a variety of scenarios with different requirements for data security and read-write performance, such as financial transaction data storage, video editing, temporary storage areas for large databases, and high-performance servers. Its technological advancement and flexibility give it broad application prospects and market value in the field of distributed memory management. In summary, the present invention significantly improves the efficiency, reliability and flexibility of distributed memory management through the optimization of hardware architecture and intelligent management methods, and has important technological advancement and application value.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0092] The embodiment of the present application also provides a memory management device, Figure 6 A schematic diagram of the structure of a memory management device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, including: The calling unit 31 is configured to call a pre-stored data processing method corresponding to a data processing request inputted by the target computing node into the data processing module; different data processing requests correspond to different data processing methods; The management unit 32 is configured to manage at least one memory node in the memory pool based on a data processing method.

[0093] Through this application, since the data processing module is connected to the target computing node and storage module, it is possible to call the corresponding data processing method in the storage module according to different data processing requests, and dynamically manage the memory nodes in the memory pool based on the data processing method. This overcomes the previous limitation of only being able to expand capacity, thereby achieving effective management of memory in the distributed memory pool. Therefore, it can solve the technical problem of how to manage memory, achieving the technical effect of realizing effective management of memory in the distributed memory pool.

[0094] Furthermore, in a possible implementation of this embodiment, the data processing request includes any one of a data writing request, a data reading request, a data cutting request, and a data redundancy protection request; In response to a configuration instruction for data writing, configuring a data writing mode corresponding to the data writing request and storing the data writing mode; the data processing mode includes the data writing mode; In response to a data reading configuration instruction, configuring a data reading mode corresponding to the data reading request and storing the data reading mode; the data processing mode includes the data reading mode; In response to a configuration instruction for data cutting, configuring a data cutting method corresponding to the data cutting request and storing the data cutting method; the data processing method includes the data cutting method; In response to the data redundancy protection configuration instruction, a data redundancy protection mode corresponding to the data redundancy protection request is configured, and the data redundancy protection mode is stored; the data processing mode includes the data redundancy protection mode.

[0095] Furthermore, in a possible implementation of this embodiment, the management unit 32 is further configured to: When the data processing request is a data writing request, a pre-stored data writing method is called; When the executed data processing mode is a data writing mode, obtaining a target data type of the data to be stored in the data writing request; Determine the target memory node type corresponding to the target data type based on the pre-established mapping relationship between the data type and the memory node type; Store the data to be stored in an idle memory node corresponding to the target memory node type.

[0096] Furthermore, in a possible implementation of this embodiment, the management unit 32 is further configured to: When the data processing request is a data reading request, a pre-stored data reading method is called; When the executed data processing mode is a data reading mode, obtaining target identification information of the data to be read in the data reading request; Determine the target storage location corresponding to the target identifier based on a pre-established mapping relationship between the identifier information and the storage location; Based on the target storage location, the to-be-read data corresponding to the target storage location is read, and the to-be-read data is transmitted to the target computing node.

[0097] Furthermore, in a possible implementation of this embodiment, the management unit 32 includes: When the data processing request is a data cutting request, a pre-stored data cutting method is called; When the executed data processing mode is the data cutting mode, obtaining the target cutting size of the data to be cut in the data cutting request; According to the target cutting size, the data to be cut is cut into at least two cutting data; the data size of each cutting data is the target cutting size.

[0098] Furthermore, in a possible implementation of this embodiment, the management unit 32 includes: When the data processing request is a data redundancy protection request, calling a pre-stored data redundancy protection method; When the executed data processing mode is a data redundancy protection mode, obtaining a target redundancy level of the data to be protected in the data redundancy protection; Determine the target redundancy protection strategy corresponding to the target redundancy level according to the pre-established mapping relationship between the redundancy level and the redundancy protection strategy; different redundancy levels correspond to different redundancy protection strategies; Based on the redundancy protection strategy, redundant protection is performed on the data to be protected.

[0099] For the description of the features in the embodiment corresponding to the memory management device, please refer to the relevant description of the embodiment corresponding to the memory management method, and no further details will be given here.

[0100] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above-mentioned memory management method embodiments.

[0101] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned memory management method embodiments when running.

[0102] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0103] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned memory management method embodiments are implemented.

[0104] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned memory management method embodiments are implemented.

[0105] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] The above is a detailed introduction to a memory management system, method, apparatus, medium, device and program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A memory management system, characterized in that: include: Data processing module, storage module; The data processing module is connected to the target computing node and the storage module respectively, and the memory pool is connected to the data processing module; the memory pool contains at least two memory nodes; In response to a data processing request inputted by the target computing node to the data processing module, the data processing module calls a data processing method corresponding to the data processing request stored in the storage module; different data processing requests correspond to different data processing methods; The data processing module manages at least one memory node in the memory pool based on the data processing method.

2. The memory management system according to claim 1, wherein: The data processing request includes any one of a data writing request, a data reading request, a data cutting request, and a data redundancy protection request; The data processing module configures a data writing mode corresponding to the data writing request in response to a configuration instruction for data writing, and stores the data writing mode in the storage module; the data processing mode includes the data writing mode; The data processing module configures a data reading mode corresponding to the data reading request in response to a configuration instruction for data reading, and stores the data reading mode in the storage module; the data processing mode includes a data reading mode; The data processing module configures a data cutting method corresponding to the data cutting request in response to the data cutting configuration instruction, and stores the data cutting method in the storage module; the data processing method includes the data cutting method; The data processing module configures a data redundancy protection mode corresponding to the data redundancy protection request in response to a data redundancy protection configuration instruction, and stores the data redundancy protection mode in the storage module; the data processing mode includes the data redundancy protection mode.

3. The memory management system according to claim 2, wherein: If the data processing request is a data writing request, the data processing module calls the data writing method stored in the storage module; The data processing module executes the data writing method to manage at least one memory node in the memory pool.

4. The memory management system according to claim 3, wherein: In a case where the data processing mode to be executed is the data writing mode, the data processing module obtains a target data type of the data to be stored in the data writing request; The data processing module determines the target memory node type corresponding to the target data type according to a pre-established mapping relationship between the data type and the memory node type; The data processing module stores the data to be stored in an idle memory node corresponding to the target memory node type.

5. The memory management system according to claim 2, wherein: If the data processing request is a data reading request, the data processing module calls the data reading method stored in the storage module; The data processing module executes the data reading method and manages at least one memory node in the memory pool.

6. The memory management system according to claim 5, characterized in that: In a case where the data processing mode executed is the data reading mode, the data processing module obtains target identification information of the to-be-read data in the data reading request; The data processing module determines the target storage location corresponding to the target identifier according to a pre-established mapping relationship between the identifier information and the storage location; The data processing module reads the to-be-read data corresponding to the target storage location based on the target storage location, and transmits the to-be-read data to the target computing node.

7. The memory management system according to claim 2, wherein: If the data processing request is a data cutting request, the data processing module calls the data cutting method stored in the storage module; The data processing module executes the data cutting method and manages at least one memory node in the memory pool.

8. The memory management system according to claim 7, wherein: In a case where the data processing mode to be executed is the data cutting mode, the data processing module obtains a target cutting size of the data to be cut in the data cutting request; The data processing module cuts the data to be cut into at least two cut data according to the target cut size; the data size of each cut data is the target cut size.

9. The memory management system according to claim 2, wherein: If the data processing request is a data redundancy protection request, the data processing module calls the data redundancy protection mode stored in the storage module; The data processing module executes the data redundancy protection mode and manages at least one memory node in the memory pool.

10. The memory management system according to claim 9, wherein: In a case where the data processing mode executed is the data redundancy protection mode, the data processing module obtains a target redundancy level of the data to be protected in the data redundancy protection; The data processing module determines the target redundancy protection strategy corresponding to the target redundancy level according to a pre-established mapping relationship between the redundancy level and the redundancy protection strategy; Different redundancy levels correspond to different redundancy protection strategies; The data processing module performs redundant protection on the data to be protected based on the redundant protection strategy.

11. A memory management method, characterized in that: include: In response to a data processing request inputted into the data processing module by the target computing node, calling a pre-stored data processing mode corresponding to the data processing request; Different data processing requests correspond to different data processing methods; Based on the data processing method, at least one memory node in the memory pool is managed.

12. The memory management method according to claim 11, wherein: The data processing request includes any one of a data writing request, a data reading request, a data cutting request, and a data redundancy protection request; In response to a data writing configuration instruction, configuring a data writing mode corresponding to the data writing request and storing the data writing mode; the data processing mode includes the data writing mode; In response to a data reading configuration instruction, configuring a data reading mode corresponding to the data reading request and storing the data reading mode; the data processing mode includes the data reading mode; In response to a data cutting configuration instruction, configuring a data cutting method corresponding to the data cutting request and storing the data cutting method; the data processing method includes the data cutting method; In response to a data redundancy protection configuration instruction, a data redundancy protection mode corresponding to the data redundancy protection request is configured, and the data redundancy protection mode is stored; the data processing mode includes the data redundancy protection mode.

13. The memory management method according to claim 12, wherein: The managing at least one memory node in the memory pool based on the data processing method includes: If the data processing request is a data writing request, calling the pre-stored data writing method; When the data processing mode to be executed is the data writing mode, obtaining a target data type of the data to be stored in the data writing request; Determine the target memory node type corresponding to the target data type based on a pre-established mapping relationship between the data type and the memory node type; The data to be stored is stored in an idle memory node corresponding to the target memory node type.

14. The memory management method according to claim 12, wherein: The managing at least one memory node in the memory pool based on the data processing method includes: If the data processing request is a data reading request, calling the pre-stored data reading method; When the data processing mode to be executed is the data reading mode, obtaining target identification information of the data to be read in the data reading request; Determining a target storage location corresponding to the target identifier based on a pre-established mapping relationship between identification information and storage locations; Based on the target storage location, the to-be-read data corresponding to the target storage location is read, and the to-be-read data is transmitted to the target computing node.

15. The memory management method according to claim 12, wherein: The managing at least one memory node in the memory pool based on the data processing method includes: If the data processing request is a data cutting request, calling the pre-stored data cutting method; When the data processing mode to be executed is the data cutting mode, obtaining a target cutting size of the data to be cut in the data cutting request; According to the target cutting size, the data to be cut is cut into at least two cutting data; the data size of each cutting data is the target cutting size.

16. The memory management method according to claim 12, wherein: The managing at least one memory node in the memory pool based on the data processing method includes: If the data processing request is a data redundancy protection request, calling the pre-stored data redundancy protection mode; When the data processing mode executed is the data redundancy protection mode, obtaining a target redundancy level of the data to be protected in the data redundancy protection; Determining a target redundancy protection strategy corresponding to the target redundancy level according to a pre-established mapping relationship between redundancy levels and redundancy protection strategies; different redundancy levels correspond to different redundancy protection strategies; Based on the redundancy protection strategy, redundant protection is performed on the data to be protected.

17. A memory management device, characterized in that: include: a calling unit, configured to call a pre-stored data processing method corresponding to a data processing request inputted into a data processing module by a target computing node, wherein different data processing requests correspond to different data processing methods; A management unit is used to manage at least one memory node in the memory pool based on the data processing method.

18. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the memory management method according to any one of claims 11 to 16 when executing the computer program.

19. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the memory management method according to any one of claims 11 to 16.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the memory management method according to any one of claims 11 to 16 are implemented.

Images