Shared memory management system based on memory pool and address mapping

Abstract

The invention relates to a shared memory management system based on a memory pool and address mapping, and belongs to the field of embedded operating systems. The system comprises a meta-information management module which serves as a data center of the system, is responsible for recording and maintaining key information of all memory blocks, is a core link for connecting all other modules, and provides unified data support for function execution of the other modules; the memory management module is a resource basis of the system, is responsible for full-life-cycle management such as memory area initialization, memory block allocation, release and capacity expansion, and provides underlying resource guarantee for shared memory interaction; the address mapping and conversion module and the data processing module form the core of shared memory safety communication, address conversion, cross-process data read-write and synchronous control are achieved, and safety interaction of the shared memory is cooperatively completed. By using the memory pool technology, the memory allocation and release efficiency is improved, and memory fragments are reduced.

Description

Technical Field

[0001] This invention belongs to the field of embedded operating systems, specifically relating to a shared memory management system based on memory pools and address mapping. Background Technology

[0002] In embedded system development, memory management is a crucial aspect, and the main memory management methods include the following.

[0003] 1. Basic Memory Management Basic memory management is performed using the `malloc` and `free` functions in the standard C library. The program requests a memory block of a certain size from the operating system using the `malloc` function, and returns the memory block to the operating system after use by calling the `free` function. This approach is simple to operate, suitable for single-process scenarios with low-frequency memory requests, and can meet the memory management needs of most embedded systems.

[0004] 2. Static memory allocation Static memory allocation is typically defined during the program compilation phase using the `static` keyword or arrays, pre-determining the required memory size and address. At runtime, this fixed memory space is used directly without dynamic allocation or deallocation. Its advantages include fast memory access, no fragmentation issues, and stable memory utilization efficiency, making it suitable for scenarios with fixed memory requirements.

[0005] 3. Memory Pool Management A memory pool is a pre-allocated contiguous memory region used during system initialization. This pool is then divided into fixed-size or dynamically sized memory blocks based on business needs. When a program requests memory, it directly retrieves the appropriate memory block from the memory pool. After use, the block is returned to the pool for reuse, reducing the overhead of calling `malloc` and `free` functions. This approach significantly improves memory allocation and deallocation efficiency, avoids fragmentation issues, and is suitable for high-frequency, similar memory request scenarios.

[0006] 4. Address Mapping Address mapping relies on the Memory Management Unit (MMU) or Input / Output Memory Management Unit (IOMMU) to map physical memory addresses to virtual memory addresses of processes through the operating system's low-level interface (such as the MMU). This allows different virtual address spaces to access the same physical memory, providing a foundation for memory sharing in multi-processor and multi-process collaboration. Address mapping is suitable for scenarios such as multi-task / multi-process concurrency and hardware peripheral access in embedded operating systems.

[0007] Although the above memory management solutions have advantages in specific scenarios, there are still three core defects in complex embedded scenarios with multi-processor and multi-process collaboration (such as distributed industrial control and multi-core intelligent devices), which restrict the reliability, adaptability and execution efficiency of the system, as follows.

[0008] 1. Fragmentation issues in basic memory management Basic memory management relies on the dynamic allocation mechanism of malloc and free. When a program frequently requests and releases memory blocks of different sizes, the memory space will gradually be divided into a large number of non-contiguous memory fragments. Although the total capacity of these fragments is sufficient, the size of a single fragment cannot meet the needs of subsequent large memory requests, resulting in the problem that the total amount of memory is sufficient but the allocation fails.

[0009] 2. The low flexibility of static memory and memory pools Static memory allocates memory blocks whose size and lifespan are fixed at compile time. If the memory requirements change during program execution, the memory capacity cannot be dynamically adjusted, requiring code modification and recompilation, resulting in poor adaptability. Although memory pools support memory block reuse, the total capacity of the memory pool is determined during the initialization phase and cannot be dynamically expanded according to business load, leading to low flexibility in the use of memory pools.

[0010] 3. Complex address mapping management In multiprocessor or multithreaded embedded systems, address mapping management is extremely complex. It requires consideration of issues such as the virtual address spaces of different processors or threads, the allocation and release of physical memory addresses, and the maintenance of address mapping tables. Improper management can easily lead to problems such as address conflicts and memory leaks. Summary of the Invention

[0011] (a) Technical problems to be solved The technical problem this invention aims to solve is how to provide a shared memory management system based on memory pools and address mapping to address issues such as memory fragmentation and low utilization efficiency in memory management.

[0012] (II) Technical Solution To address the aforementioned technical problems, this invention proposes a shared memory management system based on memory pools and address mapping. This shared memory management system includes: a memory management module, an address mapping and translation module, a metadata management module, and a data processing module. The memory management module is responsible for the full lifecycle management of system memory resources. Its core functions cover memory pool lifecycle management, memory block allocation and release, memory expansion, address mapping management and memory metadata tracking and maintenance, providing underlying support for cross-process memory sharing scenarios and the need for efficient memory utilization. The address mapping and translation module obtains a memory block of a specified size from the memory pool to create the memory block. At the same time, it establishes a mapping relationship between the internal address and the external address of the memory block and generates a globally unique mapping identifier to achieve unified memory identification between different address spaces. During the address translation stage, the linked list storing memory metadata is traversed to locate the mapping identifier corresponding to the target memory block. Then, the internal address of the current process is converted into an external address that can be recognized by the target process through the underlying translation interface to ensure the accuracy of cross-process access operations. As a core component of the system, the metadata management module is responsible for the centralized maintenance and management of metadata for various memory blocks. Through a unified data structure and linked list mechanism, it enables the tracking and control of the entire lifecycle of memory resources, providing basic data support for memory allocation, release, address mapping and cross-process interaction operations. As a core component of the system, the data processing module is responsible for cross-process shared memory read / write interaction, data format adaptation, and synchronization control. Through standardized data structures, communication mechanisms, and memory operation specifications, it ensures efficient and secure sharing of memory data between different processes, providing support for cross-process collaborative computing.

[0013] (III) Beneficial Effects This invention proposes a shared memory management system based on memory pools and address mapping. Through systematic design of memory management, metadata tracking, cross-process sharing, and data format standardization, this invention achieves multi-dimensional technical effects, specifically reflected in the following aspects: From a memory resource management perspective, by classifying and managing memory pools, pool blocks, and independent memory blocks, combined with dynamic expansion and precise metadata tracking, memory utilization efficiency is significantly improved. The memory pool reuse mechanism reduces the memory overhead caused by frequent malloc and free operations, the flexible management of independent memory blocks meets diverse memory needs, and the full lifecycle metadata synchronization effectively avoids memory leaks and invalid accesses, ensuring the security and stability of memory usage.

[0014] In terms of cross-process collaboration, the innovative address mapping mechanism successfully solves the core problem of inconsistent address spaces between different processes, significantly improving the accuracy and efficiency of cross-process memory access. Combined with synchronous communication via message queues, it enables low-latency data interaction between processes, providing a reliable foundation for memory sharing in distributed collaborative computing.

[0015] The standardized design of data interaction is one of the important technical effects of this system. By pre-setting task metadata structure and result metadata structure, the metadata format of task description and result feedback in cross-process interaction is standardized. At the same time, the different interaction scenarios (such as computing tasks, data transmission tasks, etc.) are clarified by combining task type enumeration, making the logical flow of task scheduling between processes and the format standard of data transmission clearer, avoiding interaction confusion or parsing errors caused by inconsistent formats.

[0016] For different types of data (including basic data types and complex data types), the system achieves efficient adaptation between memory blocks and structured data through a dedicated data conversion mechanism: whether it is a basic type such as integers and floating-point numbers, or a complex type such as arrays and pointers, the format conversion can be completed through a standardized conversion process. This not only ensures the consistency of data format transmission across processes and modules, but also greatly reduces the complexity of data parsing. The correct identification and processing of data can be achieved without the need for additional development of adaptation logic.

[0017] Regarding system scalability and compatibility, this solution achieves multi-resource adaptation and problem localization optimization through two types of core enumerations: one is the computation type enumeration and executor type enumeration, which clearly defines the classification criteria for different computational needs and execution carriers, and can flexibly adapt to various computing resources such as CPU and GPU. When adding computing devices in the future, only the enumeration type needs to be expanded to achieve compatibility, without the need to refactor the core logic; the other is the error status enumeration, which defines error type identifiers for the client and server respectively (such as request timeout, memory allocation failure, etc.), providing a clear basis for problem localization during system operation, and facilitating debugging and maintenance when expanding functions in the future.

[0018] Furthermore, the system incorporates a packet-splitting transmission mechanism in the memory sharing stage. By setting fields such as packet flags, group identifiers, and the total number of fragments in the metadata structure, it supports splitting large data blocks exceeding the capacity of a single memory block into multiple smaller fragments for transmission. After transmission, these fragments are reassembled into complete data, effectively solving the memory limitation problem when sharing large-size data and significantly improving the system's adaptability to massive data processing scenarios. In summary, this invention achieves a comprehensive effect of efficient memory utilization, reliable cross-process communication, and standardized data interaction through refined resource management, cross-process collaborative optimization, format standardization, and scalable design, providing stable, efficient, and easily scalable technical support for multi-process collaborative computing scenarios. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the shared memory management system of the present invention; Figure 2 Diagram showing the components of the memory management module; Figure 3 Here is a flowchart of the memory management module; Figure 4 Diagram showing the components of the metadata management module; Figure 5 Diagram showing the composition of the address mapping and conversion module; Figure 6 This is a diagram showing the components of the data processing module. Detailed Implementation

[0020] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0021] The purpose of this invention is to provide a shared memory management system and method based on memory pools and address mapping. By using memory pool technology, it improves the efficiency of memory allocation and deallocation, reduces memory fragmentation, and provides address mapping and translation functions to facilitate memory sharing with other processors. Specifically, the objectives of this invention include: 1. Reduce memory fragmentation and improve memory utilization efficiency. This shared memory management system pre-allocates a fixed area of ​​memory as a memory pool by enabling region memory management. When memory is needed, it prioritizes obtaining memory blocks from this memory pool instead of directly allocating them using the malloc function. This concentrates memory allocation operations within the memory pool, avoiding the large amount of memory fragmentation caused by frequent use of malloc.

[0022] 2. Implement dynamic expansion of the memory pool to meet the actual needs of program runtime. During program execution, memory requirements may change dynamically. To ensure the system can promptly meet these changing needs, this shared memory management system implements a dynamic expansion mechanism for the memory pool. When the available memory space in the memory pool is insufficient to meet new memory allocation requests, the system automatically triggers an expansion operation, increasing the size of the memory pool.

[0023] 3. Simplify address mapping management and improve its efficiency and reliability. Address mapping is a crucial function in a shared memory management system. The operating system creates memory blocks and establishes a mapping relationship between internal and external addresses, generating unique mapping identifiers. During address translation, the target memory block's mapping identifier is found by traversing the metadata linked list, and the internal address is converted into an external address recognizable by the target process, facilitating communication between different components.

[0024] 4. Provides an effective data sharing and synchronization mechanism to ensure the consistency and correctness of shared data. In a multi-threaded or multi-process environment, data sharing and synchronization are critical issues. This shared memory management system provides an effective data sharing and synchronization mechanism through a shared memory communication module to ensure the consistency and correctness of shared data. This invention focuses on a shared memory management system, aiming to solve problems such as memory fragmentation and low efficiency in memory management, while realizing dynamic expansion of the memory pool, simplifying address mapping management, and providing an efficient data sharing and synchronization mechanism.

[0025] The shared memory management system of the present invention includes four core modules: a memory management module, an address mapping and translation module, a metadata management module, and a data processing module. The system composition is shown in Figure 1. Each module is functionally independent yet deeply collaborative.

[0026] The metadata management module, as the data hub of the system, is responsible for recording and maintaining key information of all memory blocks. It is the core link connecting all other modules and provides unified data support for the function execution of other modules. The memory management module is the resource foundation of the system. It is responsible for the entire lifecycle management of memory regions, including initialization, allocation, release, and expansion, and provides underlying resource guarantees for shared memory interaction. The address mapping and conversion module and the data processing module constitute the core of secure communication in shared memory, realizing address conversion, cross-process data reading and writing, and synchronization control, and working together to complete secure interaction in shared memory.

[0027] These modules not only have a clear division of labor in the macro architecture, but also contain sophisticated design mechanisms. The following text will focus on the functional implementation, process logic and implementation details of each module.

[0028] 1. Memory Management Module The memory management module is responsible for the full lifecycle management of system memory resources. Its core functions cover five core components: memory pool lifecycle management, memory block allocation and release, memory expansion, address mapping management, and memory metadata tracking and maintenance. It provides underlying support for cross-process memory sharing scenarios and the need for efficient memory utilization. The composition of the shared memory management module is shown in Figure 2.

[0029] Memory pool lifecycle management comprises two parts: memory pool initialization and memory pool destruction. During memory region initialization, the memory management module creates a globally unified memory region and generates a unique identifier to distinguish it from other system memory resources, thus completing the memory pool creation. Simultaneously, it initializes a metadata list specifically for recording the core information of all memory blocks (including memory pools, in-pool memory blocks, and independent memory blocks). Memory blocks are categorized into three types: memory pools, in-pool memory blocks, and independent memory blocks, each corresponding to different management strategies. The memory pool serves as the basic memory resource pool, uniformly handling the allocation and reclamation of blocks within the pool. In-pool memory blocks are allocated from the memory pool as needed, released, and then recycled back to the memory pool for reuse. Independent memory blocks directly request independent memory space from the system, suitable for large-size or long-term memory requirements. When a memory region needs to be destroyed, the module releases all allocated memory block resources and clears the metadata list, ensuring no memory residue or invalid metadata, thus completely avoiding the risk of resource leaks.

[0030] Memory block allocation and deallocation includes: memory allocation and memory deallocation. During memory allocation, the memory management module employs two different memory allocation strategies depending on whether the system has enabled parallel memory management. If parallel management is enabled, a memory segment of a specified size is allocated from the initialized global memory region; if not enabled, independent memory is directly requested from the system. The memory allocation strategies of the memory management module are shown in Figure 3. Regardless of the path used, the metadata (type, address, size, etc.) of the newly allocated memory block is synchronously stored in the metadata linked list, enabling allocation and tracking. For memory blocks that require cross-process access, after basic allocation, the module further establishes a mapping relationship between internal and external addresses, generating a unique mapping identifier to ensure that other processes can identify and access the target memory block through this identifier, providing an address basis for cross-process collaboration.

[0031] During the memory release phase, the module executes differentiated reclamation logic based on the memory block type. For memory blocks within a pool, they are reclaimed to the corresponding memory pool after release for subsequent reallocation and utilization. For independent memory blocks, the memory space they occupy is directly released to the system. Simultaneously with the release operation, the module removes the metadata of the memory block from the metadata list and destroys its associated address mapping relationship, ensuring complete synchronization of the lifecycle of memory resources with metadata and mapping relationships, and avoiding address conflicts caused by invalid mappings.

[0032] In addition, the module supports memory expansion, allowing adjustment of the size of allocated memory blocks according to business needs. For blocks within a memory pool, the required expansion space is allocated from the memory pool; for independent memory blocks, additional memory is directly requested from the system and merged with the original memory block, and the memory block size record in the metadata linked list is updated synchronously after expansion.

[0033] Address mapping management includes: In terms of cross-process access support, the module creates internal and external address mappings of memory blocks and generates unique identifiers through a dedicated mechanism. During address translation, it traverses the metadata linked list, matches the internal address of the target memory block and extracts the corresponding mapping identifier, and finally completes the translation from internal address to external address, ensuring the accuracy of cross-process address resolution.

[0034] Memory metadata tracking and maintenance includes: The module also realizes the traceability management of the entire life cycle of memory blocks through metadata linked lists, and records key information such as the type, address, size, and mapping relationship of all memory blocks in real time; at the same time, it provides memory usage statistics function, which can calculate the current total memory usage, the usage ratio of various types of memory blocks, and other data in real time, providing data support for system memory resource scheduling and optimization, and ultimately realizing efficient allocation, safe reclamation and cross-process collaborative management of memory resources.

[0035] 2. Address Mapping and Conversion Module To address memory access errors caused by address space inconsistencies across different processors or processes, this invention designs an address mapping and translation module. This module retrieves a memory block of a specified size from the memory pool to create the memory block, simultaneously establishing a mapping relationship between the internal address (the virtual address of the current process) and the external address (the virtual address of the target process) of the memory block, and generating a globally unique mapping identifier to achieve unified memory identification across different address spaces. During the address translation phase, the module traverses the linked list storing memory metadata, locates the mapping identifier corresponding to the target memory block, and then converts the internal address of the current process into an external address recognizable by the target process through the underlying translation interface, ensuring the accuracy of cross-process access operations. The system composition of the address mapping and translation module is shown in Figure 4.

[0036] The core functions of the address mapping and translation module focus on two main aspects: address mapping creation and internal / external address translation. The address mapping creation process requires receiving the internal address, external address, and memory length of the memory block as basic parameters, calling the underlying interface to initialize the mapping relationship, and generating a unique mapping identifier. Subsequently, the mapping information containing the mapping identifier and the correspondence between the internal and external addresses is associated with the metadata structure of the memory block and included in the metadata linked list for unified management. When the system allocates a memory block that needs to be accessed across processes, it will automatically trigger the above address mapping creation process according to the type of memory block (a block within the memory pool or a directly requested independent memory block), ensuring that such memory blocks have cross-domain access capabilities from the allocation stage.

[0037] The core objective of address translation is to achieve accurate conversion from internal addresses of the current process to external addresses of the target process. Specifically, the module traverses the metadata list, examining the metadata of each memory block stored in the list, filtering out memory blocks with cross-domain access attributes (i.e., pooled blocks and independent memory blocks). By determining whether the target internal address is within the address range of the memory block, the module locates the matching memory block, extracts its corresponding mapping identifier, and then calls the underlying translation interface to complete the address translation. If no matching mapping is found after traversal, or if an exception occurs during the address translation process, the module automatically records an error log and returns an invalid address to avoid access risks caused by erroneous addresses.

[0038] Furthermore, the address mapping and translation module, in collaboration with the metadata linked list, achieves closed-loop management of the entire lifecycle of mapping relationships: when a memory block is allocated and a mapping is created, its mapping information is synchronously recorded in the metadata linked list; when a memory block is reclaimed and released, the corresponding mapping information is cleaned up and deleted from the linked list along with the memory block metadata, ensuring that mapped resources are free of residue and leakage. Simultaneously, relying on the global uniqueness of the mapping identifier, the module can accurately distinguish the mapping relationships of different memory blocks. Even in cases where memory block address ranges overlap, the mapping accuracy can be guaranteed through the dedicated mapping identifier. This provides stable address consistency support for data transmission and computation result interaction across processes, becoming a key technical mechanism for the system to achieve collaborative work among multiple processors, processes, and other entities.

[0039] 3. Meta-information Management Module As a core component of the system, the metadata management module is responsible for the centralized maintenance and management of metadata for various memory blocks. Through a unified data structure and linked list mechanism, it enables the tracking and control of the entire lifecycle of memory resources, and provides basic data support for operations such as memory allocation, release, address mapping and cross-process interaction. The system composition of the metadata management module is shown in Figure 5.

[0040] This module uses a specific structure as its core data carrier. This structure contains key metadata about memory blocks: it distinguishes memory block categories through a type field (1 represents a memory pool, 2 represents a memory block within a pool, and 3 represents an independent memory block); it records the internal starting address, external mapped address, unique identifier of the associated address mapping, and the size of the memory block. This metadata is uniformly stored and managed through a global linked list, which acts as a global container throughout the entire lifecycle of a memory block, from creation to destruction.

[0041] In terms of specific implementation, the module maintains various types of metadata through a differentiated mechanism. For the memory pool, it records its starting address and total length, without involving address mapping, and only serves as a marker for the basic memory region. For memory blocks allocated from the memory pool, after allocation, it associates their internal address, external mapping address, and length, and generates a unique identifier for the address mapping, thus completing the binding of metadata with address mapping. For independent memory blocks, in addition to recording the address and length, it also generates a unique identifier for the address mapping, and accumulates the total size of independent memory blocks into the global memory statistics.

[0042] The initialization of metadata is synchronized with the creation of memory regions. When initializing the global memory region, a global linked list for storing metadata is created simultaneously, and metadata for the initial memory pool is added to ensure that basic memory resources are managed from the outset. During memory allocation, each newly allocated memory block (whether it is a block within a pool or an independent memory block) generates a corresponding metadata instance and stores it in the global linked list, achieving a management effect of allocation and tracking.

[0043] The module also provides metadata query support for other functional modules. For example, during address translation, it traverses the global linked list, matches the corresponding memory block metadata based on the internal address, and extracts the unique identifier of the address mapping to complete the conversion from internal to external address. This process relies on the type (only pool blocks and independent memory blocks) and address range recorded in the metadata to ensure the accuracy of cross-process access. When memory is released, the module removes the corresponding metadata node from the global linked list to ensure that the metadata is always consistent with the actual memory state, avoiding management chaos caused by invalid information residue.

[0044] 4. Data Processing Module As a core component of the system, the data processing module is responsible for cross-process shared memory read / write interaction, data format adaptation, and synchronization control. Through standardized data structures, communication mechanisms, and memory operation specifications, it ensures efficient and secure sharing of memory data between different processes, providing support for cross-process collaborative computing.

[0045] This module uses two types of core data structures to implement its functions: one type serves as the context carrier for shared memory communication, encapsulating message queue identifiers, local semaphores, shared memory addresses, and mutexes to achieve synchronization control and resource identification; the other type serves as the metadata carrier for data transmission, recording the shared memory address, external starting address, and memory length to achieve precise location and description of data blocks. The system composition of the data processing module is shown in Figure 6.

[0046] In the shared memory write process, when the memory pool is enabled, access control is first implemented using semaphores and mutexes to avoid concurrent operation conflicts. Then, a shared memory block is allocated, the data to be shared is copied to this memory block, and information such as the memory address and length is encapsulated in a metadata carrier. Finally, the metadata is sent via a message queue to notify the completion of the data write. Throughout this process, the data is stored in the form of raw memory blocks, and the metadata is transmitted through a fixed structure to ensure that the receiver can accurately locate the data.

[0047] The shared memory read process also relies on mutexes to ensure thread safety. It retrieves the metadata carrier through a message queue and extracts the shared memory address. After reading, the mutex is released, notifying the writer that writing can continue. This process directly locates the shared memory block by parsing a fixed-format metadata structure, achieving efficient data reading.

[0048] The core of the shared memory release mechanism is to release the shared memory block and record the release status through logs to ensure that shared resources are reclaimed in a timely manner after use, thus avoiding memory leaks.

[0049] In addition, the module supports cross-process address consistency through an address mapping mechanism. It creates internal and external address mappings for memory blocks and generates unique identifiers, associating local memory addresses with external process-visible addresses. This ensures that different processes resolve the same shared memory block in a consistent manner, which is a key format adaptation step in cross-process memory sharing.

[0050] For shared data format parsing, the module has the ability to convert raw data in shared memory into structured data. By allocating local memory, it copies and parses the data in shared memory according to a preset structure, providing the upper-layer application with a data format that meets expectations.

[0051] The key features of this invention are mainly reflected in the refined management of memory resources, efficient support for cross-process collaboration, and standardized design of interaction formats, specifically including: First, an innovative categorized memory management strategy is adopted, dividing memory blocks into three types: memory pools, pool blocks, and independent memory blocks. Different management interfaces enable differentiated allocation and release logic, which improves the reuse efficiency of memory pools and ensures the flexible use of independent memory blocks. At the same time, the metadata management module enables unified tracking of the entire lifecycle of various types of memory blocks.

[0052] Secondly, a cross-process memory sharing address mapping mechanism was constructed. The internal and external address mappings of memory blocks were created through the address mapping and conversion module, and a unique port ID was generated. This solved the problem of inconsistent memory address spaces between different processes and provided address consistency support for cross-process data access. It is the core technical foundation for realizing process collaboration.

[0053] Furthermore, a unified metadata management system was designed to centrally store key metadata such as the type, address, size, and mapping relationship of memory blocks. By adding and removing linked list nodes, the metadata is synchronized with the actual memory status in real time, providing reliable data for memory allocation, release, address translation, and other operations, and realizing the traceability and controllability of memory resources.

[0054] In addition, the interaction format and task type definition have been standardized, clarifying the task types for various interaction scenarios such as data transmission, computation tasks, and result confirmation. The encapsulation format of task data has been standardized, and multiple error enumeration states and computation-related type enumerations have been defined, providing a unified format standard and interaction specification for efficient communication and task collaboration between processes within the system.

[0055] Finally, through the organic combination of memory management, metadata tracking, address mapping and standardized formats, a complete cross-process memory sharing and collaborative computing support system is formed, which not only ensures the efficient use and secure management of memory resources, but also improves the consistency and reliability of data interaction and task execution between different processes.

[0056] This invention achieves multi-dimensional technical effects through a systematic design of memory management, metadata tracking, cross-process sharing, and data format standardization, specifically reflected in the following aspects: From a memory resource management perspective, by classifying and managing memory pools, pool blocks, and independent memory blocks, combined with dynamic expansion and precise metadata tracking, memory utilization efficiency is significantly improved. The memory pool reuse mechanism reduces the memory overhead caused by frequent malloc and free operations, the flexible management of independent memory blocks meets diverse memory needs, and the full lifecycle metadata synchronization effectively avoids memory leaks and invalid accesses, ensuring the security and stability of memory usage.

[0057] In terms of cross-process collaboration, the innovative address mapping mechanism successfully solves the core problem of inconsistent address spaces between different processes, significantly improving the accuracy and efficiency of cross-process memory access. Combined with synchronous communication via message queues, it enables low-latency data interaction between processes, providing a reliable foundation for memory sharing in distributed collaborative computing.

[0058] The standardized design of data interaction is one of the important technical effects of this system. By pre-setting task metadata structure and result metadata structure, the metadata format of task description and result feedback in cross-process interaction is standardized. At the same time, the different interaction scenarios (such as computing tasks, data transmission tasks, etc.) are clarified by combining task type enumeration, making the logical flow of task scheduling between processes and the format standard of data transmission clearer, avoiding interaction confusion or parsing errors caused by inconsistent formats.

[0059] For different types of data (including basic data types and complex data types), the system achieves efficient adaptation between memory blocks and structured data through a dedicated data conversion mechanism: whether it is a basic type such as integers and floating-point numbers, or a complex type such as arrays and pointers, the format conversion can be completed through a standardized conversion process. This not only ensures the consistency of data format transmission across processes and modules, but also greatly reduces the complexity of data parsing. The correct identification and processing of data can be achieved without the need for additional development of adaptation logic.

[0060] Regarding system scalability and compatibility, this solution achieves multi-resource adaptation and problem localization optimization through two types of core enumerations: one is the computation type enumeration and executor type enumeration, which clearly defines the classification criteria for different computational needs and execution carriers, and can flexibly adapt to various computing resources such as CPU and GPU. When adding computing devices in the future, only the enumeration type needs to be expanded to achieve compatibility, without the need to refactor the core logic; the other is the error status enumeration, which defines error type identifiers for the client and server respectively (such as request timeout, memory allocation failure, etc.), providing a clear basis for problem localization during system operation, and facilitating debugging and maintenance when expanding functions in the future.

[0061] Furthermore, the system incorporates a packet-splitting transmission mechanism in the memory sharing stage. By setting fields such as packet flags, group identifiers, and the total number of fragments in the metadata structure, it supports splitting large data blocks exceeding the capacity of a single memory block into multiple smaller fragments for transmission. After transmission, these fragments are reassembled into complete data, effectively solving the memory limitation problem when sharing large-size data and significantly improving the system's adaptability to massive data processing scenarios. In summary, this invention achieves a comprehensive effect of efficient memory utilization, reliable cross-process communication, and standardized data interaction through refined resource management, cross-process collaborative optimization, format standardization, and scalable design, providing stable, efficient, and easily scalable technical support for multi-process collaborative computing scenarios. The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A shared memory management system based on memory pools and address mapping, characterized in that, The shared memory management system includes: a memory management module, an address mapping and translation module, a metadata management module, and a data processing module; The memory management module is responsible for the full lifecycle management of system memory resources. Its core functions cover memory pool lifecycle management, memory block allocation and release, memory expansion, address mapping management and memory metadata tracking and maintenance, providing underlying support for cross-process memory sharing scenarios and the need for efficient memory utilization. The address mapping and translation module obtains a memory block of a specified size from the memory pool to create the memory block. At the same time, it establishes a mapping relationship between the internal address and the external address of the memory block and generates a globally unique mapping identifier to achieve unified memory identification between different address spaces. During the address translation stage, the linked list storing memory metadata is traversed to locate the mapping identifier corresponding to the target memory block. Then, the internal address of the current process is converted into an external address that can be recognized by the target process through the underlying translation interface to ensure the accuracy of cross-process access operations. As a core component of the system, the metadata management module is responsible for the centralized maintenance and management of metadata for various memory blocks. Through a unified data structure and linked list mechanism, it enables the tracking and control of the entire lifecycle of memory resources, providing basic data support for memory allocation, release, address mapping and cross-process interaction operations. As a core component of the system, the data processing module is responsible for cross-process shared memory read / write interaction, data format adaptation, and synchronization control. Through standardized data structures, communication mechanisms, and memory operation specifications, it ensures efficient and secure sharing of memory data between different processes, providing support for cross-process collaborative computing.

2. The shared memory management system based on memory pool and address mapping as described in claim 1, characterized in that, The memory pool lifecycle management of the memory management module includes two parts: memory pool initialization and memory pool destruction. During memory region initialization, the memory management module creates a globally unified memory region and generates a unique identifier to distinguish it from other system memory resources, thus completing the memory pool creation. Simultaneously, it initializes a metadata list specifically for recording the core information of all memory blocks. Memory blocks are divided into three categories: memory pool, memory blocks within the pool, and independent memory blocks, corresponding to different management strategies. The memory pool serves as the basic memory resource pool, uniformly handling the allocation and reclamation of blocks within the pool. Memory blocks within the pool are allocated from the memory pool as needed, released, and then recycled back to the memory pool for reuse. Independent memory blocks directly request independent memory space from the system, suitable for large-size or long-term memory requirements. When a memory region needs to be destroyed, the module releases all allocated memory block resources and clears the metadata list, ensuring no memory residue or invalid metadata, thus completely avoiding the risk of resource leakage.

3. The shared memory management system based on memory pool and address mapping as described in claim 1, characterized in that, The memory block allocation and release of the memory management module includes: a memory allocation phase and a memory release phase; During memory allocation, the memory management module employs two different memory allocation strategies depending on whether the system has enabled parallel memory management. If parallel management is enabled, a memory segment of a specified size is allocated from the initialized global memory region. If not enabled, independent memory is requested directly from the system. The metadata of newly allocated memory blocks is synchronously stored in the metadata linked list, enabling allocation and tracking. For memory blocks that need to be accessed across processes, after the basic allocation is completed, the module further establishes a mapping relationship between internal and external addresses, generating a unique mapping identifier to ensure that other processes can identify and access the target memory block through this identifier, providing an address basis for cross-process collaboration. During the memory release process, the module executes differentiated reclamation logic based on the memory block type. For memory blocks within the pool, they are reclaimed to the corresponding memory pool after release for subsequent reallocation and utilization. For independent memory blocks, the memory space they occupy is directly released to the system. Simultaneously with the release operation, the module removes the metadata of the memory block from the metadata chain and destroys its associated address mapping relationship, ensuring that the lifecycle of memory resources and metadata and mapping relationships are completely synchronized.

4. The shared memory management system based on memory pool and address mapping as described in claim 1, characterized in that, The memory management module supports memory expansion functionality, adjusting the size of allocated memory blocks according to business needs; for blocks within the pool, it supplements the required expansion space from the memory pool; for independent memory blocks, it directly requests additional memory from the system and merges it with the original memory block, and after expansion, it synchronously updates the memory block size record in the metadata chain list.

5. The shared memory management system based on memory pool and address mapping as described in claim 1, characterized in that, The address mapping management of the memory management module includes: in supporting cross-process access, the module creates internal and external address mappings of memory blocks and generates unique identifiers. During address translation, it traverses the metadata linked list, matches the internal address of the target memory block and extracts the corresponding mapping identifier, and finally completes the translation from internal address to external address, ensuring the accuracy of cross-process address resolution.

6. The shared memory management system based on memory pool and address mapping as described in claim 1, characterized in that, The memory management module's memory metadata tracking and maintenance includes: implementing full lifecycle traceability management of memory blocks through a metadata linked list, recording key information such as the type, address, size, and mapping relationship of all memory blocks in real time; and providing memory usage statistics functions to calculate the current total memory usage and the usage ratio of various types of memory blocks in real time, providing data support for system memory resource scheduling and optimization.

7. The shared memory management system based on memory pool and address mapping as described in claim 1, characterized in that, The address mapping and translation module's address mapping creation process requires receiving the internal address, external address, and memory length of the memory block as basic parameters, calling the underlying interface to initialize the mapping relationship, and generating a unique mapping identifier. Subsequently, the mapping information, which includes the mapping identifier and the correspondence between the internal and external addresses, is associated with the metadata structure of the memory block and included in the metadata linked list for unified management. When the system allocates a memory block that needs to be accessed across processes, it will automatically trigger the above address mapping creation process according to the type of memory block, ensuring that such memory blocks have cross-domain access capabilities from the allocation stage.

8. The shared memory management system based on memory pool and address mapping as described in claim 1, characterized in that, The address mapping and conversion module aims to achieve accurate conversion between internal addresses of the current process and external addresses of the target process. Specifically, it traverses the metadata linked list, checks the metadata of each memory block stored in the list, filters out memory blocks with cross-domain access attributes, locates the matching memory block by determining whether the target internal address is within the address range of the memory block, extracts its corresponding mapping identifier, and then calls the underlying conversion interface to complete the address conversion. If no matching mapping relationship is found after traversal, or if an exception occurs during the address conversion process, the module will automatically record an error log and return an invalid address to avoid access risks caused by erroneous addresses.

9. The shared memory management system based on memory pool and address mapping as described in claim 1, characterized in that, The metadata management module uses a specific structure as the core data carrier. This structure contains key metadata of memory blocks: it distinguishes memory block categories through a type field; it records the internal starting address, external mapped address, unique identifier of associated address mapping, and the size of the memory block; this metadata is uniformly stored and managed through a global linked list, which serves as a global container throughout the entire lifecycle of the memory block from creation to destruction. The metadata management module maintains various types of metadata through a differentiated mechanism. For the memory pool, it records its starting address and total length, without address mapping, and only serves as a marker for the basic memory region. For memory blocks allocated from the memory pool, it associates their internal address, external mapping address, and length after allocation, and generates a unique identifier for the address mapping, thus completing the binding of metadata with address mapping. For independent memory blocks, in addition to recording the address and length, a unique identifier for the address mapping is also generated, and the total size of independent memory blocks is accumulated in the global memory statistics; The initialization of metadata is synchronized with the creation of memory regions. When initializing the global memory region, a global linked list for storing metadata is created simultaneously, and metadata of the initial memory pool is added to ensure that basic memory resources are managed from the beginning. During memory allocation, each newly allocated memory block generates a corresponding metadata instance and stores it in the global linked list, achieving the management effect of allocation and tracking.

10. The shared memory management system based on memory pool and address mapping as described in claim 1, characterized in that, The data processing module uses two types of core data structures to implement its functions: one type serves as the context carrier for shared memory communication, encapsulating message queue identifiers, local semaphores, shared memory addresses, and mutexes to achieve synchronization control and resource identification; the other type serves as the metadata carrier for data transmission, recording the shared memory address, external starting address, and memory length to achieve precise location and description of data blocks. In the shared memory write process, when the memory pool is enabled, access control is first implemented through semaphores and mutexes to avoid concurrent operation conflicts. Then, a shared memory block is allocated, the data to be shared is copied to the memory block, and the memory address and length information are encapsulated in the metadata carrier; finally, the metadata is sent through the message queue to complete the notification of data writing. During this process, data is stored in the form of raw memory blocks, and metadata is transmitted through a fixed structure to ensure that the receiver can accurately locate the data. The shared memory reading process also relies on mutexes to ensure thread safety. Meta-information carriers are obtained through message queues to extract the shared memory address. After reading is completed, the mutex is released, and the writing party is notified to continue writing. This process directly locates the shared memory block by parsing a fixed-format meta-information structure, achieving efficient data reading. The core of the memory sharing release mechanism is to release the shared memory block and record the release status through logs to ensure that the shared resources are reclaimed in a timely manner after use, thus avoiding memory leaks. Cross-process address consistency is supported through an address mapping mechanism. This mechanism creates internal and external address mappings for memory blocks and generates unique identifiers, associating local memory addresses with addresses visible to external processes, thus ensuring consistent address resolution for the same shared memory block across different processes.

Images