A memory management method and apparatus

Abstract

The embodiment of the present application discloses a memory management method and device. The data of the buffer whose data type is mainly read data is copied to the local memory of each processor of the processor cluster, and the permission of the page table is marked as read-only, so that the processor generates a write violation when writing a page, and in the processor generating a write violation when writing the same page, the write operation of each processor is completed by one of the processors, and other processors acquire data by remotely accessing the processor performing the write operation. The pre-fetching of the write data can be realized under the premise of ensuring the data consistency between the processors, and the processor performance of the computer is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a memory management method and apparatus. Background Technology

[0002] Unified Memory is a memory management technology for data center systems that allows the CPU and GPU to share the same visible memory address space, simplifying the programming model and reducing the complexity of traditional explicit data transfer programming. Unified Memory logically provides a shared memory space for the CPU and GPU, eliminating the need for programmers to explicitly manage data transfer between host (CPU) memory and device (GPU) memory. However, very large datasets or high-performance applications may require manual optimization of data transfer and memory usage to fully utilize hardware capabilities.

[0003] In existing technologies, manually optimizing data transfer and memory usage typically involves pre-copying read-only data to the GPU's local memory. However, due to GPU consistency requirements, this method is only suitable for read-only data; it becomes inapplicable whenever write data is involved. Existing technologies lack optimization methods for write data. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a memory management method and apparatus that can prefetch write data while ensuring data consistency among processors, thereby improving the processor performance of the computer.

[0005] In a first aspect, embodiments of the present invention provide a memory management method, the method comprising:

[0006] The data in the buffer with the primary read data type is copied to the local memory of each processor in the processor cluster, and the page table permissions are marked as read-only so that a write violation occurs when a processor writes to the page.

[0007] Among the processors that generate write violations when writing to the same page, one processor completes the write operations of each processor, while other processors obtain data by remotely accessing the processor that performed the write operation.

[0008] The processor cluster includes one or more CPUs, and / or one or more GPUs.

[0009] In some embodiments, copying the data from the buffer whose data type is primary read data to the local memory of each processor in the processor cluster includes:

[0010] The application determines the data type of the data in the buffer;

[0011] In response to the data type being primary read data, the application sends a first notification to the software;

[0012] Before the kernel starts, the data in the buffer is copied to the local memory of each processor in the processor cluster via direct memory access through the driver.

[0013] In some embodiments, the method further includes:

[0014] When a thread block is writing data, it controls other thread blocks to be in an idle state when accessing the same range of data.

[0015] In some embodiments, among the processors that generate write violations when writing to the same page, one processor completes the write operations of each processor, and other processors obtain data by remotely accessing the processor performing the write operation.

[0016] In response to a write violation detected before a missing request is retrieved from the missing status holding register, the operation on the relevant entry is stopped and the entry is set to a pending state.

[0017] A second notification is sent to the software to indicate that a write violation has occurred.

[0018] In some embodiments, among the processors that generate write violations when writing to the same page, the write operations of each processor are completed by one of the processors, and the other processors obtain data by remotely accessing the processor performing the write operation.

[0019] In response to the second notification, the corresponding process is removed from the running list of each processor via context switching;

[0020] Send commands to clear the translation backstop buffer and clear the cache.

[0021] In some embodiments, among the processors that generate write violations when writing to the same page, the write operations of each processor are completed by one of the processors, and the other processors obtain data by remotely accessing the processor performing the write operation.

[0022] The driver changes the buffer pages of the processor that it has already prefetched into local memory to page table entry granularity;

[0023] The permission status of each processor is updated at the granularity of the page table entries.

[0024] In some embodiments, updating the permission status of each processor at the page table entry granularity includes:

[0025] For the page being written to, grant write permission to the first processor that detects the write violation;

[0026] Grant other processors that write to the same page permission to retrieve data by remotely accessing the first processor;

[0027] The page tables of processors that have not accessed pages will be marked as non-resident.

[0028] In some embodiments, the method further includes:

[0029] Re-parse the missing request and obtain the physical address.

[0030] In some embodiments, the method further includes:

[0031] In response to an access request for a non-resident page, new data is retrieved from the data update processor and copied to the local memory of other processors.

[0032] In a second aspect, embodiments of the present invention provide a memory management device, the device comprising:

[0033] The prefetch unit is used to copy data from the buffer whose data type is primary read data to the local memory of each processor in the processor cluster, and to mark the page table permissions as read-only so that the processor will generate a write violation when writing to the page.

[0034] The operation unit is used to complete the write operations of each processor in the process of writing to the same page, and other processors obtain data by remotely accessing the processor that performed the write operation.

[0035] Thirdly, embodiments of the present invention provide an electronic device including a processor cluster and a memory, the processor cluster including one or more CPUs and one or more GPUs, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the CPU and / or GPU to implement the method as described in the first aspect.

[0036] Fourthly, embodiments of the present invention provide a computer program product comprising a computer program, wherein when the computer program is run on a computer, the computer executes the method described in the first aspect above.

[0037] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a CPU and / or a GPU, implement the method described in the first aspect.

[0038] The technical solution of this invention copies data from a buffer primarily used for reading data to the local memory of each processor in a processor cluster, and marks the page table permissions as read-only. This causes write violations when processors attempt to write to a page. Among the processors that experience write violations when writing to the same page, one processor completes the write operation for each processor, while other processors obtain the data by remotely accessing the processor performing the write operation. This allows for prefetching of write data while ensuring data consistency among processors, thereby improving the computer's processor performance. Attached Figure Description

[0039] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0040] Figure 1 This is a schematic diagram of a memory management system according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of a comparative memory management system according to an embodiment of the present invention;

[0042] Figure 3 This is a flowchart of a memory management method according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of data conversion in a memory management system according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of data conversion in a memory management system according to an embodiment of the present invention;

[0045] Figure 6 This is a flowchart of a memory management method according to another embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram of a memory management device according to an embodiment of the present invention. Detailed Implementation

[0047] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0048] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0049] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0050] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0051] The CPU (Central Processing Unit) is the core of a computer system for computation and control, and is the final execution unit for information processing and program execution. The GPU (Graphics Processing Unit), also known as a display core, visual processor, or display chip, is a microprocessor specifically designed for performing image and graphics-related computations in personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones).

[0052] In traditional computer architectures, the CPU and GPU have their own independent memory spaces. The CPU's memory is called host memory, and the GPU's memory is called device memory. Host memory, also known as system memory or RAM (Random Access Memory), is the primary component in a computer used to store running programs and data. The CPU executes instructions and processes data by accessing host memory. The capacity and speed of host memory have a significant impact on the overall performance of a computer, especially when handling complex tasks and multitasking environments. Device memory, also known as video memory or graphics memory, is memory specifically designed for graphics processing. It stores images, textures, and other related information required by the GPU when processing graphics data. Device memory typically has high bandwidth and low latency to support the high-speed graphics processing and rendering needs of the GPU. In modern computing, GPUs are used not only for graphics processing but also extensively in fields such as parallel computing and deep learning; therefore, the performance and capacity of device memory have become increasingly important. Data needs to be transferred between host memory and device memory. This increases the complexity and latency of data transfer.

[0053] Unified memory allows the CPU and GPU to share the same memory address space, thus eliminating the need for data transfer. The CPU and GPU can directly access data in shared memory, improving data transfer efficiency.

[0054] Unified Memory is a memory management technology that allows shared memory between the host (CPU) and the device (GPU). In systems using Unified Memory, data can be moved between the host and the device only once, instead of requiring two or more data transfers as before. This significantly improves computational efficiency and reduces memory usage. Unified Memory is a new memory management mechanism that, instead of specific hardware memory (such as device memory or host memory), provides a virtual memory space accessible by the system's processor (CPU or GPU) and guarantees memory consistency. This means there's no longer a need to worry about the specific location of data or manually transfer data back and forth between host memory and device memory, greatly simplifying program development.

[0055] Unified memory describes a system that provides memory management services for various programs, enabling processors such as CPUs and GPUs to access a single, coherent memory image with a shared address space. Its principles are based on virtual memory mapping, memory migration, on-demand migration, migration control, and memory access consistency.

[0056] Virtual memory mapping: The operating system allocates a virtual memory space for the entire system, which can be accessed by both the CPU and GPU. This allows programmers to use the same pointers to reference data on both the CPU and GPU.

[0057] Memory migration: When a program runs on the CPU or GPU, the relevant data may not be in the processor's physical memory. In this case, the data needs to be migrated from one location to another. This process is called page faulting, and the operating system and hardware system will automatically perform the necessary data migration.

[0058] Demand Paging: Unified Memory uses a demand paging strategy, meaning that paging only occurs when the program needs to access a block of memory. This minimizes the number of data transfers and improves overall performance.

[0059] Migration control: The unified memory system controls memory migration through the collaboration of hardware and the operating system. This decision is automatically managed by the hardware and the operating system based on the program's access patterns and needs.

[0060] Memory access consistency: Unified memory requires ensuring consistent memory access across the CPU and GPU. This necessitates that the system detect modifications to memory and ensure that these modifications are visible across all devices.

[0061] Unified memory simplifies data transfer, making data transfer between the CPU and GPU simpler and more efficient, eliminating the need for explicit memory transfers. It improves performance by reducing the number of data transfers and minimizing memory fragmentation, thus enhancing overall system performance. Furthermore, unified memory simplifies the process of transferring data between the CPU and GPU, improving program usability.

[0062] However, when unified memory is applied to very large datasets or applications that require high performance, it may be necessary to manually optimize data transfer and memory usage to fully utilize hardware performance.

[0063] In existing technologies, manual optimization of data transfer and memory usage methods includes data limited to a specific GPU and read-only data.

[0064] For data confined to a specific GPU, if the application knows that its use is limited to that GPU, it can instruct the software to prefetch it via an API (Application Programming Interface). The driver will then copy the data to the GPU's device memory via DMA (Direct Memory Access) before the GPU kernel starts. A unified memory architecture aims to manage cross-processor memory centrally by the software stack, making data location and migration invisible to the application. Therefore, this optimization idea actually contradicts the unified memory architecture. In AI (Artificial Intelligence) applications, applications explicitly know about kernel dispatch (task allocation and resource management functions performed by the scheduler in the operating system kernel) and data usage, and can explicitly move data via DMA. In this case, unified memory itself becomes largely meaningless.

[0065] For read-only data, if the application knows that the data in a certain buffer is read-only, it can pass a read-only advice (RO advice) to the software during request. Before the kernel starts, the driver uses DMA to copy all the data in that buffer to the local memory of all GPUs in the cluster and updates the page table to RO.

[0066] However, due to GPU consistency requirements, the prefetching method for read-only data is only applicable to read-only data; this optimization method is no longer suitable as long as there is write data. Existing technologies lack optimization methods for write data. Therefore, this invention provides an optimization method for write data.

[0067] Figure 1 This is a schematic diagram of a memory management system according to an embodiment of the present invention. Figure 1 As shown in the figure, the memory management system processor cluster of this invention includes at least one CPU and at least one GPU, that is, the processor cluster includes multiple processors, wherein the processor is a CPU or a GPU. Figure 1 In the illustrated embodiment, one CPU and three GPUs are used as an example. The three GPUs are GPU1, GPU2, and GPU3. Data is stored in the local memory of both the CPU and the GPUs.

[0068] The CPU and GPU each have their own independent memory space, and data can be transferred between CPUs and GPUs, between CPUs, and between GPUs.

[0069] It should be noted that the CPU and GPUs in the embodiments of the present invention can be different modules, or multiple CPUs and / or GPUs can be integrated into one module. The specific implementation method of the embodiments of the present invention is not limited.

[0070] Figure 2 This is a schematic diagram of a comparative memory management system according to an embodiment of the present invention. Figure 2 As shown, for data confined to a specific GPU, when allocating memory to create a buffer, if the application knows that the use of this data is limited to a particular GPU, it can instruct the software via API to prefetch it. Before the kernel starts, the driver copies the data to the GPU's device memory via DMA. The GPU's translate function obtains a PA (Physical Address) pointing to local memory. For other GPUs, the translate function can obtain a PA pointing to remote memory, or certain pages can be migrated to local memory via ODP (On-Demand Paging).

[0071] exist Figure 2In the illustrated embodiment, taking data acquired from the CPU and limited to GPU1 as an example, the driver copies the data from the CPU to the local memory of GPU1 before the kernel starts. At this time, GPU1's translate operation obtains a PA (Physical Address) pointing to the local memory. GPU2 and GPU3's translate operations can obtain PAs pointing to the device memory of GPU1. Alternatively, GPU2 and GPU3 can also migrate data from GPU1's device memory to their local memory via ODP.

[0072] However, in AI application scenarios, applications are fully aware of kernel dispatch (the mechanism in the operating system kernel responsible for scheduling and assigning tasks) and data usage. In this case, all the data that needs to be used will be explicitly moved to local memory via DMA, making unified memory meaningless.

[0073] Figure 3 This is a flowchart of a memory management method according to an embodiment of the present invention. Figure 3 As shown, the memory management method of this invention includes the following steps:

[0074] Step S101: The application determines the data type of the data in the buffer.

[0075] In this embodiment, the data types include read-only data and read-mostly data. Read-only data refers to data that, once written, will not be modified during normal program execution. This data is typically used to store immutable information. The access mode for read-only data is primarily read operations, with no write operations permitted. This restriction ensures data integrity and security, preventing accidental or malicious modification of the data by the program. Read-mostly data refers to data that is read most of the time and rarely written. This data is primarily used for read operations during program execution, but occasional write operations may be required. The access mode for read-mostly data is primarily read operations, but occasional write operations are allowed.

[0076] Determining whether the data in a buffer is read-only or read-mostly when an application interacts with the operating system or underlying hardware can be achieved in various ways, such as program logic analysis, data usage, performance analysis tools, user input or configuration, operating system or hardware interfaces, heuristics, documentation, and conventions.

[0077] Program logic analysis: The application itself may contain explicit logic regarding buffer data access patterns. For example, if the program only reads data from the buffer without modifying it, then the buffer can be considered read-only. If the program reads data from the buffer most of the time, and write operations are relatively infrequent, then the buffer can be considered primarily read-only.

[0078] Data Purpose: The purpose of a buffer can also provide clues about its access patterns. For example, buffers used to store configuration parameters or static data are typically read-only. If a buffer is used to store temporary calculation results or user input, it may be written to more frequently.

[0079] Performance analysis tools: During the program development phase, performance analysis tools can be used to monitor buffer access patterns. These tools can record the frequency of read and write operations, thus helping to determine whether the buffer is read-only or primarily read-based.

[0080] User input or configuration: In some cases, applications may allow users to specify the access mode of a buffer through input or a configuration file. For example, a user may choose to set a buffer to read-only to prevent the program from accidentally modifying its data.

[0081] Operating System or Hardware Interface: The operating system or underlying hardware interface may provide mechanisms to query or set access permissions for buffers. For example, when allocating memory, a program can specify the required access permissions (such as read-only, read-write, etc.). For read-mostly buffers, although the operating system may not directly provide such labels, the program can make decisions by analyzing its own access patterns and adjust its memory management strategy accordingly.

[0082] Heuristics: In the absence of explicit logic or performance data, programs can use heuristics to estimate buffer access patterns.

[0083] Documentation and Conventions: In large software projects, documentation and conventions may define the access patterns for buffers. For example, project documentation may explicitly state which buffers are read-only, which are primarily read, and the reasons behind these decisions.

[0084] The application of this invention can determine the data type of the data in the buffer based on any one or more of the above-described methods.

[0085] Step S102: In response to the data type being primary read data, the application sends a first notification to the software.

[0086] In this embodiment, if the application knows that the data type of a certain buffer is read-mostly, the application can send a first notification to the software when requesting the data. The first notification is a suggestion of "RdMostly" to inform the software that the data to be acquired is read-mostly.

[0087] It's important to clarify that an application program refers to a computer program written or used by a user to perform a specific task or function. These programs can be applications, scripts, tools, etc., and they typically run on top of an operating system, utilizing the services and resources provided by the operating system to accomplish their tasks. In this scenario, an application program refers to those programs that need to access and use buffer data. These programs may know or predict that certain buffers will primarily be used for reading rather than frequent writing operations. Therefore, the application wants to communicate this information to the system or software when requesting these buffers so that the system or software can optimize the handling of these buffers. Software refers to the collection of programs and data in a computer system used to control, manage, operate, or support the operation of hardware. In this scenario, software refers to the programs or systems responsible for allocating, managing, or optimizing buffer resources. This software may run at the operating system level or be part of an application program. When an application passes a "RdMostly" suggestion to the software, the software will use this information to optimize buffer allocation, access permission settings, caching strategies, etc., to improve the overall performance and efficiency of the system.

[0088] Step S103: Before the kernel starts, the data in the buffer is copied to the local memory of each processor in the processor cluster using direct memory access via the driver, and the page table permissions are marked as read-only so that the GPU will generate a write violation when writing to the page.

[0089] In this embodiment, the kernel refers to both the CPU and GPU kernels. After receiving the first notification, before the kernel starts, the software uses Direct Memory Access (DMA) via the driver to copy the data in the buffer to the local memory of each GPU in the cluster, and marks the page table permissions as read-only. That is, before the kernel starts, the driver inserts a DMA operation to copy all the data in the buffer to the local memory of all GPUs in the cluster. Subsequently, the driver updates the page table permissions, setting these pages to read-only (RO) to prevent unnecessary write operations and ensure data consistency.

[0090] A driver is a type of software that acts as an interface between computer hardware and the operating system or applications. Its main task is to ensure that the hardware can correctly respond to instructions from the operating system or applications and process data generated by the hardware. Drivers are typically responsible for initializing hardware, configuring hardware parameters, handling interrupts, and performing input / output operations. In this embodiment, the driver is responsible for performing DMA operations and updating page table permissions before the kernel starts.

[0091] A page table is a data structure used in computer systems' virtual memory. Stored in memory, it establishes an index relationship between pages and physical blocks. Each process has its own page table, which records the physical block number corresponding to each page of that process.

[0092] A page is a unit in paging memory management. In paging memory management, the application's address space is divided into several fixed-size regions, which are called "pages". Correspondingly, the physical address space of memory is also divided into several physical blocks of equal size, and these physical blocks correspond one-to-one with pages.

[0093] Step S104: When a thread block is writing data, it controls other thread blocks to be in an idle state when accessing the same range of data.

[0094] In this embodiment, the application has different thread blocks (tb). Different thread blocks need to maintain read-write consistency within the same data range. If a thread block performs a write operation, other thread blocks in the cluster should use various fences to ensure that access to that data range is idle, thereby avoiding data races.

[0095] In parallel computing and computer architecture, fence instructions (or fence operations, fence mechanisms) are a technique used to ensure the order of instruction execution and the consistency of memory accesses. They are typically used in multiprocessor or multi-core systems to ensure that all instructions preceding the fence instruction have completed execution, and that their effects are visible to instructions following the fence instruction. This prevents data inconsistencies caused by out-of-order instruction execution. Fence instructions can be categorized into several types, such as I-fence (instruction fence), D-fence (data fence), and unified fence. Different types of fence instructions synchronize different types of instructions or memory accesses. For example, I-fence ensures that all preceding instructions have completed execution before executing subsequent instructions, while D-fence ensures that all preceding memory access operations have completed before performing subsequent memory accesses. In GPU programming, fence instructions can be used to ensure synchronization between thread blocks or threads, and to ensure data consistency in global memory.

[0096] Idle processing refers to the low-power idle state of a processor (CPU or GPU) in a computer system when there are no tasks to execute. This state helps save energy and reduce processor wear. In GPUs, the idle state can be further subdivided into several levels, such as idle, shallow idle, and deep idle. These levels correspond to different power consumption and recovery times. For example, the shallow idle state might simply reduce the processor's clock frequency, while the deep idle state might completely shut down the processor's clock and power. When all thread blocks on the GPU have finished executing and no new tasks are submitted, the GPU enters the idle state. At this time, the GPU's power consumption is significantly reduced until a new task arrives and wakes the GPU up. In parallel computing, to optimize performance and energy efficiency, the processor's idle time is usually minimized. This can be achieved through proper task scheduling and load balancing.

[0097] Step S105: In response to the detection of a write violation before retrieving the missing request from the missing status holding register, stop the operation of the relevant entry and set it to a pending state.

[0098] In this embodiment, page table read / write permission checks are typically performed in the top-level Translation Backing Buffer (TLB, such as TLB0). If a write violation (wr_vio) is encountered before attempting to pop an MSHR (Miss-status Handling Registers), the relevant page table entry will stop being popped and return to a pending state.

[0099] The MSHR (Miss-status Handling Registers) is a register in the Memory Management Unit (MMU) used to track unresolved memory access requests. A page table miss occurs when the GPU attempts to access a virtual address but no corresponding mapping is found in the TLB (Translation Lookaside Buffer) or page tables. At this time, the MSHR records the status of this miss request, including the requested virtual address, the type of miss (e.g., page table miss, page miss, etc.), and any related error information. The MSHR is also responsible for coordinating subsequent page scheduling, page table updates, and other operations until the miss is resolved.

[0100] A write violation occurs when a GPU or CPU attempts to write to a read-only page or a page without write permissions. During page table read / write permission checks, a write violation occurs if the TLB mapping indicates the page is read-only, but the CPU or GPU attempts to write to it. When a write violation occurs, the relevant MSHR entry is affected. Specifically, for memory access requests that fail to complete due to a write violation, an entry is generated and stored in the MSHR. The entry contains the virtual address, request type, status information, error code or reason, and other relevant information. The virtual address is the virtual address of the memory access request that triggered the write violation; this is basic information in the MSHR entry and identifies the source of the request. In the case of a write violation, the request type is a write operation, and the MSHR needs to distinguish between read and write operations because write violations are usually associated with write operations. Status information: The entry records the current status, such as "pending." When a write violation occurs, the MSHR updates the status of the relevant entry to "pending," indicating that the request requires software intervention to resolve. Error code or reason: The MSHR entry may also contain the specific error code or reason that caused the write violation. This helps the software quickly locate the problem during subsequent processing. Other relevant information: Depending on the specific implementation, MSHR entries may also contain other request-related information, such as the request timestamp and request priority.

[0101] When processing unresolved memory access requests in MSHR, if MSHR is attempting to pop (i.e., process and remove) an entry related to a write violation, the write violation will prevent this popping process. MSHR will mark the entry as pending, waiting for the software to take appropriate measures to resolve the write violation.

[0102] Step S106: Send a second notification to the software to indicate that a write violation has occurred.

[0103] In this embodiment, TLB0 notifies the software of the virtual address (VA) via an interrupt mechanism similar to ODP (On-Demand Paging) (wr_vio interrupt) to indicate that a write violation has occurred, so that the software can take appropriate action. The second notification is generated by the ODP (On-Demand Paging) interrupt mechanism.

[0104] On-Demand Paging (ODP) is a memory management technique that allows processes to load pages into memory only when needed. When a page table miss or write violation occurs, if the relevant page is not in memory, the ODP mechanism triggers a page scheduling operation to load the missing page. In the case of a write violation, the TLB0 (or more generally the MMU) notifies the software (usually the operating system) of the virtual address via an interrupt mechanism (such as the wr_vio interrupt). The operating system checks the cause of the write violation and may take one of the following actions:

[0105] If a write violation occurs because the page is incorrectly marked as read-only, the operating system can update the page table entries to allow write operations.

[0106] If the write violation occurs because the process does not have permission to write to the page, the operating system may terminate the process or send a signal to the process to notify of the error.

[0107] In some cases, the operating system may use page scheduling to load a writable page to replace the current read-only page.

[0108] Step S107: In response to the second notification, the corresponding process is removed from the running list of each processor by context switching.

[0109] Step S108: Send instructions to clear the translation backup buffer and clear the cache.

[0110] In steps S107-S108, the software responds to the second notification, learning that a write violation has occurred. To handle the write violation, the operating system may remove the process from the run list of all GPUs via context switching. Subsequently, full-range inv-tlb (clear translation backstop) and inv L1 / L2 (clear cache) instructions are sent to ensure that all relevant cache and TLB entries are cleared.

[0111] Full Range Inv-TLB refers to an operation that invalidates all entries in the processor's TLB. A TLB is a cache used to store the translation results from virtual addresses to physical addresses to speed up memory access. When an entry in the TLB is invalidated, the processor needs to re-translate the address when accessing memory, which typically involves accessing the page table.

[0112] Invalidate L1 / L2 (Invalidate L1 / L2 Cache) refers to the operation of invalidating some or all entries in the processor's L1 and L2 caches. Caches are high-speed internal memory of the processor used to store recently accessed data and instructions to speed up data access. The L1 cache is typically divided into instruction cache and data cache, while the L2 cache is a larger, slower cache used to store more data and instructions.

[0113] Step S109: The driver changes the processor's pages prefetched to local memory to page table entry granularity.

[0114] In this embodiment, the driver changes the buffer pages of the processor that have been prefetched into local memory to the page table entry (PTE) granularity, so as to change the access permissions or status of the buffer pages to finer-grained control.

[0115] As described above, each processor prefetches data into its local memory. When a write violation is encountered, the page that needs to be modified can be determined based on the information corresponding to the write violation. Once the page that needs to be modified is determined, the driver updates the corresponding Page Entity Table (PTE) in the page table. Here, "changing to PTE granularity" actually refers to changing the access permissions or status of the page to a finer-grained control. That is, the access permissions or status of each page can be changed through PTE granularity, as described in steps S110-S112.

[0116] Step S110: For the page being written to, grant write permission to the first processor that detects the write violation.

[0117] In this embodiment, MSHR may already store one or more entries corresponding to write violations. For the same page that is being written to, write permission is granted to the first processor that detects the write violation by changing the PTE granularity.

[0118] Step S111: Grant other processors that write to the same page permission to obtain data by remotely accessing the first processor.

[0119] In this embodiment, for the same page being written to, write permissions are granted to the first processor that detects a write violation by changing the PTE granularity. Other processors writing to the same page are granted permissions to remotely access the data from the first processor.

[0120] Specifically, the permission to obtain data by remotely accessing the first processor refers to: other processors remotely accessing the data of the first processor, or other processors remotely accessing and synchronizing the data of the first processor to their local memory.

[0121] Step S112: The page tables of processors that do not access pages will be marked as non-resident.

[0122] In this embodiment, for the same page being written to, write permissions are granted to the first processor that detects a write violation by changing the PTE granularity. Other processors writing to the same page are granted permissions to remotely access the data from the first processor. The page tables of processors that do not access the page are marked as non-resident, indicating that these processors do not have a valid mapping for that page.

[0123] It should be noted that the embodiments of the present invention are illustrated by giving write permission to the first processor that detects a write violation, but the embodiments of the present invention are not limited to this. Write permission can be given to any one of the processors that detect a write violation, and the other processors can remotely access the processor with write permission.

[0124] Step S113: Re-parse the missing request and obtain the physical address.

[0125] In this embodiment, the driver sends a resolve register (resolve_mshr) command to TLB0 to re-resolve the previous missing request. Through this process, TLB0 will be able to obtain the correct physical address (PA) and update the page table so that subsequent accesses can be performed correctly.

[0126] Step S114: In response to receiving an access request for a non-resident page, retrieve new data from the data update processor and copy it to the local memory of other processors.

[0127] In this embodiment, as described above, the page tables of processors that do not access pages are marked as non-resident. Therefore, in response to receiving an access request from a processor for a non-resident page, another context switch is triggered to retrieve new data from the processor updating the data and copy it to the local memory of other processors. The processors updating the data include those that performed write operations, i.e., the first processor to detect a write violation and other processors that wrote to the same page.

[0128] Therefore, when multiple addresses of a page are written to by different GPUs before a context switch, the memory management method of this invention ensures that after the context switch, the pages of other GPUs will point to the first GPU that violated the write rule. After replay, the write operation is completed, and the data is merged into a single GPU's page. GPUs that haven't accessed this page will have their pages become not resident. Subsequent read / write operations will trigger ODP, re-entering the context switch. During the switch phase, the latest data is broadcast, and the page of all GPUs that have prefetched data is separated and restored to read-only (RO). Afterward, GPU address translation and read / write operations are completed.

[0129] Figure 4 This is a schematic diagram illustrating data conversion in a memory management system according to an embodiment of the present invention. Figure 4 As shown, the memory management system of this embodiment includes at least one CPU and at least one GPU. Figure 4 In the illustrated embodiment, one CPU and three GPUs are used as an example. The three GPUs are GPU1, GPU2, and GPU3. Data is stored in the local memory of both the CPU and the GPUs.

[0130] exist Figure 4 In the embodiment shown, the data conversion process is divided into three states: the first state S1, the second state S2, and the third state S3.

[0131] The first state, S1, is the data prefetching process. Figure 4 The example below illustrates how GPU1, GPU2, and GPU3 prefetch data from the CPU. After prefetching, the data from GPU1, GPU2, GPU3, and the CPU is identical.

[0132] The second state, S2, is the data write access process. For example... Figure 4 As shown, GPU1, GPU2, GPU3, and CPU each represent a page of data. A buffer contains eight pages; for clarity, the four on the left, from top to bottom, are designated A1-A4, and the four on the right, from top to bottom, are designated B1-B4. In the second state S2, GPU1 performs a write operation on page A1, and GPU2, GPU3, and the CPU mark the corresponding page A1 as "not resident." GPU3 performs a write operation on page B4, and GPU1, GPU2, and the CPU mark the corresponding page B4 as a "not resident" page. This forms the second state S2.

[0133] The third state, S3, describes the processing flow for access requests to non-resident pages. Specifically, in the third state, when a processor (any one of GPU1, GPU2, GPU3, and CPU) accesses page A1 of any one of GPU2, GPU3, and CPU, since GPU2, GPU3, and CPU marked the corresponding page A1 as a "not resident" page in the second state, an access request for a non-resident page is received. Therefore, new data is retrieved from the GPU that updated the data (GPU1 in this case). The new data (the data in page A1 on GPU1) is then copied to the other GPUs and CPUs (GPU2, GPU3, and CPU).

[0134] Figure 4 The other pages in the list are those other than those for write operations and those marked as not resident.

[0135] Therefore, in this embodiment of the invention, page table entries for GPUs that have not been accessed before the write operation (i.e., GPUs with page table entries marked as "not resident") are marked as "not resident," indicating that these pages are not currently in the local memory of that GPU. When another GPU attempts to access a page marked as "not resident," the system triggers the On-Demand Paging (ODP) mechanism. The ODP mechanism triggers another context switch (or a lighter-weight operation) as needed to move the latest data from the GPU that holds the latest data to other GPUs or CPUs.

[0136] Figure 5 This is a schematic diagram illustrating data conversion in a memory management system according to an embodiment of the present invention. Figure 5 As shown, the memory management system of this embodiment includes three GPUs, namely GPU1, GPU2, and GPU3. Figure 5 In the embodiment shown, the data conversion process is divided into four states: the fourth state S4, the fifth state S5, the sixth state S6, and the seventh state S7.

[0137] In state S4, before kernel startup, the data in the buffer is copied to the local memory of GPUs 1, 2, and 3 in the cluster via direct memory access through the driver, and the page tables of GPUs 1, 2, and 3 are marked as read-only (RO). GPU 1 encounters a write violation (wr_vio) on one address of a page, and GPU 3 encounters a write violation (wr_vio) on another address of the same page.

[0138] In state S5, a write violation is detected before the missing request is retrieved from the missing status holding register, triggering a context switch. During the context switch, write permissions are granted to GPU1 and GPU3 (W in the diagram), while GPU2's page table is marked as not resident. Simultaneously, GPU3 notifies GPU1 of the write operation to be performed. At this point, GPU1 becomes the first GPU scheduled for a write violation on that page.

[0139] In state S6, GPU1 completes the write operations required by GPU1 and GPU3. Simultaneously, other CPUs or GPUs access the corresponding pages on GPU2.

[0140] Since the page is marked as not resident, this triggers the next context switch, leading to the seventh state, S7.

[0141] In the seventh state S7, GPU1 is the GPU that updates the data. Therefore, the new data in GPU1 is copied to the other GPUs (GPU2 and GPU3), and at the same time, the page tables of each GPU are marked as read-only (RO).

[0142] In some embodiments, when the GPU3 performs remote mapping on a page, it can temporarily record this mapping relationship in the TLB, but the page's state in the HBM (High Bandwidth Memory) is marked as not resident, meaning that the page's data does not actually reside in the GPU3's HBM. This optimizes memory access, avoids unnecessary memory access, and allows the GPU3 to retrieve the page's data when needed through some mechanism (such as subsequent data transfer operations).

[0143] Simultaneously, when multiple GPUs in the processor cluster attempt to access the same page, any TLB evict can trigger ODP to perform data broadcasting and decoupling. The purpose of data broadcasting is to ensure that all GPUs requiring the page's data receive the latest copy. This process may also involve decoupling, which breaks any previously existing data synchronization or association state so that each GPU can independently process its own copy of the data.

[0144] When a GPU needs to access a page that is not in its local HBM, an ODP operation can be triggered to fetch the latest copy of that page from another GPU or memory location. This process can be performed without a full context switch, thus reducing performance overhead. After the fetch is complete, the GPU can then process the page's data in its own HBM.

[0145] For pages obtained via remote mapping, if a TLB entry is removed due to evict, the GPU may trigger an ODP operation again to retrieve the data. This process is similar to the data broadcasting and detaching mechanisms mentioned earlier, ensuring data consistency and correctness.

[0146] This invention replicates data from a buffer primarily used for reading data to the local memory of each processor in a processor cluster, and marks the page table permissions as read-only. This causes write violations when processors attempt to write to a page. Among processors that experience write violations when writing to the same page, one processor completes the write operation for each processor, while other processors retrieve the data by remotely accessing the processor performing the write operation. This allows for prefetching of write data while ensuring data consistency across processors, thereby improving the computer's processor performance.

[0147] Figure 6 This is a flowchart of a memory management method according to another embodiment of the present invention. Figure 6 As shown, the memory management method of this invention includes the following steps:

[0148] Step S210: Copy the data in the buffer with the data type of primary read data to the local memory of each processor in the processor cluster, and mark the page table permissions as read-only so that the processor will generate a write violation when writing to the page.

[0149] Step S210: Among the processors that generate write violations when writing to the same page, one of the processors completes the write operation of each processor, and the other processors obtain the data by remotely accessing the processor that performed the write operation.

[0150] The processor cluster includes one or more CPUs, and / or one or more GPUs.

[0151] In some embodiments, copying the data from the buffer whose data type is primary read data to the local memory of each processor in the processor cluster includes:

[0152] The application determines the data type of the data in the buffer;

[0153] In response to the data type being primary read data, the application sends a first notification to the software;

[0154] Before the kernel starts, the data in the buffer is copied to the local memory of each processor in the processor cluster via direct memory access through the driver.

[0155] In some embodiments, the method further includes:

[0156] When a thread block is writing data, it controls other thread blocks to be in an idle state when accessing the same range of data.

[0157] In some embodiments, among the processors that generate write violations when writing to the same page, one processor completes the write operations of each processor, and other processors obtain data by remotely accessing the processor performing the write operation.

[0158] In response to a write violation detected before a missing request is retrieved from the missing status holding register, the operation on the relevant entry is stopped and the entry is set to a pending state.

[0159] A second notification is sent to the software to indicate that a write violation has occurred.

[0160] In some embodiments, among the processors that generate write violations when writing to the same page, the write operations of each processor are completed by one of the processors, and the other processors obtain data by remotely accessing the processor performing the write operation.

[0161] In response to the second notification, the corresponding process is removed from the running list of each processor via context switching;

[0162] Send commands to clear the translation backstop buffer and clear the cache.

[0163] In some embodiments, among the processors that generate write violations when writing to the same page, the write operations of each processor are completed by one of the processors, and the other processors obtain data by remotely accessing the processor performing the write operation.

[0164] The driver changes the buffer pages of the processor that it has already prefetched into local memory to page table entry granularity;

[0165] The permission status of each processor is updated at the granularity of the page table entries.

[0166] In some embodiments, updating the permission status of each processor at the page table entry granularity includes:

[0167] For the page being written to, grant write permission to the first processor that detects the write violation;

[0168] Grant other processors that write to the same page permission to retrieve data by remotely accessing the first processor;

[0169] The page tables of processors that have not accessed pages will be marked as non-resident.

[0170] In some embodiments, the method further includes:

[0171] Re-parse the missing request and obtain the physical address.

[0172] In some embodiments, the method further includes:

[0173] In response to an access request for a non-resident page, new data is retrieved from the data update processor and copied to the local memory of other processors.

[0174] This invention replicates data from a buffer primarily used for reading data to the local memory of each processor in a processor cluster, and marks the page table permissions as read-only. This causes write violations when processors attempt to write to a page. Among processors that experience write violations when writing to the same page, one processor completes the write operation for each processor, while other processors retrieve the data by remotely accessing the processor performing the write operation. This allows for prefetching of write data while ensuring data consistency across processors, thereby improving the computer's processor performance.

[0175] Figure 7 This is a schematic diagram of a memory management device according to an embodiment of the present invention. Figure 7 As shown, the memory management device of this embodiment includes a prefetch unit 71 and an operation unit 72. The prefetch unit 71 copies data from a buffer containing data of primary read type to the local memory of each processor in the processor cluster, and marks the page table permissions as read-only, causing a write violation when a processor writes to a page. The operation unit 72, among the processors that cause write violations when writing to the same page, allows one processor to complete the write operation for each processor, while other processors obtain data by remotely accessing the processor performing the write operation.

[0176] In some embodiments, the prefetch unit is used for:

[0177] The application determines the data type of the data in the buffer;

[0178] In response to the data type being primary read data, the application sends a first notification to the software;

[0179] Before the kernel starts, the data in the buffer is copied to the local memory of each processor in the processor cluster via direct memory access through the driver.

[0180] In some embodiments, the apparatus further includes:

[0181] The state control unit is used to control other thread blocks to be in an idle state when they are accessing the same range of data while a thread block is writing data.

[0182] In some embodiments, the operating unit is used for:

[0183] In response to a write violation detected before a missing request is retrieved from the missing status holding register, the operation on the relevant entry is stopped and the entry is set to a pending state.

[0184] A second notification is sent to the software to indicate that a write violation has occurred.

[0185] In some embodiments, the operation unit is further configured to:

[0186] In response to the second notification, the corresponding process is removed from the running list of each processor via context switching;

[0187] Send commands to clear the translation backstop buffer and clear the cache.

[0188] In some embodiments, the operating unit further includes:

[0189] The driver changes the buffer pages of the processor that it has already prefetched into local memory to page table entry granularity;

[0190] The permission status of each processor is updated at the granularity of the page table entries.

[0191] In some embodiments, the operation unit is used for:

[0192] For the page being written to, grant write permission to the first processor that detects the write violation;

[0193] Grant other processors that write to the same page permission to retrieve data by remotely accessing the first processor;

[0194] The page tables of processors that have not accessed pages will be marked as non-resident.

[0195] In some embodiments, the apparatus further includes:

[0196] The address acquisition unit is used to re-parse the missing request and obtain the physical address.

[0197] In some embodiments, the apparatus further includes:

[0198] The data synchronization unit is used to respond to access requests for non-resident pages by retrieving new data from the processor that updates the data and copying it to the local memory of other processors.

[0199] This invention replicates data from a buffer primarily used for reading data to the local memory of each processor in a processor cluster, and marks the page table permissions as read-only. This causes write violations when processors attempt to write to a page. Among processors that experience write violations when writing to the same page, one processor completes the write operation for each processor, while other processors retrieve the data by remotely accessing the processor performing the write operation. This allows for prefetching of write data while ensuring data consistency across processors, thereby improving the computer's processor performance.

[0200] In some embodiments, the present invention also provides an electronic device including a processor cluster and a memory, the processor cluster including one or more CPUs and one or more GPUs, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the CPU and / or GPU to implement the memory management method of the present invention.

[0201] The CPU and GPU each have their own independent local memory. The CPU's local memory is called host memory, and the GPU's local memory is called device memory. To achieve seamless collaboration between the CPU and GPU, the electronic device in this embodiment of the invention uses unified memory, allowing the CPU and GPU to share the same memory space. This enables efficient data access and sharing without data copying, significantly reducing data transmission latency and improving overall system performance.

[0202] Electronic devices can be integrated or non-integrated. In integrated devices, the CPU and GPU can be designed as highly integrated components, which helps reduce system complexity and power consumption while improving data transfer speeds. In non-integrated devices, the CPU and GPU can be configured independently and connected via a high-speed bus. This approach allows users to flexibly choose the performance levels of the CPU and GPU according to their actual needs, achieving the optimal balance between performance and cost.

[0203] In this system, one or more CPUs and / or GPUs can be connected to memory via a bus or other means. Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The CPU and / or GPU execute various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory, thus implementing the aforementioned memory management method.

[0204] The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include memory remotely configured relative to the CPU and / or GPU, which can be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0205] One or more modules are stored in memory and, when executed by one or more CPUs and / or GPUs, perform the memory management methods in any of the above method embodiments.

[0206] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

[0207] This invention replicates data from a buffer primarily used for reading data to the local memory of each processor in a processor cluster, and marks the page table permissions as read-only. This causes write violations when processors attempt to write to a page. Among processors that experience write violations when writing to the same page, one processor completes the write operation for each processor, while other processors retrieve the data by remotely accessing the processor performing the write operation. This allows for prefetching of write data while ensuring data consistency across processors, thereby improving the computer's processor performance.

[0208] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0209] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or a CPU and / or GPU to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0210] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A memory management method, characterized in that, The method includes: The data in the buffer with the primary read data type is copied to the local memory of each processor in the processor cluster, and the page table permissions are marked as read-only so that a write violation occurs when a processor writes to the page. Among the processors that generate write violations when writing to the same page, one processor completes the write operations of each processor, while other processors obtain data by remotely accessing the processor that performed the write operation. The processor cluster includes one or more CPUs and one or more GPUs.

2. The method according to claim 1, characterized in that, The step of copying data from the buffer whose data type is primary read data to the local memory of each processor in the processor cluster includes: The application determines the data type of the data in the buffer; In response to the data type being primary read data, the application sends a first notification to the software; Before the kernel starts, the data in the buffer is copied to the local memory of each processor in the processor cluster via direct memory access through the driver.

3. The method according to claim 1, characterized in that, The method further includes: When a thread block is writing data, it controls other thread blocks to be in an idle state when accessing the same range of data.

4. The method according to claim 1, characterized in that, In the case of processors that generate write violations when writing to the same page, one processor completes the write operations of each processor, and other processors obtain data by remotely accessing the processor performing the write operation. In response to a write violation detected before a missing request is retrieved from the missing status holding register, the operation on the relevant entry is stopped and the entry is set to a pending state. A second notification is sent to the software to indicate that a write violation has occurred.

5. The method according to claim 4, characterized in that, In the process of generating a write violation when writing to the same page, the write operations of each processor are completed by one of the processors, and other processors obtain data by remotely accessing the processor performing the write operation. In response to the second notification, the corresponding process is removed from the running list of each processor via context switching; Send commands to clear the translation backstop buffer and clear the cache.

6. The method according to claim 5, characterized in that, In the process of generating a write violation when writing to the same page, the write operations of each processor are completed by one of the processors, and other processors obtain data by remotely accessing the processor performing the write operation. The driver changes the buffer pages of the processor that it has already prefetched into local memory to page table entry granularity; The permission status of each processor is updated at the granularity of the page table entries.

7. The method according to claim 6, characterized in that, The step of updating the permission status of each processor at the page table entry granularity includes: For the page being written to, grant write permission to the first processor that detects the write violation; Grant other processors that write to the same page permission to retrieve data by remotely accessing the first processor; The page tables of processors that have not accessed pages will be marked as non-resident.

8. The method according to claim 4, characterized in that, The method further includes: Re-parse the missing request and obtain the physical address.

9. The method according to claim 7, characterized in that, The method further includes: In response to an access request for a non-resident page, new data is retrieved from the data update processor and copied to the local memory of other processors.

10. A memory management device, characterized in that, The device includes: The prefetch unit is used to copy data from the buffer whose data type is primary read data to the local memory of each processor in the processor cluster, and to mark the page table permissions as read-only so that the processor will generate a write violation when writing to the page. The operation unit is used to complete the write operations of each processor in the processor that generates a write violation when writing to the same page, and other processors obtain data by remotely accessing the processor that performs the write operation. The processor cluster includes one or more CPUs and one or more GPUs.

11. An electronic device comprising a processor cluster and a memory, characterized in that, The processor cluster includes one or more CPUs and one or more GPUs, and the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the CPUs and / or GPUs to implement the method as described in any one of claims 1-9.

12. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is run on a computer, the computer performs the method according to any one of claims 1-9.

13. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by the CPU and / or GPU, implement the method as described in any one of claims 1-9.

Images