Data writing method and related equipment
By writing data to local memory in a NUMA architecture and synchronizing it to the target memory by the processor core, the problems of low cross-node write performance and poor correctness of PCIe devices are solved, achieving more efficient data transmission.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2026-03-10
AI Technical Summary
In a NUMA architecture, when a PCIe device writes data directly to the memory of other NUMA nodes, there are problems with low data write performance and poor data accuracy.
PCIe devices write data to the local memory of their connected NUMA nodes and synchronize the data in the local memory to the target memory through the currently running processor core of the NUMA architecture, avoiding direct cross-node writing.
It improves data writing performance and data accuracy, especially in high-performance and high-pressure application scenarios, thereby enhancing data applicability.
Smart Images
Figure CN121636401A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to a data writing method and related equipment. Background Technology
[0002] A NUMA (Non-Uniform Memory Access) architecture includes at least two NUMA nodes, each containing a processor and memory. In addition, at least one NUMA node may include PCIe (Peripheral Component Interconnect Express) devices. PCIe devices can access the memory of their local NUMA node and other NUMA nodes in response to DMA (Direct Memory Access) requests. However, when a PCIe device directly writes data to the memory of other NUMA nodes in response to DMA write requests, issues arise regarding low data write performance and poor data integrity. Summary of the Invention
[0003] This application discloses a data writing method and related device to solve the problems of low data writing performance and poor data correctness when a PCIe device directly writes data to the memory of other NUMA nodes in response to a DMA write request.
[0004] In a first aspect, this application discloses a data writing method applied to a PCIe device with a NUMA architecture. The NUMA architecture includes multiple NUMA nodes and multiple PCIe devices, each PCIe device being connected to one of the NUMA nodes. Each NUMA node includes a processor and memory, and the processor includes at least one processor core. The method includes: receiving a DMA write request, the DMA write request being used to request that target data be directly written to target memory; responding to the DMA write request, determining whether the target memory is local memory; the local memory being the memory of the NUMA node connected to the PCIe device; if the target memory is not local memory, then directly writing the target data to the local memory, and sending a synchronization instruction to the processor core currently running in the NUMA architecture, the synchronization instruction being used to instruct the processor core to synchronize the target data in the local memory to the target memory.
[0005] In some embodiments of this application, the memory of any NUMA node includes a transit cache pool, and writing the target data directly to the local memory includes: writing the target data directly to the transit cache pool of the local memory; the transit cache pool is used to cache data written by PCIe devices of the same NUMA node.
[0006] In some embodiments of this application, writing the target data directly to the intermediate cache pool of the local memory includes: determining whether a first condition, a second condition, and a third condition are met; if the first condition, the second condition, and the third condition are met simultaneously, then the target data is directly written to the intermediate cache pool of the local memory; wherein, the first condition includes that the PCIe device has permission to use the intermediate cache pool; the second condition includes that the intermediate cache pool of the local memory is valid; and the third condition includes that the intermediate cache pool is capable of storing the target data.
[0007] In some embodiments of this application, the intermediate cache pool includes at least a default cache pool, and the default cache pool includes multiple intermediate cache areas. Determining whether the intermediate cache pool can store the target data includes: determining whether the default cache pool has an intermediate cache area capable of storing the target data; if the default cache pool does not have an intermediate cache area capable of storing the target data, then a creation instruction is sent to the processor core currently running in the NUMA architecture, the creation instruction being used to instruct the processor core to create a new extended cache pool in the local memory.
[0008] In some embodiments of this application, the intermediate cache pool includes a default cache pool and an extended cache pool. Both the default cache pool and the extended cache pool include multiple intermediate cache areas. Determining whether the intermediate cache pool can store the target data includes: determining whether the default cache pool and a preset number of extended cache pools have intermediate cache areas capable of storing the target data; if the default cache pool and the preset number of extended cache pools do not have intermediate cache areas capable of storing the target data, then a creation instruction is sent to the processor core currently running in the NUMA architecture. The creation instruction is used to instruct the processor core to create a new extended cache pool in the local memory.
[0009] In some embodiments of this application, when determining whether any default cache pool or extended cache pool has a transit cache area capable of storing the target data, priority is given to determining whether the transit cache area in the default cache pool or the extended cache pool that has affinity with the processor core currently running the NUMA architecture can store the target data.
[0010] In some embodiments of this application, the method further includes: if any extended cache pool is not occupied, sending a release instruction to the processor core currently running the NUMA architecture, the release instruction being used to instruct the processor core to release the extended cache pool.
[0011] In some embodiments of this application, before sending the synchronization instruction to the processor core currently running the NUMA architecture, the method further includes: determining whether a first condition, a second condition, and a fifth condition are met; if the first condition, the second condition, and the fifth condition are met simultaneously, then a synchronization instruction is sent to the processor core currently running the NUMA architecture; wherein, the first condition includes that the PCIe device has permission to use the intermediate cache pool; the second condition includes that the intermediate cache pool of the local memory is valid; and the fifth condition includes that the target data has been written to the intermediate cache pool of the local memory.
[0012] Secondly, this application discloses a data writing method applied to a processor core of a NUMA architecture. The NUMA architecture includes multiple NUMA nodes and multiple PCIe devices, each PCIe device being connected to one of the NUMA nodes. Each NUMA node includes a processor and memory, and the processor includes at least one processor core. The method includes: receiving a synchronization instruction sent by the PCIe device, the synchronization instruction being used to synchronize target data in local memory to target memory, the local memory being the memory of the NUMA node connected to the PCIe device; and responding to the synchronization instruction, synchronizing the target data in the local memory to the target memory.
[0013] In some embodiments of this application, synchronizing the target data in the local memory to the target memory includes: synchronizing the target data in the intermediate cache pool in the local memory to the target memory; the intermediate cache pool is used to cache data written by PCIe devices on the same NUMA node.
[0014] In some embodiments of this application, the method further includes: receiving a creation instruction sent by the PCIe device; and, in response to the creation instruction, creating a new extended cache pool in the local memory.
[0015] In some embodiments of this application, the method further includes: receiving a release command sent by the PCIe device; and releasing the corresponding extended cache pool in response to the release command.
[0016] Thirdly, this application discloses a PCIe device applied to a NUMA architecture, the NUMA architecture including multiple NUMA nodes and multiple PCIe devices, each PCIe device being connected to one of the NUMA nodes, the PCIe device including a processor and a memory; wherein, the memory is connected to the processor, the memory being used to store computer programs; the processor being used to implement the data writing method as described in any of the preceding claims by running the computer programs stored in the memory.
[0017] Fourthly, this application discloses a processor applied to a NUMA architecture, the NUMA architecture including multiple NUMA nodes and multiple PCIe devices, each PCIe device being connected to one of the NUMA nodes, each NUMA node including a processor and memory, the processor including at least one processor core, the processor core being configured to perform the data writing method as described in any of the preceding claims.
[0018] Fifthly, this application discloses a NUMA architecture, including the PCIe device and the processor as described above.
[0019] Sixthly, this application discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the data writing method as described in any of the preceding claims.
[0020] This application discloses a data writing method and related apparatus. After receiving a DMA write request to write target data directly to target memory, the PCIe device determines whether the target memory is the local memory of a NUMA node connected to the PCIe device. If the target memory is not local memory, the target data is written directly to the local memory, and a synchronization instruction is sent to the processor core currently running in the NUMA architecture. This synchronization instruction instructs the processor core to synchronize the target data in the local memory to the target memory. Therefore, the PCIe device does not need to write data directly to the memory of other NUMA nodes. Instead, the PCIe device only needs to write data directly to the local memory of its connected NUMA node, and then the processor core of the NUMA architecture synchronizes the data in the local memory to the memory of other NUMA nodes. This improves data writing performance and data correctness. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.
[0022] Figure 1 This is a schematic diagram of a NUMA architecture disclosed in an embodiment of this application.
[0023] Figure 2 This is a flowchart of a data writing method disclosed in an embodiment of this application.
[0024] Figure 3 This is a flowchart illustrating another data writing method disclosed in an embodiment of this application.
[0025] Figure 4 This is a flowchart illustrating a method for determining whether a transit cache pool can store target data, as disclosed in an embodiment of this application.
[0026] Figure 5 This is a schematic diagram of the structure of a default cache pool and an extended cache pool disclosed in an embodiment of this application.
[0027] Figure 6 This is a flowchart illustrating a method for sending synchronization instructions to a processor core currently running on a NUMA architecture, as disclosed in an embodiment of this application.
[0028] Figure 7 This is a flowchart illustrating another data writing method disclosed in an embodiment of this application. Detailed Implementation
[0029] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0030] like Figure 1 As shown, Figure 1 This is a schematic diagram of a NUMA architecture disclosed in an embodiment of this application. The NUMA architecture includes a first NUMA node, a second NUMA node, and two PCIe devices. Both NUMA nodes include a processor and memory, and each processor includes at least one processor core. The processor cores of different NUMA nodes are connected to each other via interconnect interfaces, and the processor cores of the same NUMA node are connected to the PCIe devices and memory via a system bus. The PCIe devices include device memory and a DMA controller. The processor cores can access the device memory of the PCIe devices, and the PCIe devices can directly access the memory of their local NUMA node and other NUMA nodes through the DMA controller.
[0031] However, when a PCIe device writes data to the memory of another NUMA node via the DMA controller—for example, when a PCIe device connected to the first NUMA node writes data to the memory of the second NUMA node via the DMA controller—the data needs to be written through the PCIe bus, system bus, and interconnect interface links. This results in low data write performance and poor data accuracy. Especially in high-performance and high-pressure applications, the processor core does not allocate memory based on the NUMA node of the PCIe device, making cross-node data writing by PCIe devices less suitable.
[0032] Based on this, this application discloses a data writing scheme. When a PCIe device needs to write data to the memory of other NUMA nodes, the PCIe device writes the data to the local memory of its connected NUMA node, and then the processor core writes the data in the local memory to the memory of other NUMA nodes, so as to improve data writing performance and data correctness.
[0033] As an optional implementation of the disclosure in this application, an embodiment of this application discloses a data writing method applied to a NUMA architecture PCIe device, for example, applied to... Figure 1 Any PCIe device with the NUMA architecture shown.
[0034] The NUMA architecture can include multiple NUMA nodes and multiple PCIe devices. Each PCIe device is connected to a NUMA node. Each NUMA node includes a processor and memory. Each processor includes at least one processor core.
[0035] like Figure 2 As shown, Figure 2 This is a flowchart of a data writing method disclosed in an embodiment of this application. The method includes:
[0036] S101: Receive DMA write request. DMA write request is used to write target data directly to target memory.
[0037] In this embodiment, the PCIe device receives a DMA write request sent by an external device. The DMA write request is used to request the PCIe device to directly write target data to target memory. The external device can be a network interface card (NIC) or similar device.
[0038] In some embodiments, after receiving a DMA write request, the PCIe device first uses the standard interface to request a receive buffer, which is used to store the target data. The receive buffer belongs to the target memory. Then, it calls the DMA standard interface to calculate the physical address of the receive buffer, which is the physical address of the buffer in the target memory that stores the target data.
[0039] It should be noted that the steps in the various embodiments of the data writing method applied to PCIe devices in this application can be executed by the PCIe device or by [other means]. Figure 1 The DMA controller shown is executed, but in this embodiment, it is only described as being executed by a PCIe device.
[0040] S102: In response to a DMA write request, determine whether the target memory is local memory; local memory is the memory of the NUMA node connected to the PCIe device.
[0041] In this embodiment, after obtaining the physical address of the cache storing the target data in the target memory, the PCIe device determines whether the target memory is local memory based on the physical address and the physical address range of the local memory. The local memory refers to the memory of the NUMA node connected to the PCIe device.
[0042] In some embodiments of this application, the PCIe device can first obtain information about the NUMA node connected to it, and then obtain the physical address range of the local memory based on the information of the NUMA node. Then, it compares the physical address of the cache storing the target data in the target memory with the physical address range of the local memory. If the physical address of the cache storing the target data in the target memory is within the physical address range of the local memory, then the target memory is local memory; otherwise, the target memory is not local memory.
[0043] It should be noted that when a PCIe device is enumerated by the Host bridge, it obtains information about the NUMA nodes connected to it. However, because some PCIe devices may modify the identifier of their connected NUMA nodes to a different value before requesting the receive buffer, such as -1, a new variable, "local node," needs to be added to the struct device data structure to store information about the NUMA nodes connected to the PCIe device. This allows the PCIe device to obtain information about its connected NUMA nodes based on the information of the local node.
[0044] S103: If the target memory is not local memory, the target data is written directly to local memory, and a synchronization instruction is sent to the processor core currently running in the NUMA architecture. This synchronization instruction is used to instruct the processor core to synchronize the target data in local memory to the target memory.
[0045] In this embodiment, the PCIe device determines whether the target memory is local memory. If the target memory is local memory, the PCIe device directly writes the target data to the target memory, i.e., the local memory. If the target memory is not local memory, the PCIe device directly writes the target data to the local memory and sends a synchronization instruction to the processor core currently running in the NUMA architecture, so that the processor core responds to the synchronization instruction and synchronizes the target data in the local memory to the target memory.
[0046] It should be noted that writing target data directly to target memory or local memory means that the PCIe device uses DMA access to write the target data directly to target memory or local memory without going through the processor core. Furthermore, the processor core currently running in the NUMA architecture can be the processor core of a NUMA node connected to the PCIe device, or it can be the processor core of a NUMA node not connected to the PCIe device.
[0047] Compared to the method of PCIe devices directly writing data to the memory of other NUMA nodes, in this embodiment, the target data is directly written to the local memory by the PCIe device, and the target data in the local memory is synchronized to the target memory by the processor core currently running in the NUMA architecture. Because the performance and correctness of writing data from the PCIe device to the local memory and the performance and correctness of the processor core writing data to the memory of other NUMA nodes are higher than those of writing data from the PCIe device to the memory of other NUMA nodes, the data writing performance and data correctness can be improved.
[0048] In some embodiments of this application, the memory of any NUMA node includes a transit cache pool, which is used to cache data written by PCIe devices on the same NUMA node. When a PCIe device writes target data directly to local memory, the PCIe device can write the target data directly to the transit cache pool of local memory.
[0049] Of course, this application is not limited to this. In other embodiments, the memory may not include a transit cache pool. The PCIe device may also request a cache of local memory from the processor core currently running in the NUMA architecture to store the target data before writing the target data directly to the local memory.
[0050] It should be noted that each NUMA node can have a transit cache pool, which can include multiple transit cache areas, and each transit cache area can include multiple transit cache slabs. Each transit cache slab has the same size, for example, 2KB, and this slab is the minimum storage area of the transit cache pool. Furthermore, different transit cache areas can be the same size or different sizes. Each transit cache area can contain a maximum of 1024 contiguous transit cache slabs, meaning each transit cache area can have a maximum size of 2MB.
[0051] In some embodiments of this application, after determining that the target memory is not local memory, it is also necessary to determine whether a first condition, a second condition, and a third condition are met. The first condition includes that the PCIe device has permission to use the intermediate cache pool; the second condition includes that the intermediate cache pool of local memory is valid; and the third condition includes that the intermediate cache pool can store the target data. If all three conditions are met simultaneously, the PCIe device directly writes the target data to the intermediate cache pool of local memory. If at least one of the first, second, and third conditions is not met, the PCIe device cannot directly write the target data to the intermediate cache pool of local memory; instead, it performs a DMA write operation to directly write the target data to the target memory.
[0052] Of course, this application is not limited to this. In some other embodiments, the PCIe device may be allowed to write the target data directly to the local memory's intermediate cache pool only when the fourth condition is met. The fourth condition includes that the data transfer direction between the PCIe device and the local memory is from the PCIe device to the local memory.
[0053] In some embodiments of this application, after determining that the target memory is not local memory, it can be determined sequentially whether the first condition, the second condition, the third condition, and the fourth condition are met. For example... Figure 3 As shown, Figure 3 This is a flowchart illustrating another data writing method disclosed in an embodiment of this application. Figure 3 Steps S101 and S102 in the process are similar to those in the previous steps. Figure 2Steps S101 and S102 are the same and will not be repeated here. After determining that the target memory is not local memory, we can first proceed to step S1021. S1021: Determine whether the PCIe device has permission to use the intermediate cache pool. If it has permission, proceed to step S1022. S1022: Determine whether the intermediate cache pool of local memory is valid. If the intermediate cache pool is valid, proceed to step S1023. S1023: Determine whether the intermediate cache pool can store the target data. If it can store the data, proceed to step S1024. S1024: Determine whether the data transfer direction between the PCIe device and local memory is from the PCIe device to local memory. If the data transfer direction is from the PCIe device to local memory, proceed to step S103. S103: Write the target data directly to the intermediate cache pool of local memory; otherwise, the target data cannot be written directly to the intermediate cache pool of local memory, and proceed to step S104. S104: Write the target data directly to the target memory.
[0054] Of course, this application is not limited to this. In some other embodiments, it is also possible to determine whether the first condition, the second condition, the third condition and the fourth condition are met in parallel, or it is also possible to determine whether the first condition, the second condition, the third condition and the fourth condition are met in other order, which will not be elaborated here.
[0055] In some embodiments of this application, the permission of a PCIe device to use the intermediate cache pool can be determined by checking whether the NUMA architecture processor is a processor of a preset type, whether the device requiring the use of the intermediate cache pool is a PCIe device, and whether the NUMA node connected to the PCIe device is valid. If the NUMA architecture processor is a processor of a preset type, the device requiring the use of the intermediate cache pool is a PCIe device, and the NUMA node connected to the PCIe device is valid, then the PCIe device has the permission to use the intermediate cache pool; otherwise, the PCIe device does not have the permission to use the intermediate cache pool.
[0056] Specifically, based on the information of the NUMA node connected to the PCIe device, it can be determined whether the NUMA node is a NUMA node of the NUMA architecture. If it is, the NUMA node connected to the PCIe device is valid; otherwise, the NUMA node connected to the PCIe device is invalid.
[0057] In some embodiments of this application, the validity of a transit cache pool can be determined by checking whether the total number of transit cache blocks contained in the transit cache pool is 0. If the total number of transit cache blocks contained in the local memory transit cache pool is not 0, then the local memory transit cache pool is valid. If the total number of transit cache blocks contained in the local memory transit cache pool is 0, then the local memory transit cache pool is invalid.
[0058] It should be noted that if the local memory intermediate cache pool is created successfully, the total number of intermediate cache blocks contained in the intermediate cache pool will be non-zero; if the local memory intermediate cache pool fails to be created, the total number of intermediate cache blocks contained in the intermediate cache pool will be zero. Of course, other reasons, such as memory corruption, may also cause the total number of intermediate cache blocks contained in the intermediate cache pool to be zero, resulting in the intermediate cache pool becoming invalid, which will not be elaborated here.
[0059] In some embodiments of this application, the transit cache pool includes a default cache pool and an extended cache pool. Both the default cache pool and the extended cache pool include multiple transit cache areas. Therefore, it can be determined whether the transit cache pool can store the target data by determining whether the default cache pool and the extended cache pool have transit cache areas capable of storing the target data.
[0060] If the default cache pool or the extended cache pool has a transit cache capable of storing the target data, the PCIe device obtains the address of the transit cache and performs a DMA write operation to directly write the target data to the transit cache corresponding to that address. If neither the default cache pool nor the extended cache pool has a transit cache capable of storing the target data, the PCIe device sends a creation instruction to the processor core currently running in the NUMA architecture. This creation instruction is used to instruct the processor core to create a new extended cache pool in local memory.
[0061] Of course, this application is not limited to this. In some other embodiments, the transit cache pool may only include the default cache pool. If the default cache pool includes multiple transit cache areas, then it can be determined whether the transit cache pool can store the target data by determining whether the default cache pool has a transit cache area that can store the target data.
[0062] If the default cache pool has a transit cache capable of storing the target data, the PCIe device obtains the address of the transit cache and performs a DMA write operation to directly write the target data to the transit cache corresponding to that address. If the default cache pool does not have a transit cache capable of storing the target data, the PCIe device sends a creation instruction to the processor core currently running in the NUMA architecture. This creation instruction is used to instruct the processor core to create a new extended cache pool in local memory.
[0063] In some embodiments of this application, such as Figure 4 As shown, Figure 4This is a flowchart illustrating a method for determining whether a transit cache pool can store target data, as disclosed in an embodiment of this application. The process begins with step S201: Determining whether the default cache pool has a transit cache area capable of storing the target data. If the default cache pool does, proceed to step S203; otherwise, proceed to step S202. S202: Randomly determining whether a preset number of expanded cache pools have a transit cache area capable of storing the target data. If the expanded cache pool does, proceed to step S203; otherwise, proceed to step S204. S203: Directly writing the target data to the transit cache area. S204: Sending a creation instruction to the currently running processor core in the NUMA architecture.
[0064] In some embodiments of this application, the preset number can be equal to 15. However, this application is not limited to this; in other embodiments, the preset number can be set according to the number of expanded cache pools, and this preset number is less than the number of expanded cache pools. Based on this, searching all expanded cache pools can be avoided, thus preventing increased time consumption and impacting device performance.
[0065] In some embodiments of this application, after the processor core currently running in the NUMA architecture receives the creation instruction, it is also necessary to determine whether a new extended cache pool has been created. If a new extended cache pool has not been created, the processor core responds to the creation instruction by creating a new extended cache pool in local memory. If a new extended cache pool has been created, the processor core sends a relay error message to the PCIe device. The relay error message is used to indicate that the extended cache pool creation has failed and to prevent the PCIe device from writing target data to target memory.
[0066] To avoid an excessively large default cache pool, which could lead to wasted or insufficient memory resources in applications that don't heavily rely on intermediate cache pools, the default cache pool size is typically set to meet the needs of most common applications. For example, the default cache pool size might be set to 256MB. Expanding the cache pool size depends on the page size of the operating system kernel on the processor core running the current NUMA architecture, calculated using the formula K = PAGE_SIZE * 2. (FORCE _ MAX _ ZONEORDER-1) Where K represents the size of the expanded cache pool, and PAGE_SIZE represents the page size of the operating system kernel of the processor core running on the current NUMA architecture. (FORCE _ MAX _ ZONEORDER-1) This indicates the maximum page size that the memory allocator can allocate at one time.
[0067] With PAGE_SIZE equal to 4KB, 2 (FORCE _ MAX _ ZONEORDER-1) Equal to 2 10 For example, the size K of the expanded cache pool is equal to 4MB. Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a default cache pool and an expanded cache pool disclosed in an embodiment of this application. The intermediate cache pool may include one default cache pool and x expanded cache pools. The default cache pool has a size of 256MB, the expanded cache pool has a size of 4MB, and each expanded cache pool includes m intermediate cache areas, where x and m are greater than or equal to 1.
[0068] After a NUMA-based processor core creates a new extended cache pool, the PCIe device can determine whether the new extended cache pool has a transit cache capable of storing the target data. If the new extended cache pool does have such a cache, the PCIe device obtains the address of the transit cache and performs a DMA write operation, directly writing the target data to the transit cache corresponding to that address. If the new extended cache pool does not have such a cache, the PCIe device sends a creation command to the NUMA-based processor core. If the processor core has not yet created a new extended cache pool, it responds to the creation command by creating a new extended cache pool in its local memory. If the processor core has already created a new extended cache pool, it sends a transit error message to the PCIe device.
[0069] It should be noted that, under normal circumstances, after creating a new extended cache pool, a transit cache area capable of storing the target data can be found in the new extended cache pool. Therefore, in order to avoid wasting resources, it is prohibited to repeatedly create new extended cache pools.
[0070] In some embodiments of this application, when determining whether any default cache pool or extended cache pool has a transit cache area capable of storing target data, it can be prioritized to determine whether a transit cache area in the default cache pool or extended cache pool that has affinity with the processor core currently running on the NUMA architecture can store the target data. If the transit cache area cannot store the target data, then the next transit cache area is searched.
[0071] It should be noted that each transit cache has a processor core with which it has affinity. Generally, one transit cache has affinity with one processor core. However, this application is not limited to this. In other embodiments, there may be a situation where one transit cache has affinity with multiple processor cores.
[0072] Because storing the target data in a transit cache that is compatible with the processor core currently running the NUMA architecture makes it easier for the processor core to quickly synchronize the target data in local memory to the target memory, when searching for a transit cache, the processor core that is compatible with the processor core currently running the NUMA architecture will be searched first.
[0073] When searching for any transit cache, the resource identifier of each transit cache block in the transit cache can be traversed. If the resource identifier determines that the transit cache has several consecutive unoccupied transit cache blocks, and the total storage space of these transit cache blocks is greater than or equal to the storage space required to store the target data, then the transit cache can store the target data; otherwise, the transit cache cannot store the target data.
[0074] If any transit buffer can store the target data, then the resource identifiers of the transit buffer blocks in that transit buffer that can store the target data can be marked as occupied, the resource identifiers of these transit buffer blocks can be updated, the addresses of these transit buffer blocks can be recorded, the number of occupied transit buffer blocks in that transit buffer can be updated, and the idle / busy status of these transit buffer blocks can be updated to busy.
[0075] In some embodiments of this application, if any extended cache pool is not occupied, the PCIe device sends a release instruction to the processor core currently running the NUMA architecture. This release instruction instructs the processor core to release the extended cache pool. After release, the extended cache pool is no longer used specifically for caching data written by PCIe devices on the same NUMA node, but is instead used by the processor core to store other data, thereby reducing the memory footprint of the extended cache pool. New extended cache pools can be created or released by the buddy system of the processor core currently running the NUMA architecture.
[0076] Specifically, the processor core can remove the expanded cache pool to be released from the cache pool list and release the memory occupied by the expanded cache pool back to the memory occupied by the buddy system. It can be understood that if all the intermediate cache blocks in the intermediate cache area are unoccupied, then the intermediate cache area is unoccupied; similarly, if all the intermediate cache blocks in the expanded cache pool are unoccupied, then the expanded cache pool is unoccupied.
[0077] In some embodiments of this application, all expanded cache pools can be queried periodically to check which expanded cache pools are not occupied. The query can be performed at preset intervals. If an expanded cache pool is not occupied after n consecutive queries, it is considered that the expanded cache pool will not be used in the short term, and thus the expanded cache pool can be released. Otherwise, the expanded cache pool is not released to avoid it being needed again immediately after being released. The preset interval can be 2 seconds, and n can be 30.
[0078] It's important to note that both querying and setting the status of the expanded buffer pool and the intermediate buffer require locking, for example, using the `rcu_read_lock()` or `rcu_read_unlock()` functions. Furthermore, releasing the expanded buffer pool also requires locking, for example, using the `spinlock` function within `struct p_io_tlb_mem`. The purpose of locking is to prevent information leakage.
[0079] It's also important to note that the default cache pool is created during the NUMA architecture initialization phase. The default cache pool is not managed by the currently running processor core of the NUMA architecture; that is, once successfully created, the default cache pool persists and will not be created or released by the currently running processor core. The currently running processor core can only create or release the extended cache pool. Furthermore, the extended cache pool may fail to create due to insufficient memory storage space. In addition, only PCIe devices can use the extended cache pool and the default cache pool; other devices cannot use them. PCIe devices can use pointer arrays to manage the extended cache pool and the default cache pool.
[0080] In some embodiments of this application, during the creation of the default cache pool, the effective NUMA nodes of the NUMA architecture can be detected first to confirm the number of currently working NUMA nodes. The memory address space in each NUMA node is recorded. Then, a memory allocator (such as memblock) is used to allocate a fixed-size memory block for each NUMA node as the default cache pool in the transit cache pool. Then, the important data structures of the transit cache pool are initialized, including the starting address of the transit cache pool, the ending address of the transit cache pool, the NUMA nodes connected to it, the number of transit cache areas, the number of transit cache blocks contained in each transit cache area, the aging time, the resource identifier, and the original address, etc. Then, the number of cache pools, transit cache areas, and transit cache blocks in the transit cache pool are counted.
[0081] In some embodiments of this application, after the PCIe device writes the target data directly to local memory, it can determine whether it needs to send a synchronization instruction to the processor core currently running on the NUMA architecture. If so, it can determine whether the first condition, the second condition, and the fifth condition are met. The first condition includes that the PCIe device has permission to use the intermediate cache pool; the second condition includes that the intermediate cache pool of local memory is valid; and the fifth condition includes that the target data has been written to the intermediate cache area in the intermediate cache pool of local memory.
[0082] If conditions one, two, and five are all met simultaneously, the PCIe device marks the memory pages in the transit cache containing the target data as synchronized and sends a synchronization instruction to the currently running processor core in the NUMA architecture, instructing the processor core to synchronize the target data in its local memory to the target memory. After the processor core synchronizes the target data in its local memory to the target memory, the PCIe device removes the receive cache mapping to prevent further data transfer to the target memory where the receive cache resides. Furthermore, if at least one of conditions one, two, and five is not met, the PCIe device will also remove the receive cache mapping.
[0083] In some embodiments of this application, after the target data is written to local memory, it can be determined sequentially whether the first condition, the second condition, and the fifth condition are met. For example... Figure 6 As shown, Figure 6 This is a flowchart illustrating a method for sending synchronization instructions to a currently running processor core in a NUMA architecture, as disclosed in an embodiment of this application. First, proceed to step S301: S301: Determine whether it is necessary to send synchronization instructions to the currently running processor core in the NUMA architecture. If so, proceed to step S302; otherwise, proceed to step S306. S302: Determine whether the PCIe device has permission to use the intermediate cache pool. If so, proceed to step S303; otherwise, proceed to step S306. S303: Determine whether the intermediate cache pool in local memory is valid. If valid, proceed to step S304; otherwise, proceed to step S306. S304: Determine whether the target data has been written to the intermediate cache area in the intermediate cache pool of local memory. If written, proceed to step S305; otherwise, proceed to step S306. S305: Send synchronization instructions to the currently running processor core in the NUMA architecture. S306: Release the receive cache mapping.
[0084] Of course, this application is not limited to this. In other embodiments, it is possible to determine whether the first condition, the second condition, and the fifth condition are met in parallel, or it is possible to determine whether the first condition, the second condition, and the fifth condition are met in other order, which will not be elaborated here.
[0085] In some embodiments of this application, after the PCIe device unmaps the receive cache, it is necessary to determine whether the first, second, and fifth conditions are met. If all three conditions are met, it is determined whether a synchronization instruction needs to be sent to the currently running processor core of the NUMA architecture. If so, the PCIe device marks the memory pages of the transit cache containing the target data as synchronized and sends a synchronization instruction to the currently running processor core of the NUMA architecture, instructing the processor core to synchronize the target data in its local memory to the target memory. If at least one of the first, second, and fifth conditions is not met, a synchronization instruction is not sent to the currently running processor core of the NUMA architecture, and the target data in its local memory is not synchronized to the target memory.
[0086] It should be noted that if the PCIe device has already sent a synchronization command to the currently running processor core of the NUMA architecture before unmapping the receive cache, then it is not necessary to send another synchronization command to the currently running processor core of the NUMA architecture after unmapping the receive cache. However, if, due to reasons such as not meeting at least one of the first, second, and fifth conditions, a synchronization command was not sent to the currently running processor core of the NUMA architecture before unmapping the receive cache, then it is necessary to send a synchronization command to the currently running processor core of the NUMA architecture after unmapping the receive cache.
[0087] In some embodiments of this application, such as Figure 6 As shown, after releasing the receive cache mapping, proceed to step S307. S307: Determine if the PCIe device has permission to use the intermediate cache pool. If yes, proceed to step S308; otherwise, proceed to step S313. S308: Determine if the local memory intermediate cache pool is valid. If valid, proceed to step S309; otherwise, proceed to step S313. S309: Determine if the target data has been written to the intermediate cache area in the local memory intermediate cache pool. If written, proceed to step S310; otherwise, proceed to step S313. S310: Determine if it is necessary to send a synchronization instruction to the currently running processor core of the NUMA architecture. If necessary, proceed to step S311. S311: Send a synchronization instruction to the currently running processor core of the NUMA architecture; otherwise, proceed to step S312. S312: Release the intermediate cache area back to the intermediate cache pool. S313: Do not send a synchronization instruction to the currently running processor core of the NUMA architecture.
[0088] Of course, this application is not limited to this. In other embodiments, it is possible to determine whether the first condition, the second condition, and the fifth condition are met in parallel, or it is possible to determine whether the first condition, the second condition, and the fifth condition are met in other order, which will not be elaborated here.
[0089] In some embodiments of this application, after the processor core currently running in the NUMA architecture synchronizes the target data in local memory to the target memory, it can release the intermediate cache area storing the target data back to the intermediate cache pool, mark and update the resource identifier of the intermediate cache block storing the target data in the intermediate cache area as unoccupied, clear the address of the intermediate cache block storing the target data to zero, update the number of occupied intermediate cache blocks in the intermediate cache area, update the idle / busy status of the intermediate cache area to idle, and mark the memory pages of the intermediate cache area storing the target data as unsynchronized.
[0090] As another optional implementation of the disclosure in this application, embodiments of this application also disclose a data writing method, applied to a NUMA architecture processor core, for example, applied to... Figure 1 The processor core of any processor in the NUMA architecture shown. The structure of this NUMA architecture is the same as that in the above embodiment, and will not be described again here.
[0091] like Figure 7 As shown, Figure 7 This is a flowchart illustrating another data writing method disclosed in an embodiment of this application. The method includes:
[0092] S401: Receives a synchronization command sent by a PCIe device. This synchronization command is used to synchronize target data in local memory to target memory, where local memory is the memory of the NUMA node connected to the PCIe device.
[0093] S402: In response to a synchronization command, synchronize the target data in local memory to the target memory.
[0094] In this embodiment, after the processor core currently running in the NUMA architecture receives the synchronization command sent by the PCIe device, it responds to the synchronization command by synchronizing the target data in the local memory of the NUMA node connected to the PCIe device to the target memory, so as to improve data write performance and data correctness.
[0095] In some embodiments of this application, if the PCIe device writes the target data directly to the intermediate cache pool in local memory, the processor core currently running in the NUMA architecture will synchronize the target data in the intermediate cache pool in local memory to the target memory.
[0096] In some embodiments of this application, if the processor core currently running the NUMA architecture receives a creation instruction sent by a PCIe device, it creates a new extended cache pool in local memory in response to the creation instruction. If the processor core currently running the NUMA architecture receives a release instruction sent by a PCIe device, it releases the corresponding extended cache pool in response to the release instruction.
[0097] In some embodiments of this application, if the processor core currently running on the NUMA architecture receives a creation instruction sent by a PCIe device, it first determines whether a new extended cache pool has been created. If a new extended cache pool has not been created, the processor responds to the creation instruction by creating a new extended cache pool in local memory. If a new extended cache pool has been created, the processor sends a relay error message to the PCIe device, which indicates that the extended cache pool creation has failed and prevents the PCIe device from writing target data to target memory.
[0098] As another optional implementation of the disclosure in this application, this application embodiment also discloses a PCIe device applied to a NUMA architecture. The NUMA architecture includes multiple NUMA nodes, and the PCIe device belongs to any NUMA node. The PCIe device includes a processor and a memory. The memory is connected to the processor and is used to store computer programs. The processor is used to implement the data writing method for the PCIe device as disclosed above by running the computer programs stored in the memory.
[0099] As another optional implementation of the disclosure of this application, embodiments of this application also disclose a processor applied to a NUMA architecture, the NUMA architecture including multiple NUMA nodes, the processor belonging to any NUMA node, the processor including at least one processor core, the processor core being configured to execute the data writing method applied to the processor core as disclosed in any of the above embodiments.
[0100] As another optional implementation of the disclosure in this application, embodiments of this application also disclose a NUMA architecture, which includes the PCIe device and the processor disclosed in any of the above embodiments.
[0101] As another optional implementation of the disclosure of this application, this application embodiment also discloses a computer-readable storage medium storing a computer program, which is executed by a processor to perform the data writing method disclosed in any of the above embodiments.
[0102] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0103] The above embodiments are merely illustrative of several implementation methods described in detail, but they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this specification, and these all fall within the scope of protection of this specification. Therefore, the scope of protection of this patent should be determined by the appended claims.
Claims
1. A data write method, characterized by, The application is applied to a PCIE device of a NUMA architecture, the NUMA architecture comprises a plurality of NUMA nodes and a plurality of PCIE devices, each of the PCIE devices is connected with one of the NUMA nodes, each of the NUMA nodes comprises a processor and a memory, the processor comprises at least one processor core, and the method comprises: receiving a DMA write request, the DMA write request is used for requesting to write target data directly to a target memory; in response to the DMA write request, determining whether the target memory is a local memory; the local memory is the memory of the NUMA node connected with the PCIE device; if the target memory is not the local memory, directly writing the target data to the local memory, and sending a synchronization instruction to the currently running processor core of the NUMA architecture, the synchronization instruction is used for instructing the processor core to synchronize the target data in the local memory to the target memory.
2. The method of claim 1, wherein, The memory of any one of the NUMA nodes comprises a transit cache pool, and the directly writing the target data to the local memory comprises: directly writing the target data to the transit cache pool of the local memory; the transit cache pool is used for buffering the data written by the PCIE device of the same NUMA node.
3. The method of claim 2, wherein, The directly writing the target data to the transit cache pool of the local memory comprises: determining whether a first condition, a second condition and a third condition are met; if the first condition, the second condition and the third condition are met simultaneously, directly writing the target data to the transit cache pool of the local memory; wherein the first condition comprises that the PCIE device has the permission to use the transit cache pool; the second condition comprises that the transit cache pool of the local memory is valid; and the third condition comprises that the transit cache pool can store the target data.
4. The method of claim 3, wherein, The transit cache pool at least comprises a default cache pool, the default cache pool comprises a plurality of transit cache areas, and the determining whether the transit cache pool can store the target data comprises: determining whether the default cache pool has a transit cache area capable of storing the target data; if the default cache pool does not have the transit cache area capable of storing the target data, sending a creation instruction to the currently running processor core of the NUMA architecture, the creation instruction is used for instructing the processor core to create a new extended cache pool in the local memory.
5. The method of claim 3, wherein, The transit cache pool comprises a default cache pool and an extended cache pool, the default cache pool and the extended cache pool both comprise a plurality of transit cache areas, and the determining whether the transit cache pool can store the target data comprises: determining whether the default cache pool and a preset number of extended cache pools have transit cache areas capable of storing the target data; if the default cache pool and the preset number of extended cache pools do not have the transit cache areas capable of storing the target data, sending a creation instruction to the currently running processor core of the NUMA architecture, the creation instruction is used for instructing the processor core to create a new extended cache pool in the local memory.
6. The method according to claim 4 or 5, characterized in that, In determining whether any default cache pool or extended cache pool has a transit cache area capable of storing the target data, priority is given to determining whether a transit cache area in the default cache pool or the extended cache pool that has an affinity with a currently running processor core of the NUMA architecture is capable of storing the target data.
7. The method according to claim 4 or 5, characterized in that, Further comprising: If any extended cache pool is not occupied, a release instruction is sent to the currently running processor core of the NUMA architecture, the release instruction being used to instruct the processor core to release the extended cache pool.
8. The method of claim 2, wherein, Before the sending of the synchronization instruction to the currently running processor core of the NUMA architecture, further comprising: determining whether a first condition, a second condition and a fifth condition are met; if the first condition, the second condition and the fifth condition are met at the same time, a synchronization instruction is sent to the currently running processor core of the NUMA architecture; wherein the first condition includes that the PCIE device has the right to use the transit cache pool; the second condition includes that the transit cache pool of the local memory is valid; and the fifth condition includes that the target data has been written to the transit cache pool of the local memory.
9. A data writing method, characterized by, Applied to a processor core of a NUMA architecture, the NUMA architecture comprising a plurality of NUMA nodes and a plurality of PCIE devices, each of the PCIE devices being connected to one of the NUMA nodes, each of the NUMA nodes comprising a processor and a memory, the processor comprising at least one processor core, the method comprising: receiving a synchronization instruction sent by the PCIE device, the synchronization instruction being used to synchronize target data in a local memory to a target memory, the local memory being a memory of a NUMA node connected to the PCIE device; in response to the synchronization instruction, synchronizing the target data in the local memory to the target memory.
10. The method of claim 9, wherein, The synchronizing of the target data in the local memory to the target memory comprises: synchronizing target data in a transit cache pool in the local memory to the target memory; the transit cache pool being used to cache data written by a PCIE device of the same NUMA node.
11. The method of claim 9, wherein, Further comprising: receiving a creation instruction sent by the PCIE device; in response to the creation instruction, creating a new extended cache pool in the local memory.
12. The method of claim 9, wherein, Further comprising: receiving a release instruction sent by the PCIE device; in response to the release instruction, releasing the corresponding extended cache pool.
13. A PCIE device, comprising: Applied to a NUMA architecture, the NUMA architecture comprising a plurality of NUMA nodes and a plurality of PCIE devices, each of the PCIE devices being connected to one of the NUMA nodes, the PCIE device comprising a processor and a memory; wherein the memory is connected to the processor, the memory being used to store a computer program; and the processor being used to realize the data writing method of any one of claims 1-8 by running the computer program stored in the memory.
14. A processor, comprising: The data write method is applied to a NUMA architecture, the NUMA architecture comprising a plurality of NUMA nodes and a plurality of PCIE devices, each of the PCIE devices being connected to one of the NUMA nodes, each of the NUMA nodes comprising a processor and a memory, the processor comprising at least one processor core, the processor core being configured to perform the data write method according to any one of claims 9-12.
15. A NUMA architecture, characterized by, The PCIE device according to claim 13 and the processor according to claim 14 are comprised.
16. A computer readable storage medium characterized by: The computer readable storage medium has stored thereon a computer program, the computer program being run by the processor to perform the data write method according to any one of claims 1-12.