Data processing system, method, device, medium and program product
By collecting acceleration card operation information through management equipment, predicting memory requirements and dynamically allocating memory, the resource competition problem when multiple acceleration cards apply for memory is solved, the system performance is improved and the flexible expansion of acceleration card memory is achieved.
Patent Information
- Application Number
- CN202510630314.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2025-06-13
- Estimated Expiration
- 2045-05-16
AI Technical Summary
In the acceleration card system, when multiple acceleration cards apply for memory from the host at the same time, it is easy to compete for resources such as bandwidth, resulting in an increase in data transmission delay and affecting overall performance.
Through the management device, collects the actual operation information of each accelerator card, predicts the memory demand of the target accelerator card, and selects the appropriate memory type, determines the memory allocation result of the accelerator card based on the memory demand and memory type, and realizes dynamic memory allocation of the accelerator card.
It avoids resource scrambling when multiple acceleration cards apply for memory at the same time, reduces data transmission delay, improves system performance, and realizes the expansion of memory for acceleration cards without affecting the system's operating performance.
Smart Images

Figure CN120144326A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technologies, and in particular, to a data processing system, method, device, medium, and program product. Background Art
[0002] Currently, the demand for the memory capacity of acceleration cards is increasing. Generally, the host memory can be used as the extended memory of the acceleration card to achieve the expansion of the acceleration card memory. However, the capacity of the host memory is limited. To provide more memory space for the acceleration card, the host extended memory can be provided for the acceleration card to use. However, the host needs to allocate memory for each acceleration card. If multiple acceleration cards apply for memory from the host simultaneously, it is easy to cause contention for resources such as bandwidth, resulting in an increase in data transmission latency and affecting the overall performance.
[0003] Therefore, how to expand the memory of the acceleration card without affecting the system operation performance is a problem that needs to be solved by those skilled in the art. Summary of the Invention
[0004] In view of this, the purpose of this application is to provide a data processing system, method, device, medium, and program product to expand the memory of the acceleration card without affecting the system operation performance.
[0005] In a first aspect, this application provides a data processing system, including: a management device, multiple acceleration cards, and multiple memory devices; the multiple acceleration cards are connected to the multiple memory devices through the management device; the management device is configured to: collect the actual operation information of each acceleration card; the management device is further configured to: for any target acceleration card among the multiple acceleration cards, predict the memory demand of the target acceleration card according to the actual operation information of the target acceleration card, and select at least one memory type, and determine the memory allocation result of the target acceleration card based on the memory demand and the at least one memory type; the target acceleration card is configured to: update the memory configuration according to the memory allocation result.
[0006] In a second aspect, this application provides a data processing method, which is applied to a management device. The management device is connected to multiple acceleration cards and multiple memory devices, and includes: collecting the actual operation information of each acceleration card; for any target acceleration card among the multiple acceleration cards, predicting the memory demand of the target acceleration card according to the actual operation information of the target acceleration card, and selecting at least one memory type; determining the memory allocation result of the target acceleration card based on the memory demand and the at least one memory type, so that the target acceleration card updates the memory configuration according to the memory allocation result.
[0007] In a third aspect, the present application provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the data processing method disclosed above.
[0008] In a fourth aspect, the present application provides a non-volatile storage medium for storing a computer program, wherein when the computer program is executed by a processor, the data processing method disclosed above is implemented.
[0009] In a fifth aspect, the present application provides a computer program product including computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the data processing method disclosed above are implemented.
[0010] As can be seen from the above solutions, the present application provides a data processing system, including: a management device, a plurality of acceleration cards, and a plurality of memory devices; the plurality of acceleration cards are connected to the plurality of memory devices through the management device; the management device is configured to: collect the actual operation information of each acceleration card; the management device is further configured to: for any target acceleration card among the plurality of acceleration cards, predict the memory demand of the target acceleration card according to the actual operation information of the target acceleration card, select at least one memory type, and determine the memory allocation result of the target acceleration card based on the memory demand and the at least one memory type; the target acceleration card is configured to: update the memory configuration according to the memory allocation result.
[0011] It can be seen that the beneficial effects of the present application are as follows: the plurality of acceleration cards are directly connected to the plurality of memory devices through the management device, and the actual operation information of each acceleration card is collected through the management device to allocate memory space to each acceleration card according to the actual operation information of the acceleration card. That is: each memory device is not perceived by the host, and these memory devices directly serve as the extended memory of each acceleration card, so there is no need for the host to allocate memory for each acceleration card, thus avoiding the situation of resource contention such as bandwidth when multiple acceleration cards apply for memory at the same time, and it is not easy to have data transmission delays, achieving the purpose of expanding the memory for the acceleration card without affecting the system operation performance.
[0012] Correspondingly, a data processing method, device, medium, and program product provided by the present application also have the above technical effects. Description of the Drawings
[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.
[0014] Figure 1 Schematic diagram of a data processing system disclosed in this application; Figure 2 Flowchart of a data processing method disclosed in this application; Figure 3 Another schematic diagram of a data processing system disclosed in this application; Figure 4 Disclosed in this application Figure 3 Corresponding system logic schematic diagram; Figure 5 Schematic diagram of an electronic device disclosed in this application; Figure 6 Structure diagram of a server provided by this application; Figure 7 Structure diagram of a terminal provided by this application. Specific implementation manners
[0015] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other examples obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.
[0016] Currently, in order to provide more memory space for the acceleration card, the host extended memory can be provided for the acceleration card to use. However, the host needs to allocate memory for each acceleration card. If multiple acceleration cards apply for memory from the host at the same time, it is easy to cause contention for resources such as bandwidth, resulting in an increase in data transmission latency and affecting the overall performance. For this reason, this application provides a data processing solution that can make each memory device not be perceived by the host. These memory devices directly serve as the extended memory of each acceleration card, so there is no need for the host to allocate memory for each acceleration card. Therefore, the contention for resources such as bandwidth that is likely to occur when multiple acceleration cards apply for memory at the same time is avoided, and data transmission latency is not likely to occur, achieving the purpose of expanding the memory for the acceleration card without affecting the system operation performance.
[0017] See Figure 1 As shown, the embodiments of this application disclose a data processing system, including: a management device, multiple acceleration cards: acceleration card 1 to acceleration card N, and multiple memory devices: memory device 1 to memory device N; the multiple acceleration cards are connected to the multiple memory devices through the management device.
[0018] Among them, the management device is used to: collect the actual operation information of each acceleration card. The management device is also used to: for any target acceleration card among multiple acceleration cards, predict the memory demand of the target acceleration card according to the actual operation information of the target acceleration card, select at least one memory type, and determine the memory allocation result of the target acceleration card based on the memory demand and at least one memory type; The target acceleration card is used to: update the memory configuration according to the memory allocation result.
[0019] It should be noted that each memory device provides the same or different types of memory, and a memory device can provide only DRAM (Dynamic Random Access Memory) resources or SSD (Solid State Drives) resources, or can also provide both DRAM resources and SSD resources at the same time. It can be seen that the memory type can be a DRAM resource type, an SSD resource type, etc.
[0020] Among them, the memory allocation result can record: which part of the memory address of which memory device is allocated to the target acceleration device, and which memory type this part of the memory address belongs to, etc. Specifically, the memory allocation result can include: the identification information of the memory device allocated to the target acceleration card, the device port, the port of the management device connected to this device port, the corresponding address segment, the memory type of this address segment, etc. Therefore, in one implementation, the management device is used to: based on the memory allocation result, determine the identification information of the memory device allocated to the target acceleration card; record the mapping relationship between the port of the identification information (i.e., the port of the management device connected to the memory device port) and the port of the target acceleration card (i.e., the port of the management device connected to the target acceleration card port) in the preset routing table.
[0021] Correspondingly, the target acceleration card is used to: based on the memory allocation result, determine the identification information of the memory device allocated to the target acceleration card and the corresponding address segment; record the identification information and the address segment in the preset memory mapping table. The preset memory mapping table is a table in the acceleration card that records its own available memory addresses, and each acceleration card is provided with a memory mapping table.
[0022] In this embodiment, the management device can collect the actual operation information of each acceleration card to sense the actual usage of each acceleration card in real time. For example, the current memory occupancy information of the acceleration card (which may include the memory usage rate, the size of the occupied memory, etc.), the current task queue information to be processed by the acceleration card (which may include the types of unfinished tasks and the number of unfinished tasks), etc. Therefore, in one implementation, the management device is used to: collect the memory occupancy information, task queue information, and throughput in each acceleration card as the actual operation information of the corresponding acceleration card. The task types may be: model inference tasks, model training tasks, batch data processing, background data backup, etc.
[0023] Correspondingly, different task types may correspond to different priorities and resource allocation rules, which can be specifically referred to the resource allocation table shown in Table 1. Therefore, in one implementation, the management device is used to: query the types of unfinished tasks in the task queue information and their corresponding allocable memory types in the preset resource allocation table to obtain at least one memory type.
[0024] Table 1
[0025] If different-priority tasks are running simultaneously in an acceleration card, the acceleration card can use the weighted fair queue algorithm to allocate bandwidth for different-priority tasks in the target acceleration card. When the highest-priority task starts running and the current available memory of the target acceleration card is lower than the memory required by the highest-priority task, the lowest-priority task in the target acceleration card is paused, and the current status information of the lowest-priority task is stored. If the management device finds that different-priority tasks are running in the acceleration cards applying for memory at the same time, the management device can also use the weighted fair queue algorithm to allocate bandwidth for different acceleration cards for their memory applications. Moreover, when the management device finds that the amount of memory available for the acceleration card where the high-priority task is located is less among all current memory devices, the memory application of the acceleration card where the low-priority task is located can be paused, and the current status information of its memory application is stored so that when there is more available memory, the memory application of the acceleration card where the low-priority task is located can continue.
[0026] In one example, corresponding trigger conditions can be set for the memory allocation of the acceleration card. For example, the management device is used to: if the memory occupancy information exceeds a preset first threshold and the number of unfinished tasks in the task queue information exceeds a preset second threshold, then predict the memory demand for the target acceleration card. The prediction method can be: using the trained prediction model with the historical memory occupancy information of the acceleration card, the types of unfinished tasks and the number of unfinished tasks in the task queue information as the data basis. Therefore, in one implementation manner, the management device is used to: query the historical memory occupancy information of the target acceleration card; predict the memory demand based on the historical memory occupancy information, the types of unfinished tasks and the number of unfinished tasks in the task queue information. For each acceleration card, the management device can record the memory occupancy information collected each time, so as to form the historical memory occupancy information of an acceleration card.
[0027] To facilitate the management of each memory device, a memory record table can be set in the management device to record the memory types (such as DRAM resource type, SSD resource type, etc.), total memory capacity, used memory amount, remaining available memory amount of each memory device, and even which part of the memory address of each memory device is allocated to which acceleration card, and these information can also be recorded. Therefore, in one implementation manner, the management device is used to: query the remaining available memory amount and memory type of each memory device in the preset memory record table to obtain a query result; determine the memory allocation result according to the query result, memory demand and at least one memory type; based on the memory allocation result, determine the identification information of the memory device allocated to the target acceleration card and the corresponding address segment; update the memory record table according to the identification information and address segment.
[0028] The memory capacity of each memory device can be allocated for use by each acceleration card, and it can also be released by each acceleration card, so as to realize the flexible utilization of the extended memory. In one implementation manner, the management device is used to: if the memory occupancy information in the target acceleration card is lower than a preset third threshold, then recycle a target amount of memory resources from the target acceleration card, and correspondingly update the memory record table and the routing table. Specifically, delete the mapping relationship between the port of the target acceleration card recorded in the routing table and the port of the corresponding memory device, and change the used memory amount, remaining available memory amount, etc. of the corresponding memory device in the memory record table. The target amount is an integer multiple of the set memory block size, where the set memory block size is the preset minimum memory size that can be applied for each time, such as set to 4K, 1G, etc. Correspondingly, the target acceleration card is used to: update the memory mapping table according to the target amount of memory resources. Specifically, delete the mapping relationship between the port of the target acceleration card recorded in the memory mapping table and the port of the corresponding memory device.
[0029] In one implementation, the management device includes a management controller and a switch controller; the management controller is connected to the switch controller through a first protocol (such as PCIe) and a second protocol (such as I2C); the switch controller is connected to multiple acceleration cards and multiple memory devices through a third protocol (such as CXL); the management controller is connected to the host and multiple acceleration cards through a fourth protocol (such as PCIe). PCIe (Peripheral Component Interconnect express) is a high-speed serial computer expansion bus standard used to connect various peripherals and expansion cards inside a computer, enabling high-bandwidth and low-latency data transmission. Among them, the management controller can perform relevant management operations on the switch controller, multiple acceleration cards, and multiple memory devices through I2C (Inter-Integrated Circuit Bus, a serial bus protocol for connecting integrated circuits), and collect the operating information of multiple acceleration cards and multiple memory devices through PCIe. The switch controller realizes memory read / write and memory allocation between multiple acceleration cards and multiple memory devices through CXL (Compute ExpressLink, an interconnection protocol for high-performance computing), which can improve efficiency. The management controller is connected to the host and multiple acceleration cards through PCIe, and can build a management network. Through this management network, the host can dispatch various tasks to each acceleration card to realize device management of each acceleration card by the host, etc. CXL is specifically a high-speed interface protocol that can achieve data exchange through low-latency and high-bandwidth connections, including three dynamic multiplexing sub-protocols: CXL.io, CXL.cache, and CXL.mem. CXL.io is similar to the IO protocol of PCIe, and this protocol is enabled for operations such as discovery and enumeration, error reporting, and host physical address lookup. CXL.cache is a protocol for accessing caches, which defines the interaction between the processor and the device, allowing the connected CXL devices to use request and response methods to efficiently cache the processor memory with extremely low latency. CXL.mem is a protocol for accessing memory, which provides the processor with access to the device-attached memory using load and store commands. Here, the processor acts as the master device, and the CXL device acts as the slave device, and it can support volatile and persistent memory architectures. Based on this, each acceleration card can perform read operations and / or write operations on the memory device connected to the port that has a mapping relationship with the port of the target acceleration card through the management device.
[0030] It can be seen that in this application, multiple acceleration cards are directly connected to multiple memory devices through a management device, and the management device collects the actual operation information of each acceleration card to allocate memory space to each acceleration card as needed according to the actual operation information of the acceleration card. That is: the host does not perceive these memory devices, and these memory devices directly serve as the extended memory of each acceleration card, so there is no need for the host to allocate memory for each acceleration card, thus avoiding the competition for resources such as bandwidth that is likely to occur when multiple acceleration cards apply for memory simultaneously, and it is not easy to have data transmission delays, achieving the purpose of expanding the memory for the acceleration card without affecting the system operation performance.
[0031] Next, a data processing method provided by an embodiment of this application will be introduced. The data processing method described below can be mutually referred to with other embodiments described in this article.
[0032] See Figure 2 As shown, an embodiment of this application discloses a data processing method, which is applied to a management device. The management device is connected to multiple acceleration cards and multiple memory devices.
[0033] The method provided in this embodiment includes: S201. Collect the actual operation information of each acceleration card.
[0034] S202. For any target acceleration card among the multiple acceleration cards, predict the memory demand of the target acceleration card according to the actual operation information of the target acceleration card, and select at least one memory type.
[0035] S203. Determine the memory allocation result of the target acceleration card based on the memory demand and at least one memory type, so that the target acceleration card updates the memory configuration according to the memory allocation result.
[0036] In one implementation manner, collecting the actual operation information of each acceleration card includes: collecting the memory occupancy information, task queue information, and throughput in each acceleration card as the actual operation information of the corresponding acceleration card.
[0037] In one implementation manner, if the memory occupancy information exceeds a preset first threshold and the number of unfinished tasks in the task queue information exceeds a preset second threshold, then the memory demand of the target acceleration card is predicted, and the following steps are performed: predicting the memory demand of the target acceleration card according to the actual operation information of the target acceleration card, and selecting at least one memory type; determining the memory allocation result of the target acceleration card based on the memory demand and at least one memory type, so that the target acceleration card updates the memory configuration according to the memory allocation result.
[0038] In one embodiment, predicting the memory requirement of a target accelerator card based on the actual running information of the target accelerator card includes: querying the historical memory occupancy information of the target accelerator card; predicting the memory requirement based on the historical memory occupancy information, the types of unfinished tasks, and the number of unfinished tasks in the task queue information.
[0039] In one embodiment, selecting at least one memory type includes: querying, in a preset resource allocation table, the types of unfinished tasks in the task queue information and their corresponding allocable memory types to obtain at least one memory type.
[0040] In one embodiment, determining the memory allocation result for the target accelerator card based on the memory requirement and at least one memory type includes: querying, in a preset memory record table, the remaining available memory amount and memory type of each memory device to obtain a query result; determining the memory allocation result based on the query result, the memory requirement, and at least one memory type. Subsequently, further based on the memory allocation result, determining the identification information of the memory device allocated to the target accelerator card and the corresponding address segment; updating the memory record table according to the identification information and the address segment.
[0041] In one embodiment, the management device also determines the identification information of the memory device allocated to the target accelerator card based on the memory allocation result; records the mapping relationship between the port of the identification information and the port of the target accelerator card in a preset routing table to update the routing table.
[0042] In one embodiment, the target accelerator card updates the memory configuration according to the memory allocation result, including: determining the identification information of the memory device allocated to the target accelerator card and the corresponding address segment based on the memory allocation result; recording the identification information and the address segment in a preset memory mapping table to update the memory mapping table.
[0043] In one embodiment, if the management device determines that the memory occupancy information in the target accelerator card is lower than a preset third threshold, it reclaims a target amount of memory resources from the target accelerator card and updates the memory record table and the routing table accordingly; the target amount is an integer multiple of the set memory block size; correspondingly, the target accelerator card is used to: update the memory mapping table according to the target amount of memory resources.
[0044] In one embodiment, in the target accelerator card, when the highest-priority task starts running, if the current available memory amount of the target accelerator card is lower than the memory amount required by the highest-priority task, the lowest-priority task in the target accelerator card is paused, and the current status information of the lowest-priority task is stored. The target accelerator card can also use the weighted fair queue algorithm to allocate bandwidth to different-priority tasks in the target accelerator card. The management device performs read operations and / or write operations on the memory device connected to the port that has a mapping relationship with the port of the target accelerator card.
[0045] Among them, for the more specific working processes of the steps in this embodiment, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details will not be elaborated herein.
[0046] It can be seen that this embodiment provides a data processing method. Multiple acceleration cards are directly connected to multiple memory devices through a management device, and the actual operation information of each acceleration card is collected through the management device, so as to allocate memory space to each acceleration card as needed according to the actual operation information of the acceleration card. That is: each memory device is not perceived by the host, and these memory devices directly serve as the extended memory of each acceleration card, so there is no need for the host to allocate memory for each acceleration card, thus avoiding the situation of resource contention such as bandwidth when multiple acceleration cards apply for memory simultaneously, and it is not easy to have data transmission delays, achieving the purpose of expanding the memory for the acceleration card without affecting the system operation performance.
[0047] Please refer to Figure 3 , in a data processing system, the management device includes a management node (i.e., a management controller) and a switching controller CXL Switch. The management node is connected to a port of the CXL Switch through the PCIe protocol, and the other ports of the CXL Switch are connected to multiple customized acceleration cards and multiple CXL memory expansion devices. The management node is also connected to the CXL Switch through I2C to quickly manage multiple customized acceleration cards and multiple CXL memory expansion devices through the I2C protocol, such as configuring memory devices. The management node and multiple customized acceleration cards construct a management network, and this management network can be connected to the host to enable the host to achieve task distribution and acceleration card management within this management network.
[0048] More specifically, multiple upstream ports of the CXL Switch are connected to N - 1 paths of customized acceleration cards. These acceleration cards integrate the relevant logic for memory application and allocation, so they are called customized acceleration cards. These acceleration cards logically constitute a computing subsystem. Multiple downstream ports of the CXL Switch are connected to M paths of CXL memory expansion devices, whose memory medium is DRAM or SSD, and support dynamic medium type recognition and hybrid addressing. The ratio of upstream and downstream ports can be flexibly adjusted as needed. A typical configuration is 8 upstream (7 customized acceleration cards + 1 management node) and 8 downstream CXL memory expansion devices, supporting up to 16 - way interconnection. The resource management control software Fabric Management (FM) in the management node is responsible for the topology management, resource scheduling, and policy coordination of the entire system, and can run in a separate management node or be embedded in the CXL Switch controller.
[0049] The management node is connected to the CXL Switch through the PCIe port, and the accelerator card and the CXL memory expansion device are connected to the CXL Switch through the CXL port. The BMCs in the management node and the CXL memory expansion device jointly control and manage the CXL Switch to implement the CXL memory expansion pool. The management node also monitors the local memory usage of each accelerator card and timely triggers the dynamic application, allocation, and release of extended memory, etc.
[0050] Referring to Figure 3 For the system described above, the corresponding logic control system can be Figure 4 . Figure 3 Each of the accelerator cards shown can form Figure 4 the computing subsystem in Figure 3 Each of the memory devices and the memories in the accelerator cards shown can form Figure 4 the storage subsystem in Figure 3 The management node, the CXL Switch, and the relevant modules (such as the simplified firmware core) responsible for address management in the accelerator cards shown can form Figure 4 the CXL subsystem in Figure 3 The PCIe ports shown form Figure 4 the PCIe subsystem in . The computing subsystem, the CXL subsystem, the PCIe subsystem, and the storage subsystem are interconnected through the system bus.
[0051] The CXL subsystem includes a simplified firmware core and a CXL root complex.
[0052] The CXL root complex is a bridge connecting the computing subsystem and CXL devices (such as CXL switch, CXL memory expansion device), responsible for establishing, maintaining the physical and logical connections between the management node and the CXL devices, recording the memory address ranges of the root ports of each CXL device, processing the memory requests from the accelerator cards, converting them into the formats required by the CXL protocol, and forwarding these requests to the corresponding CXL memory expansion devices. The role of the simplified firmware core is to support and enhance the functions of the root complex, enabling the system to more efficiently manage memory and data transfer, responsible for initializing the storage devices connected to the root complex, responsible for the initialization of the HDM (Host Managed Device Memory) decoder in the accelerator card, and managing the relevant physical memory address ranges to ensure that the system can correctly access and use these memory resources. Identify the CXL endpoints by checking the configuration space of the endpoints and the PCIe BAR addresses, and summarize the memory address spaces of each endpoint by checking the HDM registry. The computing subsystem includes the main computing cores or stream processors, etc., of the accelerator cards. The PCIe subsystem mainly includes PCIe ports and is responsible for establishing communication with the management node or the host. The storage subsystem also includes the DDR controller and the DDR storage unit in the accelerator card.
[0053] Referring to Figure 3 and Figure 4 for illustration, the management node can dynamically allocate extended memory according to the actual needs of the acceleration card, and the acceleration card can read and write to the extended memory. For example: when the local memory utilization rate of acceleration card 1 exceeds the threshold, the FM dynamically allocates 2GB of DRAM resources in the extended memory pool to this acceleration card through the CXL Switch, and the acceleration card directly accesses this memory through the CXL subsystem.
[0054] The management node is connected to the uplink port of the CXL Switch, monitors the memory usage of each acceleration card in real time, and dynamically allocates memory through the FM. Specifically, it can include the following steps: Step 1: Data collection and monitoring.
[0055] The management node collects the usage of the local memory of each acceleration card through the management network. For example, the local memory of acceleration card 1 occupies 95%, the local memory of acceleration card 2 occupies 60%, and the task queue information, such as a large batch of data is loaded in the task queue of acceleration card 1.
[0056] Step 2: Trigger the memory expansion strategy.
[0057] According to the preset rules, trigger the expansion request in a timely manner. Such as the preset threshold rule, if the local memory utilization rate of the acceleration card or the predicted acceleration card memory utilization rate exceeds 90% and the task priority running in the current acceleration card is high, then trigger the expansion request.
[0058] In some implementation manners, the FM can adopt a memory allocation strategy based on load prediction, so as to trigger the expansion strategy based on the prediction result. The prediction model is pre-trained and used to output the future memory demand. Data such as the historical memory utilization rate of the acceleration card (statistical by a certain time window), task type characteristics (model training, image inference, batch data processing, etc.), queue depth, and throughput (such as the number of queued instructions of the processor core) can be collected to train the prediction model. The model can be selected such as LSTM (Long Short Term Memory), random forest, etc. After training, use the trained prediction model to predict the future memory demand and trigger the expansion strategy according to the prediction result.
[0059] In some implementation methods, FM can adopt a priority scheduling algorithm to balance resource contention among multiple acceleration cards. As shown in Table 1, memory resource allocation and priority division are performed according to the types of tasks running on different acceleration cards. Specifically, tasks with higher priorities reserve more bandwidth to ensure low latency, while tasks with lower priorities share the remaining bandwidth. Specifically, algorithms such as weighted fair queue can be used for scheduling. A resource preemption policy can also be set. For example, when a task with the highest priority is triggered, it can preempt the extended memory allocated to a low-priority task and save the intermediate state of the low-priority task to the SSD.
[0060] Step 3: Address space allocation.
[0061] The FM of the management node checks the status of the global memory pool, selects an available area (locates the specific available memory device according to the records in the memory record table), generates an allocation instruction, and allocates the corresponding memory area to the corresponding acceleration card. The expansion result can be sent to the simplified firmware core in the corresponding acceleration card through the management network.
[0062] Step 4: HDM decoder update.
[0063] The acceleration card modifies the HDM decoder therein through the simplified firmware core to expand its physical address range to cover the newly expanded memory area. Specifically, the configuration parameters of the HDM decoder can be modified so that it can recognize and access the expanded physical memory area. For example, the base address and size in the register definition file of the HDM decoder are modified to adjust the upper limit of the physical address supported by the decoder; the device memory mapping table is updated, the BAR in the PCIe configuration space is adjusted, and the address window of the PCIe BAR is expanded to map a larger range of memory to the acceleration card.
[0064] Step 5: CXL Switch routing table synchronization.
[0065] The CXL Switch updates the port mapping relationship in the routing table, binds the downstream port connected to the corresponding memory device to the upstream port corresponding to Acceleration Card 1 to ensure correct routing of requests.
[0066] In one implementation, the specific steps for memory release are as follows: Step 1: Data collection and monitoring.
[0067] The management node collects the usage of the local memory of each acceleration card and the task queue information through the management network.
[0068] Step 2: Memory release policy trigger.
[0069] According to the preset rules, a memory release request is triggered in a timely manner. For example, according to the preset threshold rule, if the local memory usage rate is lower than 50% and the task priority is low, a release request is triggered.
[0070] Step 3: Address space release.
[0071] The FM of the management node checks the status of the global memory pool, generates a release instruction to reclaim the extended memory that has been allocated to the accelerator card, and sends the recovery result to the simplified firmware core of the corresponding accelerator card through the management network.
[0072] Step 4: HDM decoder update.
[0073] The simplified firmware core of the accelerator card modifies the HDM decoder of the accelerator card and reduces its physical address range to remove the addresses of the released extended memory area.
[0074] Step 5: CXL Switch routing table synchronization.
[0075] CXL Switch updates the port mapping relationship, unbinds the downstream port of the corresponding memory device from the upstream port corresponding to the accelerator card, and releases the routing relationship between the two.
[0076] In one implementation, the process of the accelerator card reading the extended memory is as follows: Step 1: Initiate a read request: The accelerator card computing unit generates a read operation.
[0077] Step 2: The storage subsystem determines the read operation address and finds out the DDR range, and transfers it to the CXL subsystem for processing.
[0078] Step 3: The root complex resolves the address, determines that the DDR range belongs to the mapping range of the extended memory, converts the request into CXL protocol format, and adds the target endpoint and transaction tag.
[0079] Step 4: The CXL Switch routes the read request to the extended memory connected to the corresponding downstream port according to the port mapping table.
[0080] Step 5: The memory extension performs a read operation, returning the data block from the target address.
[0081] Step 6: The response data is transmitted back to the root complex of the accelerator card through the CXL Switch, converted into the accelerator card local bus format, and then sent to the accelerator card computing unit.
[0082] In one implementation, the process of the accelerator card writing to the extended memory is as follows: Step 1: Initiate a write request: The accelerator card computing unit generates a write operation.
[0083] Step 2: The storage subsystem determines the write operation address and finds out the DDR range, and transfers it to the CXL subsystem for processing.
[0084] Step 3: The root complex resolves the address, determines that the DDR range belongs to the mapped range of the extended memory, converts the request into the CXL protocol format, and adds the target endpoint and transaction label.
[0085] Step 4: The CXL Switch routes the read request to the extended memory connected to the downstream port according to the port mapping table.
[0086] Step 5: The memory extension performs a write operation, writes the data to the target address, and returns an acknowledgment signal.
[0087] Step 6: The acknowledgment signal is sent back to the root complex of the accelerator card through the CXL Switch, converted into the local bus format of the accelerator card, and then notified to the computing unit of the accelerator card.
[0088] It can be seen that in this embodiment, an extended memory pool shared by multiple accelerator cards is constructed based on the CXL high-speed interconnect protocol, which can realize the dynamic memory extension of the accelerator card and improve the utilization rate of the computing resources of the accelerator card in computing scenarios such as machine learning. This scheme design can bypass the host memory, realize the direct memory extension of the accelerator card, reduce data transfer, and effectively reduce the access latency of the extended memory. The management node can dynamically allocate extended memory resources based on the real-time monitoring of the memory usage of the accelerator card and the subsequent task queue, which can effectively avoid resource idleness and improve the overall resource utilization rate of the system.
[0089] In terms of architecture design, by integrating the CXL subsystem (including the root complex and the simplified firmware core) and the single-layer CXL Switch topology design, an extended memory pool shared by multiple accelerator cards is constructed to achieve direct extension bypassing the host memory. The root complex is responsible for the conversion and routing between local memory requests and the CXL protocol. The simplified firmware core dynamically manages the address space and the HDM decoder. Combined with the flexible port mapping of the Switch, the traditional PCIe bus contention problem is solved, and the bandwidth utilization efficiency of the extended memory is improved.
[0090] A cooperative control mechanism for dynamic memory capacity management is provided in memory allocation. By the management node's real-time monitoring of the memory usage status of the accelerator card to trigger the memory extension or release process, combined with the dynamic adjustment of the HDM decoder (extending or reducing the physical address range) and the synchronous update of the CXL Switch routing table, the on-demand allocation of pooled memory is realized. This mechanism ensures the dynamic elastic scheduling of extended memory resources among multiple accelerator cards through the cooperation of firmware and hardware (such as FM global scheduling, address space binding and unbinding), avoiding waste of storage resources.
[0091] Next, an electronic device provided by an embodiment of the present application is introduced. The following-described electronic device can be referred to each other with other embodiments described in this article. This electronic device can be any device described in the foregoing embodiments, such as a management device, an accelerator card, etc.
[0092] See Figure 5 As shown, an embodiment of the present application discloses an electronic device, including: a memory 501 for storing a computer program; a processor 502 for executing the computer program to implement the method disclosed in any of the above embodiments.
[0093] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: collecting the actual operation information of each acceleration card; for any target acceleration card among multiple acceleration cards, predicting the memory demand of the target acceleration card according to the actual operation information of the target acceleration card, and selecting at least one memory type; determining the memory allocation result of the target acceleration card based on the memory demand and at least one memory type, so that the target acceleration card updates the memory configuration according to the memory allocation result.
[0094] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: collecting the memory occupancy information, task queue information, and throughput in each acceleration card as the actual operation information of the corresponding acceleration card.
[0095] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: if the memory occupancy information exceeds a preset first threshold and the number of unfinished tasks in the task queue information exceeds a preset second threshold, then predict the memory demand for the target acceleration card.
[0096] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: querying the historical memory occupancy information of the target acceleration card; predicting the memory demand according to the historical memory occupancy information, the types of unfinished tasks, and the number of unfinished tasks in the task queue information.
[0097] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: querying the types of unfinished tasks in the task queue information and the corresponding allocable memory types in a preset resource allocation table to obtain at least one memory type.
[0098] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: querying the remaining available memory amount and memory type of each memory device in a preset memory record table to obtain a query result; determining the memory allocation result according to the query result, the memory demand, and at least one memory type; determining the identification information of the memory device allocated to the target acceleration card and the corresponding address segment based on the memory allocation result; updating the memory record table according to the identification information and the address segment.
[0099] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: Based on the memory allocation result, determine the identification information of the memory device allocated to the target acceleration card; Record the mapping relationship between the port of the identification information and the port of the target acceleration card in the preset routing table.
[0100] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: Based on the memory allocation result, determine the identification information of the memory device allocated to the target acceleration card and the corresponding address segment; Record the identification information and the address segment in the preset memory mapping table.
[0101] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: If the memory occupancy information in the target acceleration card is lower than the preset third threshold, recycle the memory resources of the target size from the target acceleration card, and update the memory record table and the routing table accordingly; The target size is an integer multiple of the set memory block size.
[0102] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: Update the memory mapping table accordingly according to the memory resources of the target size.
[0103] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: If when the highest-priority task starts running, the current available memory of the target acceleration card is lower than the memory amount required by the highest-priority task, pause the lowest-priority task in the target acceleration card, and store the current status information of the lowest-priority task.
[0104] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: Use the weighted fair queueing algorithm to allocate bandwidth to different-priority tasks in the target acceleration card.
[0105] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: Perform read operations and / or write operations on the memory device connected to the port that has a mapping relationship with the port of the target acceleration card through the management device.
[0106] Further, an embodiment of the present application also provides an electronic device. Among them, the above-mentioned electronic device can be either a Figure 6 server as shown, or a Figure 7 terminal as shown. Figure 6 and Figure 7 are both structural diagrams of electronic devices shown according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the scope of use of the present application.
[0107] Figure 6 This is a schematic structural diagram of a server provided by an embodiment of the present application. The server may specifically include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. Among them, the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the relevant steps in the data processing disclosed in any of the foregoing embodiments.
[0108] In this embodiment, the power supply is used to provide working voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and specific limitations are not imposed here; the input / output interface is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and specific limitations are not made here.
[0109] In addition, as a carrier for resource storage, the memory can be a read-only memory, a random access memory, a disk, or an optical disc, etc., and the resources stored thereon include an operating system, a computer program, and data, etc., and the storage method can be temporary storage or permanent storage.
[0110] Among them, the operating system is used to manage and control each hardware device and computer program on the server to enable the processor to perform operations and processing on the data in the memory, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the data processing method disclosed in any of the foregoing embodiments, the computer program can further include a computer program that can be used to complete other specific tasks. In addition to data such as update information of the application program, the data can also include data such as developer information of the application program.
[0111] Figure 7 This is a schematic structural diagram of a terminal provided by an embodiment of the present application. The terminal may specifically include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer, etc.
[0112] Generally, the terminal in this embodiment includes: a processor and a memory.
[0113] Among them, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.
[0114] The memory may include one or more computer non-volatile storage media, and the computer non-volatile storage media may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In this embodiment, the memory is at least used to store the following computer programs. After the computer programs are loaded and executed by the processor, they can implement the relevant steps in the data processing methods executed by the terminal side disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory may also include an operating system and data, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, update information of the application program.
[0115] In some embodiments, the terminal may further include a display screen, an input / output interface, a communication interface, sensors, a power supply, and a communication bus.
[0116] Those skilled in the art can understand that Figure 7 the structure shown in
[0117] does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure.
[0118] A non-volatile storage medium for storing a computer program, wherein when the computer program is executed by a processor, it implements the data processing method disclosed in the foregoing embodiments. The non-volatile storage medium is a computer-readable non-volatile storage medium. As a carrier for storing resources, it can be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon include an operating system, a computer program, and data, etc. The storage method can be transient storage or permanent storage.
[0119] Next, a computer program product provided by an embodiment of the present application will be introduced. A computer program product described below can be referred to each other with other embodiments described in this article.
[0120] A computer program product includes computer programs / instructions, and when the computer programs / instructions are executed by a processor, they implement the steps of the data processing method disclosed above.
[0121] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.
[0122] The steps of the method or algorithm described in combination with the embodiments disclosed in this article can be directly implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of non-volatile storage medium well-known in the technical field.
[0123] Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.
Claims
1. A data processing system, characterized in that: include: A management device, a plurality of accelerator cards and a plurality of memory devices; the plurality of accelerator cards are connected to the plurality of memory devices via the management device; The management device is used to: collect actual operation information of each accelerator card; The management device is further used to: for any target accelerator card among the multiple accelerator cards, predict the memory requirement of the target accelerator card according to the actual operation information of the target accelerator card, select at least one memory type, and determine the memory allocation result of the target accelerator card based on the memory requirement and the at least one memory type; The target acceleration card is used to update the memory configuration according to the memory allocation result.
2. The data processing system according to claim 1, characterized in that The management device is used for: The memory usage information, task queue information, and throughput of each accelerator card are collected as the actual operation information of the corresponding accelerator card.
3. The data processing system according to claim 2, characterized in that: The management device is used for: If the memory usage information exceeds a preset first threshold and the number of unfinished tasks in the task queue information exceeds a preset second threshold, a memory requirement is predicted for the target accelerator card.
4. The data processing system according to claim 2, characterized in that: The management device is used for: Query the historical memory usage information of the target accelerator card; The memory requirement is predicted based on the historical memory occupancy information, the type of unfinished tasks and the number of unfinished tasks in the task queue information.
5. The data processing system according to claim 2, characterized in that: The management device is used to query the unfinished task types and their corresponding allocatable memory types in the task queue information in a preset resource allocation table to obtain the at least one memory type.
6. The data processing system according to claim 1, characterized in that: The management device is used for: In the preset memory record table, query the remaining available memory amount and memory type of each memory device to obtain the query result; Determine the memory allocation result according to the query result, the memory requirement and the at least one memory type; Based on the memory allocation result, determining identification information of a memory device allocated to the target accelerator card and a corresponding address segment; The memory record table is updated according to the identification information and the address segment.
7. The data processing system according to claim 1, characterized in that: The management device is used for: Based on the memory allocation result, determining identification information of a memory device allocated to the target accelerator card; The mapping relationship between the port of the identification information and the port of the target acceleration card is recorded in a preset routing table.
8. The data processing system according to claim 1, characterized in that: The target accelerator card is used for: Based on the memory allocation result, determining identification information of a memory device allocated to the target accelerator card and a corresponding address segment; The identification information and the address segment are recorded in a preset memory mapping table.
9. The data processing system according to claim 1, characterized in that: The management device is used for: If the memory occupancy information in the target accelerator card is lower than a preset third threshold, memory resources of a target size are recovered from the target accelerator card, and a memory record table and a routing table are updated accordingly; the target size is an integer multiple of a set memory block size; Correspondingly, the target accelerator card is used to update the memory mapping table according to the memory resources of the target size.
10. The data processing system according to claim 1, characterized in that: The target accelerator card is used for: If the current available memory of the target accelerator card is lower than the memory required by the highest priority task when the highest priority task starts to run, the lowest priority task in the target accelerator card is suspended, and the current status information of the lowest priority task is stored.
11. The data processing system according to any one of claims 1 to 10, characterized in that: The management device includes a management controller and a switching controller; The management controller is connected to the switching controller via a first protocol and a second protocol; The switch controller connects the plurality of accelerator cards and the plurality of memory devices via a third protocol; The management controller connects the host and the plurality of acceleration cards via a fourth protocol.
12. A data processing method, characterized in that: Applied to a management device, the management device is connected to multiple accelerator cards and multiple memory devices, including: Collect the actual operation information of each accelerator card; For any target accelerator card among the multiple accelerator cards, predict the memory requirement of the target accelerator card according to actual operation information of the target accelerator card, and select at least one memory type; A memory allocation result of the target accelerator card is determined based on the memory requirement and the at least one memory type, so that the target accelerator card updates a memory configuration according to the memory allocation result.
13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to claim 12.
14. A non-volatile storage medium, characterized in that: Used to store a computer program, wherein the computer program implements the method according to claim 12 when executed by a processor.
15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method of claim 12 is implemented.
Citation Information
Patent Citations
Streaming data heterogeneous computing memory optimization method based on dynamic telescopic memory pool
CN114048025A
Method, system and device for efficient memory replacement between GPU devices and storage medium
CN114860461A
Graph search method and device, computer equipment and storage medium
CN116775920A
Parallel scheme planning and model training method, program product, medium and equipment
CN118227314A
Data processing method, product, server and medium
CN118409871A
Cited By
Computer system, resource sharing method, device and equipment, medium and product
CN120371767A
Storage system, method and device, electronic equipment, medium and product
CN120447838A
Distributed system, data processing method, equipment, medium and program product
CN120578509A
A distributed system, data processing method, device, medium and program product
CN120578509B
Inference control method, device configuration method, device, equipment, medium and product
CN120725078A
