Write task processing method and device, equipment and storage medium
By acquiring write task parameters and splitting them into multiple write commands, and combining data migration paths and pre-reading data during the sending cycle, the bubbling problem across the 4KB boundary in the PCIe protocol was solved, achieving efficient write task scheduling and data migration, and improving DMA transfer efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JINAN MAIWEI INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2026-01-30
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies suffer from data bubbling issues when the PCIe protocol crosses 4KB boundaries, resulting in low DMA transfer efficiency, especially with significant performance loss when MPS is small.
By obtaining the parameters of the write task, including the cache index, initial send length, secondary send length, send cycle data, and secondary data shift value, the write task is split into multiple write commands. Data is pre-read according to the data migration path and send cycle to ensure the coordination between data migration and write commands and avoid transmission gaps.
It completely eliminates the cavitation problem in DMA cross-boundary transfer, realizes efficient scheduling of write tasks and data migration, and improves transmission efficiency.
Smart Images
Figure CN122152737A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of task scheduling technology, and in particular to write task processing methods, apparatus, devices and storage media. Background Technology
[0002] With the rapid development of cloud computing, artificial intelligence, and high-throughput network applications, the demand for data transmission speed and efficiency is exploding. PCIe (peripheral component interconnect express) DMA (direct memory access) technology, leveraging the high bandwidth of the PCIe bus, enables data to be transferred rapidly between PCIe devices and memory.
[0003] To circumvent the 4KB boundary issue in the PCIe protocol, related technologies split cross-boundary write tasks into two 4KB aligned subtasks, and then further split them into MWR (Memory Write) commands based on MPS (Maximum Payload Size). MWR command sending and cache data reading operations are executed in parallel. While this ensures data transmission compatibility and adapts to the transmission characteristics of the PCIe bus to some extent, an MWR response requires two reads from the cache, leading to discontinuous data transmission and transmission bubbles. This performance penalty is particularly severe when MPS is low, significantly limiting DMA transfer efficiency. Therefore, solving the bubble problem in DMA cross-4KB boundary transfers and efficiently implementing task scheduling and data migration has become an urgent problem to be solved. Summary of the Invention
[0004] This application provides a write task processing method, apparatus, device, and storage medium to at least solve the problem of low efficiency in task scheduling and data migration in related technologies.
[0005] This application provides a write task processing method, including:
[0006] Upon receiving a cross-boundary write task, obtain the parameters of the write task; these parameters include the cache index, the initial send length, the secondary send length, the send cycle data, and the secondary data shift value. The send cycle data includes the initial send cycle and the secondary send cycle. Based on the preset unit, the initial transmission length, and the secondary transmission length, the write task is split into multiple write commands, which are then written into the write command queue. Based on the cache index, determine the data migration path and the number of migrations; Based on the data migration path, migration count, and secondary data shift value, data is pre-read from the data cache to obtain the target data to be written to the target storage area; Based on the first and second sending cycles, the target data is stored in the write data queue; Write commands are obtained sequentially from the write command queue. In response to the write command, the target data is read from the write data queue and written to the target storage area.
[0007] This application also provides a write task processing apparatus, including: The parameter acquisition module is used to acquire the parameters of the write task when a cross-boundary write task is received. The parameters include the cache index, the initial send length, the secondary send length, the send cycle data, and the secondary data shift value. The send cycle data includes the initial send cycle and the secondary send cycle. The command queue construction module is used to split the write task into multiple write commands based on the preset unit, the first send length, and the second send length, and write the multiple write commands into the write command queue. The migration parameter determination module is used to determine the data migration path and number of migrations based on the cache index; The target data determination module is used to pre-read data from the data cache based on the data moving path, number of moving times and secondary data shift value to obtain the target data to be written to the target storage area; The data queue construction module is used to store target data into the write data queue according to the first sending cycle and the second sending cycle; The write task processing module is used to obtain write commands sequentially from the write command queue, read target data from the write data queue in response to the write command, and write the target data to the target storage area.
[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described write task processing methods when executing the computer program.
[0009] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described write task processing methods.
[0010] In some embodiments of this application, based on the received cross-boundary write task, relevant parameters of the write task are obtained. The write task is split into multiple write commands according to a preset unit and transmission length, and written to a write command queue. Simultaneously, the data migration path, migration count, and secondary data shift value are determined based on the cache index. Target data is obtained by pre-reading data from the data cache area, and then the target data is stored in the write data queue according to the transmission cycle data. Finally, write commands are sequentially obtained, and the corresponding target data is synchronously read from the write data queue and written to the target storage area. In this way, by obtaining all core parameters in advance, the coordination between data migration and write command splitting can be ensured. For example, pre-reading data while splitting write commands can avoid transmission gaps caused by data preparation delays, thus avoiding bubbling problems. In this way, bubbling problems in the DMA cross-boundary transmission process can be completely eliminated, and write task scheduling and data migration can be achieved efficiently. Attached Figure Description
[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 A schematic diagram of the conventional DMA transmission process provided in this application; Figure 2 A flowchart illustrating a write task processing method provided for some embodiments of this application; Figure 3 A schematic diagram of the overall structure of the write task sending device provided in this application; Figure 4 A schematic diagram of the pre-computation unit provided in this application; Figure 5 A schematic diagram illustrating the calculation of the initial transmission length and the secondary transmission length provided for this application; Figure 6 This is a diagram illustrating the data storage provided in this application; Figure 7 A schematic diagram illustrating the calculation of the total transmission cycle provided for this application; Figure 8 A schematic diagram of the transmission management unit workflow provided in this application; Figure 9 Schematic diagram of a write task processing apparatus provided for some embodiments of this application; Figure 10 A schematic diagram of the modules of an electronic device provided for some embodiments of this application. Detailed Implementation
[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0016] See also Figure 1 This is a schematic diagram of the traditional DMA transmission process provided in this application. Figure 1 In this process, cross-boundary write tasks are split into initial send subtasks and secondary send subtasks, corresponding to the data to be transmitted within the current memory page and the remaining data to be transmitted after crossing the page, respectively. Based on MPS, the initial send subtask is first split into segments to generate multiple MWR commands with uniform specifications, and then the same splitting logic is executed on the secondary send subtask. All the split MWR commands are scheduled and managed in the order of initial send first and secondary send last, while interacting with the internal cache to obtain the target data matching the MWR command from the internal cache.
[0017] In some technical solutions, to circumvent the 4KB boundary issue of the PCIe protocol and adapt to the data transmission rate requirements of cloud computing, artificial intelligence, and high-throughput network applications, related technologies split the write task across the 4KB boundary into two 4KB aligned subtasks, and then further split them into MWR commands according to MPS. Simultaneously, MWR command sending and buffer data reading operations are executed in parallel. While this ensures data transmission compatibility and, to some extent, adapts to the high-bandwidth transmission characteristics of the PCIe bus, leveraging the fundamental transmission advantages of PCIe DMA technology, a single MWR response data requires two read operations from the buffer, leading to discontinuous data transmission and transmission bubbles. This performance loss is particularly severe when MPS is low, significantly limiting DMA transmission efficiency. Therefore, how to solve the bubble problem in DMA transmission across the 4KB boundary and efficiently achieve task scheduling and data migration has become an urgent problem to be solved.
[0018] In view of this, this application provides a write task processing method that can solve the above problems. The write task processing method can be applied to a DMA scheduling device. (See also...) Figure 2 This is a flowchart illustrating a write task processing method provided in some embodiments of this application. Figure 2 In Chinese, the task processing method includes the following steps: Step S101: Upon receiving a cross-boundary write task, obtain the parameters of the write task; wherein, the parameters include cache index, initial transmission length, secondary transmission length, transmission cycle data and secondary data shift value, and the transmission cycle data includes the initial transmission cycle and the secondary transmission cycle.
[0019] Specifically, a cross-boundary write task refers to a PCIe DMA write task that crosses the 4KB memory page boundary. That is, the transmission range from the starting write address to the target write address of the write task spans the 4KB memory page boundary, which is divided into 4096-byte units. This write task is initiated by the PCIe device and is used to realize high-speed data transmission between the PCIe device and the target storage area.
[0020] Specifically, a cache index refers to index information used to locate the storage location of the data corresponding to a write task in the data cache area.
[0021] Specifically, the initial send length refers to the number of bytes that can be continuously transferred from the starting write address of the write task to the nearest 4KB memory page boundary.
[0022] Specifically, the second transmission length refers to the number of bytes remaining to be transmitted after the write task crosses the 4KB memory page boundary.
[0023] Specifically, the transmission cycle data refers to the data pre-read count used to determine the number of times data is read from the data buffer, including the first transmission cycle and the second transmission cycle, which correspond to the number of data pre-read counts for the first transmission length and the second transmission length, respectively.
[0024] Specifically, the secondary data shift value refers to the parameter for shifting and aligning data after splicing across 4KB boundaries. This ensures that the spliced data across boundaries can be accurately aligned, avoids data transmission misalignment, and eliminates transmission bubbles.
[0025] Understandably, the system first receives the DMA write task issued by the PCIe device. If it is determined to be a cross-boundary write task, it obtains the cache index, the first transmission length, the second transmission length, the transmission cycle data, and the second data shift value.
[0026] Understandably, the secondary data shift value and preset unit can be dynamically adjusted based on the bit width specification of the data buffer and the actual transmission load of the PCIe bus. When the bus load is higher than the preset threshold and the transmission delay increases, the calculation precision of the secondary data shift value can be appropriately reduced to adapt to a smaller preset unit specification, prioritizing the stability of data transmission. When the bus load is low and high-efficiency transmission is pursued, the calculation precision of the secondary data shift value can be increased to adapt to a larger preset unit specification, further improving the processing efficiency of cross-boundary write tasks and adapting to the transmission needs in different scenarios.
[0027] Step S102: Based on the preset unit, the first transmission length, and the second transmission length, split the write task to obtain multiple write commands, and write the multiple write commands into the write command queue.
[0028] Specifically, the preset unit refers to MPS, which is the maximum data length that can be transmitted in a single transaction as specified in the PCIe protocol. It is the standard for splitting write commands.
[0029] Specifically, a write command is an instruction used to control the writing of data to the target storage area. It is an MWR command, and each write command corresponds to a fixed-length data segment.
[0030] Specifically, the write command queue refers to a queue used to temporarily cache multiple write commands after they have been split, and it adopts a first-in-first-out storage rule.
[0031] Understandably, when there are write tasks and the write command queue is not full, the write task partitioning process is initiated. The write tasks are partitioned based on the preset unit, the initial send length, and the secondary send length. After the write commands are partitioned, all write commands are written to the write command queue in the order of partitioning.
[0032] Step S103: Determine the data migration path and migration count based on the cache index.
[0033] Specifically, the data migration path refers to the transmission path of the data to be transferred from the data buffer to the target storage area.
[0034] Specifically, the number of moves refers to the number of times it is required to move all the data to be transferred corresponding to the write task from the data cache to the target storage area.
[0035] Understandably, the process involves first accurately locating the starting storage location and distribution range of the data to be transferred in the data cache area using the cache index, planning the transfer path from the cache area to subsequent stages, and calculating the number of transfers required to complete all data transfers.
[0036] Step S104: Based on the data migration path, migration count, and secondary data shift value, pre-read data from the data cache to obtain the target data to be written to the target storage area.
[0037] Specifically, the data buffer refers to the DMA task data buffer, which is the source of the data.
[0038] Specifically, the target storage area refers to the storage area where the data to be transferred in the write task will eventually be written.
[0039] Specifically, target data refers to data that, after basic processing such as pre-reading and alignment, meets the specifications for writing to the target storage area and can be directly used for subsequent write operations.
[0040] Understandably, data is first read from the data cache in batches according to the preset data migration path and the predetermined number of migrations. For data misalignment, the pre-read data is aligned and adjusted using secondary data shift values to correct the deviation. After the above pre-reading and adjustment operations, all the pre-read data that meets the requirements is integrated to finally form the target data to be written to the target storage area.
[0041] Step S105: Store the target data in the write data queue according to the first sending cycle and the second sending cycle.
[0042] Specifically, the write data queue adopts a first-in-first-out storage rule that matches the write command queue, and is a storage medium used to temporarily cache target data.
[0043] Understandably, the target data corresponding to the first sending length is first stored in batches into the write data queue according to the first sending cycle, and then the target data corresponding to the second sending length is stored in batches into the write data queue according to the second sending cycle. The storage order of the target data in the write data queue corresponds one-to-one with the writing order of the write commands in the write command queue.
[0044] Step S106: Sequentially obtain write commands from the write command queue, read target data from the write data queue in response to the write command, and write the target data to the target storage area.
[0045] Understandably, write commands are retrieved one by one according to the storage order of the write command queue; each time a write command is retrieved, the corresponding response operation is triggered to accurately read the target data that matches it from the write data queue; after the target data is read, the target data is written to the target storage area.
[0046] In summary, in the technical solutions of some embodiments of this application, based on the received cross-boundary write task, the relevant parameters of the write task are obtained. The write task is split into multiple write commands according to a preset unit and transmission length and written to a write command queue. Simultaneously, the data migration path, migration count, and secondary data shift value are determined based on the cache index. Target data is obtained by pre-reading data from the data cache area. Then, the target data is stored in the write data queue according to the transmission cycle data. Finally, write commands are sequentially obtained, and the corresponding target data is synchronously read from the write data queue and written to the target storage area. In this way, by obtaining all core parameters in advance, the coordination between data migration and write command splitting can be ensured. For example, pre-reading data while splitting write commands can avoid transmission gaps caused by data preparation delays and prevent bubbling problems. Thus, the bubbling problem in the DMA cross-boundary transmission process can be completely eliminated, and write task scheduling and data migration can be efficiently achieved.
[0047] In some embodiments, the step S101 of obtaining parameters of the write task upon receiving a cross-boundary write task includes: Step S1011: Obtain the write address and task length of the write task.
[0048] Step S1012: Calculate the distance boundary length based on the write address; where the distance boundary length is the initial transmission length.
[0049] Step S1013: Calculate the task offset length based on the task length and the offset value of the write address.
[0050] Step S1014: Calculate the second transmission length based on the task offset length and the first transmission length.
[0051] Step S1015: Calculate the initial transmission period and the secondary data shift value based on the initial transmission length.
[0052] Step S1016: Calculate the secondary transmission period based on the secondary transmission length.
[0053] Specifically, the write address refers to the starting storage address of the data to be transferred in the write task, used to locate the starting position of data transmission. The task length refers to the total number of bytes of data to be transferred in the write task.
[0054] Specifically, the distance to the boundary length is calculated by taking 4096 bytes as a memory page boundary unit, subtracting the remainder of the write address divided by 4096 bytes from 4096 bytes, and the difference is the distance to the boundary length, which is also the initial transmission length.
[0055] Specifically, the task offset length is the task length plus the offset value of the write address, and the offset value of the write address is the byte offset corresponding to the lower 2 bits of the write address.
[0056] Specifically, the second transmission length refers to the task offset length minus the first transmission length.
[0057] Specifically, the calculation rules for the first and second transmission cycles are the same. Both are obtained by taking the transmission length in units of the data buffer width. For example, if the data buffer width is 512 bits, or 64 bytes, the lower 6 bits of the corresponding transmission length are extracted. If the lower 6 bits are 0, the transmission cycle = the higher 6 bits of the corresponding transmission length; if the lower 6 bits are not 0, the transmission cycle = the higher 6 bits of the corresponding transmission length + 1.
[0058] Specifically, the calculation rule for the secondary data shift value is to extract the lower 6 bits of the initial transmission length. If the lower 6 bits are 0, the secondary data shift value is set to 512. If the lower 6 bits are not 0, the corresponding decimal value of the lower 6 bits is shifted left by 3 bits to obtain the secondary data shift value, which is used for the alignment of cross-boundary data.
[0059] Understandably, the process involves first obtaining the write address and task length of the write task; then, by calculating the write address against the 4KB boundary, determining the amount of data that can be transferred within the current 4KB memory page, and calculating the initial transmission length; next, combining the task length and the offset value of the write address, correcting the deviation in the data transfer calculation, and obtaining the task offset length; using the difference between the task offset length and the initial transmission length, calculating the remaining secondary transmission length after crossing the boundary; based on the initial transmission length, determining the data pre-reading rhythm and the cross-boundary data alignment standard, and calculating the initial transmission period and the secondary data shift value; finally, according to the secondary transmission length and following the same rules as the initial transmission period, calculating the secondary transmission period, and gradually completing the acquisition of all parameters.
[0060] In the above embodiments, all parameters of the write task were obtained, and the boundaries of the segmented transmission of the write task, the standards for data pre-reading and data alignment were clarified, providing reliable parameter support for subsequent steps of write task splitting, data migration, data pre-reading and alignment, and data writing.
[0061] In some embodiments, step S102, which involves splitting the write task into multiple write commands based on a preset unit, the initial transmission length, and the secondary transmission length, and writing the multiple write commands into a write command queue, includes: Step S1021: The write task is split into an initial send part and a secondary send part.
[0062] Step S1022: Based on the initial transmission length and preset unit, split the initial transmission portion to obtain the first number of initial write commands.
[0063] Step S1023: Based on the secondary transmission length and preset unit, split the secondary transmission part to obtain the second number of secondary write commands.
[0064] Step S1024: Write the first number of initial write commands and the second number of secondary write commands into the write command queue in sequence.
[0065] Specifically, the first transmission segment refers to the data segment to be transmitted in the write task corresponding to the first transmission length, that is, all the data that can be transmitted within the current 4KB memory page, and is the first segment after the write task is divided.
[0066] Specifically, the secondary transmission portion refers to the data segment to be transmitted in the write task corresponding to the secondary transmission length, that is, the remaining data to be transmitted after crossing the 4KB memory page boundary, which is the second segment after the write task is split.
[0067] Specifically, the first write command refers to the MWR command used to instruct the initial transmission of a portion of the data. Each first write command corresponds to a segment of data that matches the preset unit or the remaining length.
[0068] Specifically, a secondary write command refers to a MWR command used to instruct the secondary transmission of part of the data. Each secondary write command corresponds to a segment of data that matches the preset unit or the remaining length.
[0069] Understandably, the write task is first divided into an initial send portion and a secondary send portion, using the initial send length and the secondary send length as boundaries. Then, each of the two send portions is further divided according to a preset unit, resulting in the first number of initial write commands and the second number of secondary write commands. Finally, all write commands are written sequentially into the FIFO structure write command queue in the order of first send first and second send last, thus completing the splitting and caching of the entire write task.
[0070] In the above embodiments, the complete write task is divided into multiple write commands of uniform specifications in an orderly manner, and stored in the write command queue in a predetermined order, which ensures the integrity and orderliness of the write task division and improves the efficiency of write task processing.
[0071] In some embodiments, step S1023, which involves splitting the secondary transmission portion according to the secondary transmission length and a preset unit to obtain a second number of secondary write commands, includes: Step a1: Obtain the remaining length; where the initial value of the remaining length is the length of the second transmission.
[0072] Step a2: Determine whether the remaining length is greater than the preset unit.
[0073] Step a3: If the remaining length is greater than the preset unit, split the secondary transmission part into a single secondary write command according to the preset unit.
[0074] Step a4: Update the remaining length according to the preset unit, and return to the step of determining whether the remaining length is greater than the preset unit.
[0075] Step a5: If the remaining length is not greater than the preset unit, the current remaining secondary transmission portion is taken as a single secondary write command to obtain the second number of secondary write commands.
[0076] Specifically, the remaining length refers to the length of data to be transmitted in the secondary transmission portion that has not yet been split. The initial value is set to the secondary transmission length, and it gradually decreases as the splitting operation progresses.
[0077] Specifically, the first quantity and the second quantity are determined by the ratio of the transmission length to the preset unit and the remainder.
[0078] Understandably, the process involves first obtaining the remaining length, initially set to the second transmission length; then determining if the remaining length exceeds a preset unit. If it does, the remaining length is split into individual second-stage write commands according to the preset unit, the remaining length is updated, and the determination and splitting steps are repeated. If the remaining length is not greater than the preset unit, the remaining second transmission portion is split into individual second-stage write commands. Finally, all split second-stage write commands are aggregated to obtain the second number of second-stage write commands. The method for obtaining the first number of initial write commands is the same as that for obtaining the second number of second-stage write commands.
[0079] In the above embodiments, the precise splitting of the secondary sending part is completed through a process of cyclic judgment and step-by-step splitting, which is consistent with the splitting logic of the first write command, ensuring the continuity of the entire write task splitting process and providing support for writing the secondary write command into the write command queue.
[0080] In some embodiments, step S104, which involves pre-reading data from the data cache based on the data migration path, migration count, and secondary data shift value to obtain the target data to be written to the target storage area, includes: Step S1041: Based on the data migration path and migration count, perform a corresponding number of data pre-reads on the data buffer to obtain multiple pre-read data blocks with a target unit bit width; Step S1042: Based on the storage address of the pre-read data block in the data buffer, concatenate the valid data in multiple pre-read data blocks to obtain a valid data block; Step S1043: Based on the secondary data shift value, perform shift alignment processing on the valid data block; Step S1044: Based on the target unit bit width, the effective data block after shift alignment is divided into multiple target data blocks; wherein, the target data is composed of multiple target data blocks.
[0081] Specifically, the target unit bit width refers to the standard for data prefetching and splicing, which is consistent with the storage bit width of the data buffer area and can be set to 512 bits, or 64 bytes.
[0082] Specifically, a prefetch data block refers to a data packet that conforms to the target unit bit width and is read from the data buffer in each prefetch operation. It is the basic unit of data prefetching.
[0083] Specifically, a target data block refers to a data unit that can be directly used for subsequent write operations, formed by dividing a shift-aligned valid data block according to the target unit bit width. Target data refers to a complete data set formed by the orderly combination of multiple target data blocks, corresponding to all valid data to be transmitted in the write task.
[0084] Understandably, the process involves first pre-reading data from the data cache in batches according to the predetermined data migration path and number of migrations, obtaining multiple pre-read data blocks with a target unit bit width; then, based on the storage address of the pre-read data blocks, invalid data is removed and valid data is concatenated to form valid data blocks; next, the valid data blocks are shifted and aligned using a secondary data shift value to correct cross-boundary data misalignment; finally, the valid data blocks are divided according to the target unit bit width to obtain multiple target data blocks, and these target data blocks are combined in an orderly manner to form the target data.
[0085] In the above embodiments, by effectively extracting, shifting, aligning, and dividing data, the problem of data misalignment in cross-boundary transmission is solved, invalid data is eliminated, the standardization of target data is ensured, transmission bubbles are avoided, and the efficiency of cross-boundary write task processing is improved.
[0086] In some embodiments, step S105, storing the target data to the write data queue according to the first transmission period and the second transmission period, includes: Step S1051: Based on the first transmission period and the second transmission period, the multiple target data blocks are divided into first transmission data blocks and second transmission data blocks; Step S1052: The first data block and the second data block are sequentially stored in the write data queue.
[0087] Specifically, the first data block to be sent refers to the set of target data blocks that correspond to the first part of the write task, distinguished from multiple target data blocks. Its number is equal to the first sending period and the data to be transmitted corresponding to the first sending length. It is the first segment of the target data.
[0088] Specifically, the secondary transmission data block refers to the set of target data blocks corresponding to the secondary transmission part of the write task. Its quantity is equal to the secondary transmission cycle. The data to be transmitted, corresponding to the secondary transmission length, is the second segment of the target data.
[0089] Understandably, based on the first and second transmission cycles, the first transmission data blocks corresponding to the first transmission part and the second transmission data blocks corresponding to the second transmission part are distinguished from multiple target data blocks. Then, in the order of the first transmission data blocks first and the second transmission data blocks last, the two types of data blocks are stored in the write data queue of the FIFO (First In First Out) structure to complete the target data caching operation.
[0090] In the above embodiments, by dividing and storing data blocks in an orderly manner, the storage order of data blocks in the write data queue is ensured to be highly matched with the execution order of write commands in the write command queue, thus providing a guarantee for subsequent response to write commands.
[0091] In some embodiments, step S106, which involves sequentially obtaining write commands from the write command queue, reading target data from the write data queue in response to the write command, and writing the target data to the target storage area, includes: Step S1061: For any write command obtained from the write command queue, in response to the write command, read the target data of the corresponding length from the write data queue according to the preset unit. Step S1062: Write the target data to the target storage area.
[0092] Understandably, the write commands are first retrieved one by one from the write command queue according to the FIFO rule; for each retrieved write command, a response operation is immediately triggered, and the target data of the corresponding length is read from the write data queue according to the length of the write command instruction; after reading, the target data is accurately written to the preset target storage area, and the process is repeated until all write commands are executed and all target data is written.
[0093] In the above embodiments, by obtaining write commands one by one, responding to commands to accurately read target data and write it to the target storage area, it is ensured that the write commands correspond one-to-one with the target data and that the transmission is without misalignment or omission, thus providing a guarantee for the smooth and complete completion of the entire cross-boundary write task.
[0094] For example, this application also provides a method for processing write tasks that do not cross boundaries. Upon receiving a write task that does not cross boundaries, the following steps are taken: First, the parameters of the write task are obtained, including only the cache index, initial transmission length, and initial transmission period. Second, the write task is split into multiple initial write commands based on a preset unit and the initial transmission length, and these commands are sequentially written into a write command queue. Third, the data migration path and migration count are determined based on the cache index, with the processing steps being the same as step S103. Fourth, data is pre-read from the data cache based on the data migration path and migration count to obtain the target data to be written to the target storage area. Fifth, the target data is stored in the write data queue based on the initial transmission period. Sixth, write commands are sequentially obtained from the write command queue, and in response to the write commands, the target data is read from the write data queue and written to the target storage area.
[0095] See also Figure 3 , Figure 3 A schematic diagram of the overall structure of the write task sending device provided in this application. Figure 3 In this process, the DMA task serves as the starting point, referring to a DMA write task initiated by the PCIe device that crosses the 4KB memory page boundary. It carries parameters such as write address, task length, cache index, initial send length, secondary send length, send cycle data, and secondary data shift value. The transmission request management unit is responsible for temporarily storing the write task to be sent and releasing it after execution. The information acquisition unit is responsible for parsing the written task and extracting its information, mainly including the write address, task length, and data cache index. The pre-calculation unit determines whether the write task is a cross-boundary task; if so, it splits it into an initial send portion and a secondary send portion. In the second transmission section, the initial transmission cycle, total transmission cycle, and secondary data shift value required by the data scheduling unit for data transfer, as well as the initial transmission length and secondary transmission length required by the transmission management unit, are calculated. The transmission management unit is responsible for initiating the data transfer process, splitting the write task into multiple MWR commands according to the MPS and writing them to the MWR command queue. The data scheduling unit is responsible for calculating the read data enable and offset address based on the initial transmission cycle and total transmission cycle, calculating the read data base address based on the cache index, and reading data from the data cache. The MWR command queue is used to store the split MWR commands, and the MWR data queue is used to store the transferred data. After the data transfer is initiated, the transmission management unit and the data scheduling unit independently schedule the data transfer, writing the commands and data to the MWR command queue and MWR data queue, respectively.
[0096] See also Figure 4 , Figure 4 A schematic diagram of the pre-calculation unit provided in this application. Figure 4In this process, the module calculates the initial transmission length and the second transmission length based on the write task, task offset length, and distance from the boundary. The distance from the boundary is used as the initial transmission length, and the remaining second transmission length after crossing the page is obtained by subtracting the initial transmission length from the task offset length, thus completing the segmented transmission definition of the write task. At the same time, the module calculates the initial transmission period and the second transmission period based on the initial transmission length and the second transmission length. Using the bit width of the data buffer as a reference, it calculates the transmission period corresponding to the two transmission segments respectively, providing rhythm support for subsequent data pre-reading and storage.
[0097] See also Figure 5 , Figure 5 A schematic diagram illustrating the calculation of the initial transmission length and the secondary transmission length provided in this application. Figure 5 In the first transmission, the length of the first transmission is directly assigned to the distance from the boundary, which is the maximum amount of data that can be transmitted within the current 4KB memory page. The length of the second transmission is assigned to the length of the remaining data to be transmitted after crossing the boundary. In the case that the strange write task does not cross the boundary, the length of the first transmission is directly assigned to the distance from the boundary.
[0098] See also Figure 6 , Figure 6 This is a diagram illustrating the data storage provided in this application. Figure 6 In the diagram, task_len is the task length, which is 38 bytes and represents the total length of the original data to be transmitted for the write task; task_raddr[1:0] is the write address offset value, which is 3 bytes and represents the byte offset corresponding to the lower 2 bits of the write address, corresponding to the byte range from Byte0 to Byte2 in the diagram. It is used to correct the deviation in task length calculation caused by the data buffer DW (Double Word) alignment storage rules; task_len_bias is the task offset length, which is 41 bytes and represents the corrected effective total transmission length.
[0099] See also Figure 7 , Figure 7 A schematic diagram illustrating the calculation of the total transmission cycle provided for this application. Figure 7 First, it checks whether the task crosses the boundary. If the write task does not cross the boundary, the total sending period is directly defined as the first sending period. If the write task crosses the boundary, it checks the lower 6 bits of the first sending length. If the lower 6 bits are 0, the total sending period = the first sending period + the second sending period; if the lower 6 bits are not 0, the total sending period = the first sending period + the second sending period - 1.
[0100] See also Figure 8 , Figure 8 A schematic diagram of the workflow of the sending management unit provided in this application. Figure 8During the process, when the system is idle, it continuously checks the write task queue status. If the queue is not empty, it retrieves a task from the queue and starts the data migration process, while initializing the remaining length to the initial transmission length. It then checks the MWR command queue status; if it's not full, it continues splitting; if it's full, it waits for the queue to become idle. The remaining length is compared to the MPS (Maximum Per Second). If the remaining length is greater than the MPS, an MWR command is split according to the MPS and written to the queue, while the remaining length is updated to the remaining length minus the MPS. If the remaining length is not greater than the MPS, the remaining length is used as the length of a single MWR command and written to the queue. When the remaining length reaches zero, if the task is not completed (i.e., there is a secondary transmission portion), the remaining length is reset to the secondary transmission length, and the MWR commands for the secondary transmission portion are split again. After all MWR commands have been split, the system checks the MWR data queue status to confirm whether all data is ready. If the data queue has completed all data buffering, the current task is released, the process ends, and the system returns to the idle state, waiting for the next task.
[0101] Corresponding to the write task processing method, this application also provides a write task processing apparatus. (See also...) Figure 9 This is a schematic diagram of a write task processing apparatus provided in some embodiments of this application. Figure 9 In the middle, the write task processing device includes: The parameter acquisition module 901 is used to acquire the parameters of the write task when a cross-boundary write task is received; the parameters include the cache index, the initial transmission length, the secondary transmission length, the transmission cycle data, and the secondary data shift value, and the transmission cycle data includes the initial transmission cycle and the secondary transmission cycle; The command queue construction module 902 is used to split the write task into multiple write commands based on the preset unit, the first send length and the second send length, and write the multiple write commands into the write command queue. The migration parameter determination module 903 is used to determine the data migration path and migration number based on the cache index; The target data determination module 904 is used to pre-read data from the data cache based on the data moving path, the number of moving times and the secondary data shift value to obtain the target data to be written to the target storage area; The data queue construction module 905 is used to store target data into the write data queue according to the first sending cycle and the second sending cycle; The write task processing module 906 is used to obtain write commands from the write command queue in sequence, read target data from the write data queue in response to the write command, and write the target data to the target storage area.
[0102] In some embodiments, the parameter acquisition module 901 includes: The acquisition unit is used to obtain the write address and task length of the write task; The distance boundary length calculation unit is used to calculate the distance boundary length based on the write address; where the distance boundary length is the initial transmission length. The task offset length calculation unit is used to calculate the task offset length based on the task length and the offset value of the write address; The secondary transmission length calculation unit is used to calculate the secondary transmission length based on the task offset length and the initial transmission length. The initial transmission period calculation unit is used to calculate the initial transmission period and the secondary data shift value based on the initial transmission length. The secondary transmission period calculation unit is used to calculate the secondary transmission period based on the secondary transmission length.
[0103] In some embodiments, the command queue construction module 902 includes: The write task splitting unit is used to split the write task into an initial transmission part and a secondary transmission part; The first write command determination unit is used to split the first transmission part according to the first transmission length and the preset unit to obtain the first number of first write commands; The secondary write command determination unit is used to split the secondary transmission part according to the secondary transmission length and the preset unit to obtain the second number of secondary write commands; The write command queue building unit is used to write the first number of initial write commands and the second number of secondary write commands into the write command queue in sequence.
[0104] In some embodiments, the secondary write command determination unit includes: The sub-unit is used to obtain the remaining length; the initial value of the remaining length is the length of the second transmission. The judgment sub-unit is used to determine whether the remaining length is greater than a preset unit; The secondary write command splitting subunit is used to split the secondary transmission part into individual secondary write commands according to the preset unit when the remaining length is greater than the preset unit. The loop sub-unit is used to update the remaining length according to the preset unit and return to the step of determining whether the remaining length is greater than the preset unit; The secondary write command determination sub-unit is used to take the current remaining secondary transmission portion as a single secondary write command when the remaining length is not greater than a preset unit, so as to obtain a second number of secondary write commands.
[0105] In some embodiments, the target data determination module 904 includes: The pre-read data block determination unit is used to perform a corresponding number of data pre-reads on the data buffer based on the data moving path and the number of moving times, so as to obtain multiple pre-read data blocks with a target unit bit width; The valid data block determination unit is used to concatenate the valid data in multiple pre-read data blocks according to the storage address of the pre-read data block in the data buffer to obtain a valid data block; The alignment processing unit is used to perform shift alignment processing on the valid data block according to the secondary data shift value; The target data block determination unit is used to divide the shifted and aligned valid data block into multiple target data blocks according to the target unit bit width; wherein the target data is composed of multiple target data blocks.
[0106] In some embodiments, the data queue construction module 905 includes: The target data block partitioning unit is used to divide multiple target data blocks into first-transmission data blocks and second-transmission data blocks according to the first transmission period and the second transmission period; The data queue construction unit is used to sequentially store the first and second data blocks sent into the write data queue.
[0107] In some embodiments, the write task processing module 906 includes: The command data correspondence unit is used to read target data of the corresponding length from the write data queue in response to any write command obtained from the write command queue, according to a preset unit. The write task processing unit is used to write target data to the target storage area.
[0108] For a description of the features in the embodiment corresponding to the task processing device, please refer to the relevant description of the embodiment corresponding to the sample data processing method, which will not be repeated here.
[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0110] See also Figure 10 Embodiments of this application also provide an electronic device, including a memory 10 and a processor 20, wherein the memory 10 stores a computer program and the processor 20 is configured to run the computer program to perform the steps in any of the above-described write task processing method embodiments.
[0111] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described write task processing method embodiments when running.
[0112] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0113] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described write task processing method embodiments.
[0114] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described write task processing method embodiments.
[0115] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0116] The foregoing has provided a detailed description of a write task processing method, apparatus, device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.
Claims
1. A write task processing method, characterized in that, The method includes: Upon receiving a cross-boundary write task, obtain the parameters of the write task; wherein, the parameters include cache index, initial send length, secondary send length, send cycle data, and secondary data shift value, and the send cycle data includes the initial send cycle and the secondary send cycle; Based on the preset unit, the initial transmission length, and the secondary transmission length, the write task is split into multiple write commands, and the multiple write commands are written into the write command queue. Based on the cache index, determine the data migration path and the number of migrations; Based on the data migration path, migration count, and secondary data shift value, data is pre-read from the data cache to obtain the target data to be written to the target storage area; Based on the first transmission cycle and the second transmission cycle, the target data is stored in the write data queue; The write command is obtained sequentially from the write command queue, and in response to the write command, the target data is read from the write data queue and written to the target storage area.
2. The write task processing method according to claim 1, characterized in that, The step of obtaining the parameters of the write task upon receiving a cross-boundary write task includes: Obtain the write address and task length of the write task; Calculate the distance boundary length based on the write address; wherein, the distance boundary length is the initial transmission length; Calculate the task offset length based on the task length and the offset value of the write address; Calculate the second transmission length based on the task offset length and the initial transmission length; Based on the initial transmission length, calculate the initial transmission period and the secondary data shift value; Calculate the secondary transmission period based on the secondary transmission length.
3. The write task processing method according to claim 1, characterized in that, The step of splitting the write task into multiple write commands based on a preset unit, the initial transmission length, and the secondary transmission length, and writing the multiple write commands into a write command queue, includes: The write task is divided into an initial transmission part and a secondary transmission part; Based on the initial transmission length and preset unit, the initial transmission portion is split to obtain a first number of initial write commands; Based on the second transmission length and the preset unit, the second transmission part is split to obtain a second number of second write commands; Write the first number of initial write commands and the second number of secondary write commands into the write command queue in sequence.
4. The write task processing method according to claim 3, characterized in that, The step of splitting the secondary transmission portion according to the secondary transmission length and a preset unit to obtain a second number of secondary write commands includes: Obtain the remaining length; wherein the initial value of the remaining length is the second transmission length; Determine whether the remaining length is greater than a preset unit; If the remaining length is greater than a preset unit, the secondary transmission portion is split into a single secondary write command according to the preset unit. Update the remaining length according to the preset unit, and return to the step of determining whether the remaining length is greater than the preset unit; If the remaining length is not greater than a preset unit, the current remaining secondary transmission portion is treated as a single secondary write command to obtain a second number of secondary write commands.
5. The write task processing method according to claim 1, characterized in that, The step of pre-reading data from the data cache based on the data migration path, migration count, and secondary data shift value to obtain the target data to be written to the target storage area includes: Based on the data migration path and migration count, the data buffer is pre-read a corresponding number of times to obtain multiple pre-read data blocks with a target unit bit width; Based on the storage address of the pre-read data block in the data cache, the valid data in multiple pre-read data blocks are concatenated to obtain a valid data block. Based on the secondary data shift value, the effective data block is shifted and aligned. Based on the target unit bit width, the shifted and aligned valid data block is divided into multiple target data blocks. The target data is composed of the plurality of target data blocks.
6. The write task processing method according to claim 5, characterized in that, The step of storing the target data into the write data queue according to the first transmission period and the second transmission period includes: Based on the first transmission period and the second transmission period, the plurality of target data blocks are divided into first transmission data blocks and second transmission data blocks; The first and second data blocks are sequentially stored in the write data queue.
7. The write task processing method according to claim 1, characterized in that, The step of sequentially obtaining the write command from the write command queue, reading the target data from the write data queue in response to the write command, and writing the target data to the target storage area includes: For any write command obtained from the write command queue, in response to the write command, target data of the corresponding length is read from the write data queue according to the preset unit; Write the target data to the target storage area.
8. A write task processing device, characterized in that, The device includes: The parameter acquisition module is used to acquire the parameters of the write task when a cross-boundary write task is received; wherein, the parameters include cache index, initial send length, secondary send length, send cycle data and secondary data shift value, and the send cycle data includes the initial send cycle and the secondary send cycle; The command queue construction module is used to split the write task into multiple write commands according to the preset unit, the first send length and the second send length, and write the multiple write commands into the write command queue. The migration parameter determination module is used to determine the data migration path and migration number based on the cache index; The target data determination module is used to pre-read data from the data cache area based on the data moving path, the number of moving times and the secondary data shift value to obtain the target data to be written to the target storage area; A data queue construction module is used to store the target data into a write data queue according to the first sending cycle and the second sending cycle; The write task processing module is used to sequentially obtain the write command from the write command queue, read the target data from the write data queue in response to the write command, and write the target data to the target storage area.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the write task processing method as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the write task processing method as described in any one of claims 1 to 7.