Data processing method and device, computer device and product

Abstract

The application discloses a data processing method and device, computer equipment and product, and relates to the technical field of computers, and comprises the following steps: when an input / output task is received, the length of protection information required by the input / output task is acquired; a target virtual page is selected from a virtual address space based on the length of the protection information, and at least one target physical page is selected from a physical address space based on the length of the protection information, wherein the size of the target virtual page matches the length of the protection information; a mapping relationship between the target virtual page and the target physical page is established; and the read / write operation of the protection information is executed based on the mapping relationship in response to the input / output task. The technical problem of improving the utilization rate of data cache resources of a storage array controller is solved, and the technical effect of improving the utilization rate of data cache resources of the storage array controller is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a data processing method and device, computer equipment and product. BACKGROUND

[0002] As a core component in a server system, a storage array controller is used to manage a redundant disk array composed of a plurality of disks connected by a downlink port, and to accelerate the calculation of redundancy in the chip. When the host writes data to the disk array, the storage array controller writes the calculated redundancy check data to a specific location of the disk array; when some disks in the disk array are damaged, the storage array controller reads the undamaged data and the redundancy check data, and then calculates the damaged data and returns it to the host.

[0003] In order to prevent data from being damaged due to errors in the transmission and processing of commands and data, server systems with high reliability requirements often need to add end-to-end data protection, such as adding a check information (Protection Information, PI) outside the data. However, since the data of the disk is in blocks, each data block is a power of 2 (usually 4KB), and discrete data pages are usually used in the host software to store data blocks, and each data page is used to store a block of data. However, the PI information is short in length, and the minimum only needs 8B, that is, each 4KB data block needs to be attached with 1 8B PI information, so the PI information cannot be merged into the data page, and due to the short length, it is also impossible to manage it using a separate data page. SUMMARY

[0004] The present application provides a data processing method and device, computer equipment and product to at least solve the problem of low utilization of data cache resources of the storage array controller in the related art.

[0005] The present application provides a data processing method, comprising:

[0006] When an input / output task is received, the length of the protection information required by the input / output task is obtained;

[0007] Based on the length of the protection information, a target virtual page is selected from a virtual address space, and at least one target physical page is selected from a physical address space based on the length of the protection information, wherein the size of the target virtual page matches the length of the protection information;

[0008] A mapping relationship between the target virtual page and the target physical page is established;

[0009] Based on the mapping relationship, the read / write operation of the protection information is responded to the input / output task.

[0010] The data processing method provided by the embodiment of the present application can dynamically allocate a target virtual page and a target physical page with a size matching the actual length of the protection information. The dynamic allocation mechanism ensures that the read and write operations on the protection information can be quickly performed by establishing an efficient mapping relationship, thereby effectively avoiding the significant space waste caused by allocating a complete data page for protection information with a short length. Thus, the data cache resource utilization rate of the storage array controller is greatly improved, and the memory fragmentation and resource overhead are significantly reduced, ensuring that the system realizes end-to-end reliable protection in the whole data transmission process, further optimizing the overall performance and reducing the system maintenance cost.

[0011] In an optional implementation, the length of the protection information required by the input / output task is obtained, including:

[0012] The total task size, the number of subtasks, and the subtask size of each subtask of the input / output task are obtained.

[0013] The length of the protection information required by the input / output task is determined based on the total task size, the number of subtasks, and the subtask size.

[0014] In an optional implementation, a target virtual page is selected from the virtual address space based on the length of the protection information, including:

[0015] A virtual page size corresponding to each virtual sub-space in the virtual address space is obtained, wherein each virtual sub-space is evenly divided into a plurality of virtual pages.

[0016] The length of the protection information is compared with the virtual page size corresponding to each virtual sub-space to obtain a comparison list.

[0017] A target virtual page is selected from the virtual sub-space based on the comparison list.

[0018] In an optional implementation, a target virtual page is selected from the virtual sub-space based on the comparison list, including:

[0019] If the first virtual page size corresponding to a first virtual sub-space in the comparison list is the same as the length of the protection information, the number of first free virtual pages in the first virtual sub-space is obtained.

[0020] If the number of first free virtual pages is greater than 0, the first virtual sub-space is taken as a target sub-space, and a first free virtual page is randomly selected from the first virtual sub-space as a target virtual page.

[0021] In an optional implementation, a target virtual page is selected from the virtual sub-space based on the comparison list, including:

[0022] If the number of the first free virtual pages is equal to 0, a second virtual sub-space with a virtual page size greater than a length of the protection information is selected as a candidate sub-space;

[0023] The number of the candidate sub-spaces is obtained;

[0024] If the number of the candidate sub-spaces is 1, the candidate sub-space is selected as a target sub-space, and a second free virtual page in the candidate sub-space is randomly selected as a target virtual page.

[0025] In an optional embodiment, the target virtual page is selected from the virtual sub-space based on the comparison list, and the method further comprises:

[0026] If the number of the candidate sub-spaces is greater than 1, a difference between a second virtual page size corresponding to the candidate sub-space and the length of the protection information is calculated;

[0027] The candidate sub-space with the smallest difference is selected as the target sub-space, and a third free virtual page in the candidate sub-space with the smallest difference is randomly selected as the target virtual page.

[0028] In an optional embodiment, the target virtual page is selected from the virtual sub-space based on the comparison list, and the method further comprises:

[0029] If the number of the candidate sub-spaces is greater than 1 or equal to 1, the number of the second free virtual pages in the candidate sub-space is obtained;

[0030] If the number of the second free virtual pages is greater than a threshold value, part of the second free virtual pages in the candidate sub-space is divided into a plurality of first free virtual pages and allocated to the first virtual sub-space.

[0031] In an optional embodiment, the target virtual page is selected from the virtual sub-space based on the comparison list, and the method further comprises:

[0032] If the number of the candidate sub-spaces is 0, an error is returned or selection waiting information corresponding to the virtual sub-space information that can be recycled based on other input and output tasks is generated.

[0033] In an optional embodiment, at least one target physical page is selected from a physical address space based on a length of the protection information, and the method comprises:

[0034] The physical page size of each target physical page in the physical address space is obtained;

[0035] At least one target physical page is selected from the physical address space based on the length of the protection information and the physical page size.

[0036] In an optional embodiment, a mapping relationship between the target virtual page and the target physical page is established, and the method comprises:

[0037] obtaining a physical page address of the target physical page and a virtual page address of the target virtual page;

[0038] creating a mapping descriptor based on the physical page address and the virtual page address;

[0039] establishing a mapping relationship between the target virtual page and the target physical page based on the mapping descriptor.

[0040] In an optional implementation, the read-write operation of the protection information is performed in response to the input-output task based on the mapping relationship, including:

[0041] obtaining a task descriptor of each subtask in the input-output task;

[0042] analyzing the read-write type of each subtask based on the task descriptor;

[0043] determining the read-write position of the protection information based on the read-write type;

[0044] performing the read-write operation of the protection information according to the read-write position and the mapping relationship.

[0045] In an optional implementation, the method further includes:

[0046] obtaining a task response result of the input-output task;

[0047] determining a completion state of the read-write operation based on the task response result;

[0048] recycling the target virtual page and the target physical page corresponding to the input-output task according to the completion state.

[0049] The application further provides a data processing apparatus, including:

[0050] an obtaining module, configured to obtain a protection information length required by an input-output task when the input-output task is received;

[0051] a selecting module, configured to select a target virtual page from a virtual address space based on the protection information length, and select at least one target physical page from a physical address space based on the protection information length, wherein the size of the target virtual page matches the protection information length;

[0052] a mapping module, configured to establish a mapping relationship between the target virtual page and the target physical page;

[0053] a processing module, configured to perform a read-write operation of the protection information in response to the input-output task based on the mapping relationship.

[0054] The data processing device provided by the embodiment of the present application can dynamically allocate target virtual pages and target physical pages of matched sizes according to the actual length of the protection information, and the dynamic allocation mechanism can ensure that the read and write operations on the protection information can be quickly performed by establishing an efficient mapping relationship, thereby effectively avoiding the significant space waste caused by allocating complete data pages for protection information of a short length. Therefore, the data cache resource utilization rate of the storage array controller is greatly improved, the memory fragmentation and resource overhead are obviously reduced, the end-to-end reliable protection of the system in the whole data transmission process is ensured, the overall performance is further optimized, and the system maintenance cost is reduced.

[0055] The present application also provides a computer device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the data processing methods.

[0056] The present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any of the data processing methods.

[0057] The present application also provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of any of the data processing methods. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0059] Figure 1 The application scenario diagram of the data processing method provided by the embodiment of the present application is shown in the figure;

[0060] Figure 2 The address converter schematic diagram of the data processing method application scenario diagram provided by the embodiment of the present application is shown in the figure;

[0061] Figure 3 The flowchart of the data processing method provided by the embodiment of the present application is shown in the figure;

[0062] Figure 4 The virtual subspace division example diagram of the data processing method provided by the embodiment of the present application is shown in the figure;

[0063] Figure 5 The virtual address space allocation schematic diagram of the data processing method provided by the embodiment of the present application is shown in the figure;

[0064] Figure 6A mapping relationship diagram of the data processing method provided by the embodiment of the present application is shown in FIG. 1.

[0065] Figure 7 A flowchart of another data processing method provided by the embodiment of the present application is shown in FIG. 2.

[0066] Figure 8 A structure block diagram of the data processing device provided by the embodiment of the present application is shown in FIG. 3.

[0067] Figure 9 A structure diagram of the computer device provided by the embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0069] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0070] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0071] Please refer to Figure 1 , Figure 1is a schematic diagram of an application environment provided by an embodiment of the present application, which includes a host and a storage array controller, the storage array controller including a DMA engine, an internal interconnection bus, an address translator, a data cache and a data engine. The storage array controller is connected with the Direct Memory Access (DMA) engine in the storage array controller through a peripheral component interconnection express (PCIe) interface. When the DMA engine moves data from the host to the storage array controller and writes the data into the data cache, the write operation first reaches the address translator through the internal interconnection bus, and then the address translator converts the write address and writes data into the converted address. For the read operation of the data engine, the data engine sends a read command to the address translator, which sends the converted command to the data cache, and then sends the data returned by the data cache to the data engine.

[0072] Please participate Figure 2 The address space inside the storage array controller will be divided into two parts. The real address for accessing the data cache is called the physical address space, which is the address space converted by the address translator. The other part is the original address space for accessing the data cache, which is called the virtual address space. The address of this space cannot directly act on the data cache and must be converted by the address translator.

[0073] The address translator is divided into virtual address allocation and recovery, cache allocation and recovery, mapping table management and mapping table.

[0074] Virtual address allocation and recovery is used to manage the entire virtual address space. It is responsible for recording whether the virtual data page in each sub-space is used or not, and allocating appropriate virtual pages for input / output tasks (i.e. IO tasks). When the IO task ends, it is responsible for recycling the corresponding virtual pages for re-allocation.

[0075] Cache allocation and recovery is used to manage the entire physical address space of the data cache. It is responsible for recording whether each physical page is used or not, and allocating one or several physical pages for IO tasks. When the IO task ends, it is responsible for recycling the corresponding physical pages for re-allocation.

[0076] Mapping table management is used to map the virtual page allocation result and the physical page allocation result, and record the result in the mapping table.

[0077] The mapping table is used to record the mapping relationship between the virtual page and the physical page. When the address translator receives a data read / write request, the table is used to convert the request address.

[0078] Based on this, according to an embodiment of the present application, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0079] The present disclosure provides a data processing method, which can be applied to a transaction level processing module in a data processing scenario in Figure 1 , or applied to a computer device running a data processing system. The computer device can include but is not limited to a tablet, a notebook, a desktop computer, a server, a controller, etc. In combination with reference to Figure 3 , a flowchart of the data processing method provided by some embodiments of the present disclosure is shown. Figure 3 In , the data processing method can include the following steps:

[0080] Step 310, when receiving an input / output task, the length of the protection information required by the input / output task is obtained.

[0081] In the above, the input / output task includes the total task size, the number of subtasks, and the subtask size of each subtask.

[0082] In some optional embodiments, when obtaining the length of the protection information required by the input / output task, the total task size of the input / output task, the number of subtasks, and the subtask size of each subtask can be obtained first; based on the total task size, the number of subtasks and the subtask size, the length of the protection information required by the input / output task is determined.

[0083] Specifically, the virtual address space will be allocated to the input / output task in blocks, each input / output task is allocated 1 block, and the size of the block needs to be greater than or equal to the protection information length of the input / output task. For example, an input / output task (i.e. IO task) is 16KB, the subtask size of the subtask is 4KB, and the protection information length (i.e. PI information length) of each subtask is 8B, then the protection information length required by the input / output task is 16KB / 4KB*8B=32B.

[0084] The data processing method provided by the embodiment of the application dynamically determines the protection information length based on the total task size, the subtask quantity and the subtask size of the input and output task, so as to match the task demand, avoid insufficient or redundant allocation of protection information, and thus optimize the utilization rate of the virtual address space. By ensuring that the block size allocated to each input and output task is always greater than or equal to the required protection information length, the memory fragmentation is reduced, and the data processing efficiency is improved. In addition, in a high-speed data transmission scenario, such as a server or controller environment, this mechanism can effectively prevent data damage or security vulnerabilities and enhance system reliability. For example, in an IO task, the protection information length of 32B accurately covers all subtasks, simplifies the storage and management of PI information, and reduces processing delay.

[0085] In step 320, a target virtual page is selected from the virtual address space based on the protection information length, and at least one target physical page is selected from the physical address space based on the protection information length.

[0086] The target virtual page is an idle virtual page in the virtual address space, and the target physical page is an idle physical page in the virtual address space.

[0087] In some optional embodiments, when the target virtual page is selected from the virtual address space based on the protection information length, a virtual page size corresponding to each virtual sub-space in the virtual address space can be obtained first, wherein each virtual sub-space is uniformly divided into a plurality of virtual pages; the protection information length is compared with the virtual page size corresponding to each virtual sub-space to obtain a comparison list; and a target virtual page is selected from the virtual sub-space based on the comparison list.

[0088] Specifically, please refer to Figure 4 The virtual address space is divided into a plurality of virtual sub-spaces according to the size of the page, such as 16B, 32B, 64B, and the like. Each virtual sub-space is equal in size to the entire data cache, and then the virtual sub-space is uniformly divided according to the corresponding page size. For example, a 16B virtual sub-space is uniformly divided into N / 16 blocks (i.e., virtual pages). If the protection information length is 32B, a virtual page needs to be selected from the 32B virtual sub-space. Since each size of virtual sub-space completely covers all cache spaces, a target virtual page can be allocated from the sub-space.

[0089] Further, please refer to Figure 5All virtual subspaces need to occupy a segment of system address. The system address here refers to the addressing space inside the storage array controller. In the entire system address space, a segment of address is needed to place the virtual address space of the data cache, and the hit of this segment of address space will be routed to the address translator by the internal interconnection bus. The remaining address space is the physical address space, which is used to access other engine devices (such as data engine, operation engine) in the system. The virtual address space is further divided into several virtual subspaces, each of which has a size of N. After receiving a task, a target virtual page will be allocated, and according to the allocation of the virtual address space in the system, the system address corresponding to the virtual page is calculated, and the system address is returned to the engine that needs to access it, such as the DMA engine or the operation engine. The latter will initiate read-write access operation on the system address, which is routed to the address translator by the internal interconnection bus for conversion.

[0090] In some optional embodiments, when selecting a target virtual page from a virtual subspace based on the comparison list, if there is a first virtual page corresponding to a first virtual subspace in the comparison list, the size of which is the same as the length of the protection information, the number of first free virtual pages in the first virtual subspace is obtained; if the number of first free virtual pages is greater than 0, the first virtual subspace is taken as the target subspace, and a first free virtual page is randomly selected from the first virtual subspace as the target virtual page.

[0091] Specifically, the comparison list includes the size identifier and the free page state information of each virtual subspace; when obtaining the number of first free virtual pages, the current free state is queried in real time by accessing the bitmap data structure of the subspace manager; if the number is greater than 0, the random selection process uses a hardware random number generator to perform uniform sampling based on the free page index list to avoid allocation bias. In addition, if the number of first free virtual pages is equal to 0, the system skips this subspace and continues to search for other size-matched virtual subspaces in the comparison list for selection; or if all matching subspaces have no free pages, an error interrupt mechanism is triggered to notify the upper layer processing module.

[0092] As an example, assuming the protection information length is 16 bytes, the comparison list record has multiple size identifiers of virtual subspaces, and the first virtual subspace with a size of 16 bytes is marked as subspace A; the system accesses the bitmap data structure of the subspace manager to query in real time that subspace A currently has 3 free virtual pages; because the number is greater than 0, the system uses the hardware random number generator to perform uniform sampling based on the free page index list (such as indexes [0, 1, 2]) to randomly select the free page corresponding to index 1 as the target virtual page, and complete the allocation. If subspace A has no free page at this time, the system skips this subspace and searches for other subspaces with a size of 16 bytes (such as subspace C); if all matching subspaces have no free page, an error interrupt mechanism is triggered, and a “resource exhausted” error code is sent to the upper layer processing module.

[0093] In some optional embodiments, when selecting a target virtual page from a virtual subspace based on the comparison list, if the number of first free virtual pages is equal to 0, a second virtual subspace with a virtual page size greater than the protection information length is taken as a candidate subspace; the number of candidate subspaces is obtained; if the number of candidate subspaces is 1, the candidate subspace is taken as a target subspace, and a second free virtual page is randomly selected from the candidate subspace as a target virtual page.

[0094] Specifically, when obtaining the number of candidate subspaces, all virtual subspaces with a virtual page size greater than the protection information length are filtered out by traversing the size identifiers in the comparison list, and the number of the virtual subspaces is counted in real time by accessing the bitmap data structure of the subspace manager; when a second free virtual page is randomly selected as a target virtual page, the hardware random number generator is used to perform uniform sampling based on the free page index list of the candidate subspace to avoid allocation bias.

[0095] As an example, assuming the protection information length is 32 bytes, the system is configured with multiple virtual subspaces, including subspace A with a size of 126 bytes, subspace B with a size of 256 bytes, and subspace C with a size of 16 bytes; when it is detected that the number of first free virtual pages is 0, the size identifiers of the comparison list are traversed, candidate subspaces (such as subspace A and subspace B) with a virtual page size greater than 64 bytes are filtered out, and the number of the candidate subspaces is counted as 2; if the number is 1 (such as only subspace B meets the condition), the candidate subspace is taken as a target subspace, its free page index list is accessed, the hardware random number generator is used to perform uniform sampling from list indexes 0 to N-1, and a second free virtual page with an index of 3 is selected as a target virtual page, to ensure that the allocation has no bias.

[0096] Further, in real-time statistics of the number of candidate subspaces, the bitmap data structure of the sub-space manager quickly queries the free state through the bit mask operation, and the bitmap update frequency is synchronized with the memory allocation event; the random sampling process combines the system clock seed to initialize the random number generator, so as to improve the sampling uniformity and performance efficiency.

[0097] In addition, when the number of candidate subspaces is greater than 1, the expandable embodiment includes secondary screening based on the sub-space priority or the fragmentation rate, but the present example focuses on the single candidate scenario to simplify the description.

[0098] In some optional embodiments, when selecting a target virtual page from the virtual sub-space based on the comparison list, if the number of candidate subspaces is greater than 1, the difference between the second virtual page size corresponding to the candidate subspaces and the length of the protection information is calculated; the candidate sub-space with the smallest difference value is taken as the target sub-space, and a third free virtual page is randomly selected from the candidate sub-space with the smallest difference value as the target virtual page.

[0099] Specifically, when calculating the difference between the second virtual page size corresponding to the candidate subspaces and the length of the protection information, the virtual page size of each candidate sub-space is obtained by traversing the size identifier of the candidate sub-space; then, the difference between each virtual page size and the length of the protection information is calculated; when the candidate sub-space with the smallest difference value is taken as the target sub-space, the candidate sub-space corresponding to the smallest difference value is selected by comparing all the difference values; when a third free virtual page is randomly selected as the target virtual page, a hardware random number generator is used to uniformly sample based on the free page index list of the target sub-space, so as to avoid allocation bias and ensure the fairness and efficiency of virtual page allocation.

[0100] As an example, assuming that the length of the protection information is 32 bytes, the candidate subspaces include a first sub-space and a second sub-space, wherein the second virtual page size corresponding to the first sub-space is 64 bytes, and the second virtual page size corresponding to the second sub-space is 128 bytes. After traversing the size identifier of the candidate sub-space, the difference between each virtual page size and the length of the protection information is calculated: the difference of the first sub-space is 64-32=32 bytes, and the difference of the second sub-space is 128-32=96 bytes. Comparing all the difference values, 32 bytes is the smallest difference value, so the first sub-space is taken as the target sub-space. Then, a hardware random number generator is used to uniformly sample based on the free page index list of the target sub-space, assuming that the index list contains page numbers 1 to 10, and the third free virtual page with page number 3 is selected as the target virtual page after random sampling, so as to ensure the fairness and efficiency of the allocation process.

[0101] In some optional embodiments, when selecting a target virtual page from the virtual subspaces based on the comparison list, if the number of candidate subspaces is greater than 1 or equal to 1, the number of second free virtual pages in the candidate subspaces is obtained; if the number of second free virtual pages is greater than a threshold value, part of the second free virtual pages in the candidate subspaces are divided into a plurality of first free virtual pages and allocated to the first virtual subspace.

[0102] Specifically, assuming that the candidate subspaces include a second virtual subspace, the corresponding second virtual page size is 128 bytes, and the number of second free virtual pages in the free page index list is 20. If the preset threshold value is 15, since 20 is greater than 15, part of the second free virtual pages in the subspace, such as 5 128-byte pages, are divided into a plurality of first free virtual pages; each 128-byte page can be divided into two 64-byte first free virtual pages, thereby generating 10 new 64-byte free pages, and the pages are allocated to the first virtual subspace. In addition, the division process is dynamically adjusted based on the page size identifier to ensure efficient use of memory resources, avoid fragmentation, and support subsequent target page selection based on the minimum difference strategy. After the division is completed, the system updates the free page index list, and uniformly samples the newly generated first free virtual pages through a hardware random number generator to maintain allocation fairness, for example, when the index list contains page numbers 1 to 20, the page number 7 is randomly selected as the target virtual page, thereby optimizing the overall performance.

[0103] In some optional embodiments, when selecting a target virtual page from the virtual subspaces based on the comparison list, if the number of candidate subspaces obtained is 0, an error is returned or selection waiting information corresponding to the recyclable virtual subspace information of other input and output tasks is generated.

[0104] Specifically, when it is detected that the number of candidate subspaces is 0, a predefined error code, such as ERR_NO_CANDIDATE_SPACE, can be returned, or a recycling process can be triggered. The recycling process identifies virtual subspaces in an idle or releasable state by scanning recyclable resource information of other input and output tasks in the system. For example, assuming that a network transmission task has been completed, the corresponding third virtual subspace size is 256 bytes, and the free page index list contains 5 free pages, the pages can be recycled and the related list is updated. Subsequently, selection waiting information is generated, such as setting a waiting timer or inserting a task queue, to delay the selection operation until a free page is available. During the recycling process, the recycling strategy is dynamically adjusted based on the page size identifier, and large-size pages are preferentially recycled to reduce fragmentation, for example, 256-byte pages are divided into a plurality of 64-byte pages and allocated to the first virtual subspace. At the same time, the recycling efficiency is monitored through a hardware counter to ensure balanced memory resources and avoid affecting high-priority tasks.

[0105] In some optional embodiments, when selecting the at least one target physical page from the physical address space based on the protection information length, the size of each target physical page in the physical address space can be obtained first; and the at least one target physical page is selected from the physical address space based on the protection information length and the size of the physical page.

[0106] Specifically, the size identifier of each physical page is obtained by accessing the system page table or the hardware configuration register; the protection information length is compared with the size of the physical page, and the physical page with a matching size is preferentially selected to minimize resource waste; if the protection information length is smaller than the size of the available physical page, the smallest available page is selected; if the protection information length is larger than the size of the available physical page, multiple small pages are combined or a large-size page is selected. For example, assuming that the protection information length is 192 bytes, and there are 64-byte and 256-byte physical pages in the physical address space, a 256-byte page is preferentially allocated to avoid fragmentation; or three 64-byte pages are combined to meet the requirement. At the same time, the selection strategy is dynamically adjusted based on the size identifier of the page to ensure the memory access efficiency of high-priority tasks, and the allocation success rate is monitored by a counter to optimize the utilization rate of physical resources.

[0107] Step 330, a mapping relationship between the target virtual page and the target physical page is established.

[0108] As described above, the mapping relationship between the target virtual page and the target physical page is established, and the read and write operations of the protection information are performed based on the mapping relationship.

[0109] In some optional embodiments, when the mapping relationship between the target virtual page and the target physical page is established, the physical page address of the target physical page and the virtual page address of the target virtual page can be obtained first; a mapping descriptor is created based on the physical page address and the virtual page address; and the mapping relationship between the target virtual page and the target physical page is established based on the mapping descriptor.

[0110] Specifically, please refer to Figure 6 , assuming that the minimum protection information length to be supported is 16B. The physical cache is divided into a plurality of physical pages with a size of 16B, i.e., one square in Figure 6 . The virtual address space is also divided into a plurality of virtual subspaces with a granularity of the minimum protection information length. In the example of Figure 6 , the virtual subspaces are also divided into a plurality of units with a size of 16B. Therefore, a virtual subspace with a size of 16B will be divided into 1 virtual page, a virtual subspace with a size of 32B will be divided into 2 virtual pages, and so on.

[0111] The mapping between the virtual page and the physical page is realized by a mapping descriptor: each virtual page can be divided into a plurality of virtual units, each virtual unit is bound to a mapping descriptor, and each mapping descriptor can point to a physical page. As Figure 6As shown, 1 virtual page of 64B contains 4 virtual units, each of which is bound with a mapping descriptor (e.g. a physical page descriptor). By filling in the physical page identifier into the physical page descriptor, the descriptor points to a physical page. Thus, the 64B virtual page maps 4 physical pages of 16B with identifiers 10, 22, 28, 47.

[0112] At step 340, the read / write operation of the protection information is performed based on the mapping relationship in response to the input / output task.

[0113] Specifically, when the read / write operation of the protection information is performed based on the mapping relationship in response to the input / output task, the target virtual unit can be identified by parsing the input / output task, and the corresponding physical page can be located according to the bound mapping descriptor; then, the read / write operation is performed on the physical page, and the secure access of the protection information is ensured through the permission verification mechanism, such as checking the access permission bit or using the encryption module to encrypt / decrypt the data, thereby improving the security and reliability of the system.

[0114] In some optional embodiments, when the read / write operation of the protection information is performed based on the mapping relationship in response to the input / output task, the task descriptors of each subtask in the input / output task can be obtained first; the read / write type of each subtask is parsed based on the task descriptor; the read / write position of the protection information is determined based on the read / write type; and the read / write operation of the protection information is performed according to the read / write position and the mapping relationship.

[0115] Specifically, assuming that the input / output task involves reading the protection information in the 64B virtual page. First, the task descriptor of the subtask is extracted from the task queue, which contains the virtual unit index and the access permission information; the read / write type is parsed to be "read" based on the task descriptor, and the protection information is identified to be located in the physical page identifier (such as one of identifiers 10, 22, 28, 47) bound with the mapping descriptor of the virtual unit. Next, the read / write position is determined to be the physical page offset corresponding to the virtual unit based on the read / write type; according to the mapping relationship (such as pointing to the physical page 10 through the mapping descriptor), the read operation is performed to obtain the 16B protection information data block from the target physical page, and the data integrity is verified to ensure secure access.

[0116] As an example, assuming that there are 256 64B virtual pages in total: when the address converter receives a read / write access, first, the corresponding virtual sub-space is found according to the high bit of the address, then the corresponding virtual page is found according to [13:6] of the address, then the corresponding physical page descriptor is found according to [5:4], and the physical address is found according to the physical page descriptor, and then the cache is accessed with the physical address as the base address and [3:0] as the offset.

[0117] When the storage array controller receives an IO task, it needs to send relevant information to the address translation module for space allocation. The address translation module first calculates the protection information length according to the IO task size, and then applies a virtual page from the virtual subspace greater than and closest to the length. If there is no free page in the subspace of this size, a page is applied from a larger subspace. Then, the address translator applies a number of physical pages according to the protection information length. It needs to be specially pointed out that the application of physical pages is not according to the virtual page size, but according to the actual number of physical pages used. For example, if an IO task needs a protection information length of 16Bx5, it needs to apply a virtual page of 128B at the minimum, but only needs to apply a physical space of 80B (5 physical pages of 16B). Then, the above physical pages are filled into the physical page descriptors in turn. Still taking the above IO task as an example, there are 8 physical page descriptors in the 128B virtual page, and only 5 physical pages are actually applied, so only the identifiers corresponding to the 5 physical pages need to be filled into the first 5 physical page descriptors.

[0118] Then, the address translator needs to return the allocated virtual address to the upper layer logic, which will inform the DMA or data engine module of the address, and the latter will initiate access through the address.

[0119] The data processing method provided by the embodiment of the application can dynamically allocate target virtual pages and target physical pages of a size matching the actual length of the protection information. This dynamic allocation mechanism ensures that the read and write operations on the protection information can be quickly executed by establishing an efficient mapping relationship, thereby effectively avoiding the significant space waste caused by allocating complete data pages for protection information of a shorter length. As a result, the utilization rate of the data cache resources of the storage array controller is greatly improved, and the memory fragmentation and resource overhead are significantly reduced, ensuring that the system realizes end-to-end reliable protection in the whole data transmission process, further optimizing the overall performance and reducing the system maintenance cost.

[0120] Figure 7 A flowchart of another embodiment of the data processing method of the application is shown. As shown in Figure 7 the method includes the following steps:

[0121] Step 710, obtaining a task response result of an input / output task.

[0122] Step 720, determining a completion state of a read / write operation based on the task response result.

[0123] Step 730, recycling target virtual pages and target physical pages corresponding to the input / output task according to the completion state.

[0124] Specifically, when the completion status indicates that the input / output task is successfully completed, the address translator releases the physical page identifiers in the physical page descriptors corresponding to the target virtual page, and returns the physical pages to the physical page pool for reuse; at the same time, the target virtual page is marked as allocatable state to release the virtual address space resources. If the completion status indicates failure, the address translator records error logs, triggers a retry mechanism or notifies the upper layer logic to handle the exception, to ensure that the system resources are not invalidly occupied.

[0125] As an example, assume that an input / output task involves reading protection information from the data cache. If the task is successfully completed, the address translator first clears the physical page identifiers in the physical page descriptors corresponding to the target virtual page, to ensure that the identifiers are no longer referenced; then, the released physical pages are returned to the physical page pool for subsequent task reuse; at the same time, the state of the target virtual page is updated to allocatable, so as to efficiently recycle the virtual address resources and avoid fragmentation. If the task fails, for example, due to hardware timeout or data verification error, the address translator generates detailed error logs, including error type and timestamp; at the same time, according to a preset strategy, a retry mechanism is triggered to automatically attempt to re-execute the task up to three times, or immediately notify the upper layer logic to intervene in handling the exception after the retry fails, to prevent the physical pages or virtual addresses from being invalidly locked.

[0126] The data processing method provided by the embodiment of the present application can effectively manage the recycling process of virtual pages and physical pages, efficiently release resources and reduce fragmentation when the input / output task is successful, and ensure that resources are not invalidly locked through automatic retry and error log mechanism when the task fails, thereby optimizing system resource utilization, improving data processing reliability, and supporting stable operation of large-scale concurrent tasks.

[0127] Figure 8 An embodiment of a data processing apparatus of the present application is shown in a structural schematic diagram. As shown in the figure, the apparatus comprises: Figure 8

[0128] The obtaining module 810 is configured to, when an input / output task is received, obtain the length of protection information required by the input / output task.

[0129] The selecting module 820 is configured to select a target virtual page from the virtual address space based on the length of the protection information, and select at least one target physical page from the physical address space based on the length of the protection information, wherein the size of the target virtual page matches the length of the protection information.

[0130] The mapping module 830 is configured to establish a mapping relationship between the target virtual page and the target physical page.

[0131] The processing module 840 is configured to, based on the mapping relationship, respond to the input / output task to perform read / write operations on the protection information.​

[0132] In an optional implementation, the obtaining module 810 comprises:

[0133] an information obtaining sub-module, configured to obtain a total task size of the input-output task, a number of sub-tasks, and a sub-task size of each sub-task;

[0134] an information determining sub-module, configured to determine a length of protection information required by the input-output task based on the total task size, the number of sub-tasks, and the sub-task size.

[0135] In an optional implementation, the selecting module 820 comprises:

[0136] a size obtaining sub-module, configured to obtain a virtual page size corresponding to each virtual sub-space in the virtual address space, wherein each virtual sub-space is evenly divided into a plurality of virtual pages;

[0137] a size comparing sub-module, configured to compare the length of protection information with the virtual page size corresponding to each virtual sub-space to obtain a comparison list;

[0138] a virtual page selecting sub-module, configured to select a target virtual page from the virtual sub-space based on the comparison list.

[0139] In an optional implementation, the virtual page selecting sub-module comprises:

[0140] a first obtaining unit, configured to, if a first virtual page size corresponding to a first virtual sub-space in the comparison list is the same as the length of protection information, obtain a number of first free virtual pages in the first virtual sub-space;

[0141] a first selecting unit, configured to, if the number of first free virtual pages is greater than 0, take the first virtual sub-space as a target sub-space, and randomly select a first free virtual page in the first virtual sub-space as the target virtual page.

[0142] In an optional implementation, the virtual page selecting sub-module further comprises:

[0143] a second selecting unit, configured to, if the number of first free virtual pages is equal to 0, take a second virtual sub-space with a virtual page size greater than the length of protection information as a candidate sub-space;

[0144] a second obtaining unit, configured to obtain a number of candidate sub-spaces;

[0145] a third selecting unit, configured to, if the number of candidate sub-spaces is 1, take the candidate sub-space as the target sub-space, and randomly select a second free virtual page in the candidate sub-space as the target virtual page.

[0146] In an optional implementation, the virtual page selection submodule further includes:

[0147] a difference calculation unit, configured to calculate a difference between the second virtual page size corresponding to the candidate subspace and the length of the protection information if the number of the candidate subspaces is greater than 1;

[0148] a fourth selection unit, configured to select a candidate subspace with a minimum difference as a target subspace, and randomly select a third free virtual page from the candidate subspace with the minimum difference as a target virtual page.

[0149] In an optional implementation, the virtual page selection submodule further includes:

[0150] a third acquisition unit, configured to acquire the number of the second free virtual pages in the candidate subspace if the number of the candidate subspaces is greater than 1 or equal to 1;

[0151] a fifth unit, configured to divide part of the second free virtual pages in the candidate subspace into a plurality of first free virtual pages and allocate the first free virtual pages to the first virtual subspace if the number of the second free virtual pages is greater than a threshold.

[0152] In an optional implementation, the virtual page selection submodule further includes:

[0153] an information feedback unit, configured to return an error or generate corresponding selection waiting information based on virtual subspace information that can be recycled by other input and output tasks if the number of the candidate subspaces is 0.

[0154] In an optional implementation, the selection module 820 further includes a physical page selection submodule, configured to acquire a physical page size of each target physical page in a physical address space; and select at least one target physical page from the physical address space based on the length of the protection information and the physical page size.

[0155] In an optional implementation, the mapping module 830 includes:

[0156] an address acquisition submodule, configured to acquire a physical page address of the target physical page and a virtual page address of the target virtual page;

[0157] a mapping descriptor submodule, configured to create a mapping descriptor based on the physical page address and the virtual page address;

[0158] a relationship mapping submodule, configured to establish a mapping relationship between the target virtual page and the target physical page based on the mapping descriptor.

[0159] In an optional implementation, the processing module 840 includes:

[0160] a task descriptor submodule, configured to acquire a task descriptor of each subtask in the input and output task;

[0161] The description analysis submodule is configured to analyze read-write types of each subtask based on the task descriptor;

[0162] The position determination submodule is configured to determine read-write positions of the protection information based on the read-write types.

[0163] In an optional implementation, the processing module 840 is further configured to obtain a task response result of the input-output task, determine a completion state of the read-write operation based on the task response result, and recycle the target virtual page and the target physical page corresponding to the input-output task according to the completion state.

[0164] The features of the embodiments of the data processing apparatus can be understood by referring to the features of the embodiments of the data processing method, which will not be described herein.

[0165] The embodiments of the present application also provide a computer device, as shown in the accompanying drawings, which comprises a memory 910 and a processor 920, the memory 910 stores a computer program, and the processor 920 is configured to run the computer program to execute the steps in any of the above-mentioned data processing method embodiments. Figure 9 The computer device further comprises a communication interface 930 for communication between the electronic device and other devices or communication networks.

[0166] The computer device further comprises a communication interface 930 for communication between the electronic device and other devices or communication networks.

[0167] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above-mentioned data processing method embodiments when running.

[0168] In an exemplary embodiment, the above-mentioned computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0169] The embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned data processing method embodiments.

[0170] The embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned data processing method embodiments.

[0171] Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of the applications and are not intended to limit the scope of the applications. Therefore, embodiments or examples described herein are not meant to be limiting, but merely to aid in the understanding of the overall more complete disclosure of the applications. Accordingly, the disclosure of various examples is meant to be illustrative, but not limiting, of the scope of the applications, which is set forth in the following claims.

[0172] The above provides a kind of data processing method, device, computer equipment and product provided by the present application in detail.The principle and implementation of the present application are described in this paper by applying specific examples, the above example is only used to help understanding the method and its core idea of the present application.It should be pointed out that, for the ordinary skilled in the art, without departing from the principle of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data processing method, characterized by, include: When an input / output task is received, the length of the protection information required by the input / output task is obtained; Based on the length of the protection information, a target virtual page is selected from the virtual address space, and at least one target physical page is selected from the physical address space, wherein the size of the target virtual page matches the length of the protection information; Establish a mapping relationship between the target virtual page and the target physical page; Based on the mapping relationship, respond to the input / output task to perform read / write operations on protection information; The step of selecting a target virtual page from the virtual address space based on the length of the protection information includes: Obtain a virtual page size corresponding to each virtual subspace in the virtual address space, wherein each virtual subspace is evenly divided into multiple virtual pages; The length of the protection information is compared with the virtual page size corresponding to each virtual subspace to obtain a comparison list; A target virtual page is selected from the virtual subspace based on the comparison list; If there exists a first virtual page in the comparison list whose size is the same as the length of the protection information, then the number of first free virtual pages in the first virtual subspace is obtained. If the number of the first free virtual pages is equal to 0, then the second virtual subspace with a virtual page size greater than the length of the protection information is selected as a candidate subspace; Obtain the number of candidate subspaces; If the number of candidate subspaces is greater than or equal to 1, then obtain the number of second free virtual pages in the candidate subspaces; If the number of second free virtual pages is greater than the threshold, then a portion of the second free virtual pages in the candidate subspace are divided into multiple first free virtual pages and allocated to the first virtual subspace.

2. The method of claim 1, wherein, The length of the protection information required to obtain the input / output task includes: Obtain the total task size, number of subtasks, and subtask size of each subtask of the input / output task; Based on the total task size, the number of subtasks, and the size of the subtasks, the length of the protection information required for the input / output task is determined.

3. The method of claim 1, wherein, The step of selecting a target virtual page from the virtual subspace based on the comparison list includes: If the number of the first free virtual pages is greater than 0, then the first virtual subspace is used as the target subspace, and a first free virtual page is randomly selected from the first virtual subspace as the target virtual page.

4. The method according to claim 1, characterized in that, The step of selecting a target virtual page from the virtual subspace based on the comparison list includes: If the number of candidate subspaces is 1, then the candidate subspace is taken as the target subspace, and a second free virtual page is randomly selected from the candidate subspaces as the target virtual page.

5. The method according to claim 1, characterized in that, The step of selecting a target virtual page from the virtual subspace based on the comparison list further includes: If the number of candidate subspaces is greater than 1, then calculate the difference between the size of the second virtual page corresponding to the candidate subspace and the length of the protection information; The candidate subspace with the smallest difference is selected as the target subspace, and a third free virtual page is randomly selected from the candidate subspace with the smallest difference as the target virtual page.

6. The method according to claim 1, characterized in that, The step of selecting a target virtual page from the virtual subspace based on the comparison list further includes: If the number of candidate subspaces obtained is 0, an error is returned or a corresponding selection wait information is generated based on the information of reclaimable virtual subspaces from other input / output tasks.

7. The method according to claim 1, characterized in that, The step of selecting at least one target physical page from the physical address space based on the length of the protection information includes: Obtain the physical page size of each target physical page in the physical address space; Based on the length of the protection information and the size of the physical page, at least one target physical page is selected from the physical address space.

8. The method according to claim 1, characterized in that, Establishing the mapping relationship between the target virtual page and the target physical page includes: Obtain the physical page address of the target physical page and the virtual page address of the target virtual page; Create a mapping descriptor based on the physical page address and the virtual page address; The mapping relationship between the target virtual page and the target physical page is established based on the mapping descriptor.

9. The method according to claim 2, characterized in that, The step of responding to the input / output task based on the mapping relationship to perform read / write operations on protected information includes: Obtain the task descriptor of each subtask in the input / output task; The read / write type of each subtask is parsed based on the task descriptor; Based on the read / write type, determine the read / write location of the protection information; The read and write operations of the protection information are performed according to the read and write positions and the mapping relationship.

10. The method according to claim 8, characterized in that, The method further includes: Obtain the task response results of the input and output tasks; Based on the task response result, determine the completion status of the read / write operation; Based on the completion status, the target virtual page and target physical page corresponding to the input / output task are reclaimed.

11. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the length of the protection information required by the input / output task when an input / output task is received; The selection module is configured to select a target virtual page from the virtual address space based on the length of the protection information, and to select at least one target physical page from the physical address space based on the length of the protection information, wherein the size of the target virtual page matches the length of the protection information; The step of selecting a target virtual page from the virtual address space based on the length of the protection information includes: Obtain a virtual page size corresponding to each virtual subspace in the virtual address space, wherein each virtual subspace is evenly divided into multiple virtual pages; The length of the protection information is compared with the virtual page size corresponding to each virtual subspace to obtain a comparison list; A target virtual page is selected from the virtual subspace based on the comparison list; If there exists a first virtual page in the comparison list whose size is the same as the length of the protection information, then the number of first free virtual pages in the first virtual subspace is obtained. If the number of the first free virtual pages is equal to 0, then the second virtual subspace with a virtual page size greater than the length of the protection information is selected as a candidate subspace; Obtain the number of candidate subspaces; If the number of candidate subspaces is greater than or equal to 1, then obtain the number of second free virtual pages in the candidate subspaces; If the number of the second free virtual pages is greater than the threshold, then a portion of the second free virtual pages in the candidate subspace are divided into multiple first free virtual pages and allocated to the first virtual subspace; The mapping module is used to establish the mapping relationship between the target virtual page and the target physical page; The processing module is used to respond to the input / output task based on the mapping relationship to perform read / write operations on the protection information.

12. A computer device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1 to 10.

13. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the method of any one of claims 1 to 10.

Images