Data processing method and device, equipment and storage medium

By dividing data into data segments and mapping it to the operating system kernel page cache, the problem of hard disk I/O performance limitations is solved, data read and write efficiency and system performance are improved, and disk service life is extended.

CN118963644APending Publication Date: 2024-11-15HANGZHOU HIGH-TECH ZONE (BINJIANG) INSTITUTE OF BLOCKCHAIN & DATA SECURITY
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410910424.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Data read and write performance is limited by the I/O performance of the hard disk. Frequent read and write operations lead to disk performance attenuation and physical bad channels, reducing the disk service life and affecting system stability.

Method used

By dividing the pending data into data segments according to the size of the preset data segment and writing the data segments into the memory page one by one, mapping the memory page to the page cache of the operating system kernel according to the preset mapping relationship, completing data writing, avoiding Frequent I/O operations.

Benefits of technology

It improves the efficiency of data read and write operations, reduces the overhead of traditional I/O operations, improves system performance and response speed, extends disk service life, and meets the needs of high-performance computing and real-time data analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963644A_ABST
    Figure CN118963644A_ABST
Patent Text Reader

Abstract

The invention is suitable for the technical field of computers, and provides a data processing method and device, equipment and a storage medium, and the method comprises the steps: receiving a data writing request and to-be-processed data; in response to the data writing request, dividing the to-be-processed data into at least one data segment according to a preset data segment size; sequentially writing the at least one data segment into a memory page, wherein the size of the memory page is equal to the value of the size of the preset data segment; and mapping the memory page to a page cache of an operating system kernel according to a preset mapping relation to complete data writing. According to the scheme, the data request is efficiently received, processed and stored by standardizing the size of the data segment and the memory page during data storage and utilizing a memory mapping mechanism.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In the process of data processing, data storage and reading operations are often involved. Among them, disks, such as hard disk drives (HDDs) and solid-state drives (SSDs), are the main data storage devices in computer systems.

[0003] At present, data reading and writing performance is limited by the I / O (input / output) performance of the hard disk, and concentrated and frequent reading and writing operations in a specific disk area will not only cause disk performance degradation, but may also cause the formation of physical bad sectors, thereby reducing the service life of the disk and affecting system stability. Therefore, how to improve the efficiency of data reading and writing operations has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and storage medium, which can solve the technical problem of how to read and write data efficiently.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0006] Receive data write requests and data to be processed.

[0007] In response to a data write request, the data to be processed is divided into at least one data segment according to a preset data segment size.

[0008] At least one data segment is sequentially written into a memory page, and the size of the memory page is equal to the value of the preset data segment size.

[0009] Map the memory page to the page cache of the operating system kernel according to the preset mapping relationship to complete the data writing.

[0010] In one implementation, writing at least one data segment into a memory page in sequence includes: calculating the amount of data to be processed to obtain the amount of data to be processed; when the size of the amount of data to be processed is greater than or equal to a preset fixed value, determining the memory page as a minimum data writing unit; writing at least one data segment into the minimum data writing unit in sequence until at least one data segment is completely written.

[0011] In one implementation, writing at least one data segment into a memory page in sequence further includes:

[0012] When the amount of data to be processed is smaller than a preset fixed value, at least one data segment is written into the current memory page, and a data segment corresponding to the next data to be processed is obtained and written into the current memory page until the storage space of the current memory page is fully occupied.

[0013] In one implementation, metadata information of a memory page is recorded, and the metadata information includes at least one of the following: identification information of the memory page, starting position information of the memory page, size information of data written to the memory page, and file handle number information corresponding to the memory page; a data index is constructed based on the metadata information, and the data index is used to query the data to be processed.

[0014] In one implementation, a target data volume is determined according to the amount of data to be processed and a preset data volume; and a target memory page quantity value is calculated according to the target data volume and the memory page size. In one implementation, a data query request is received, the data query request including: identification information of the memory page to be queried, starting position information of the memory page to be queried, and data size information of the data to be queried; in response to the data query request, the data to be queried is queried according to the data index.

[0015] In one implementation, querying the data to be queried according to the data index includes: determining a target memory page from the memory pages according to the identification information of the memory page to be queried; determining the offset of the data to be queried in the target memory page according to the starting position information of the memory page to be queried; determining the length information of the data to be queried according to the data size information of the data to be queried; determining and reading the data to be queried according to the offset of the data to be queried in the target memory page and the length information of the data to be queried.

[0016] In a second aspect, an embodiment of the present application provides a data processing device, which has the function of implementing the method in the first aspect or any possible implementation thereof. Specifically, the device includes a unit for implementing the method in the first aspect or any possible implementation thereof.

[0017] In one embodiment, the device comprises:

[0018] The acquisition unit is used to receive data writing requests and data to be processed.

[0019] The processing unit is used to divide the data to be processed into at least one data segment according to a preset data segment size in response to a data write request.

[0020] The processing unit is further used to write at least one data segment into a memory page in sequence, and the size of the memory page is equal to the value of the preset data segment size.

[0021] The processing unit is also used to map the memory page to the page cache of the operating system kernel according to a preset mapping relationship to complete data writing.

[0022] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements a method according to any one of the implementation modes of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a computer device, the computer device implements the method of any one of the implementation methods of the first aspect above.

[0024] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device executes a method of any one of the implementation modes of the first aspect above.

[0025] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the data to be processed is divided into preset data segment sizes according to preset data segment sizes through data write requests, and the size of the memory page is set to be consistent with the size of the data segment, and then the divided data segments are written into the memory page one by one to ensure the persistence of the data in the memory; the memory page is mapped to the page cache of the operating system kernel according to the preset mapping relationship, and the system can efficiently receive, process and store data write requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of a scenario in which a buffer mechanism is used to store data, provided in an embodiment of the present application.

[0027] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application.

[0028] Figure 3 It is a schematic diagram of a memory mapping scenario provided in an embodiment of the present application.

[0029] Figure 4 It is a flow chart of writing data into a memory page provided in an embodiment of the present application.

[0030] Figure 5 It is a flow chart of writing data into a memory page provided in an embodiment of the present application.

[0031] Figure 6 It is a structural diagram of a data processing device provided in an embodiment of the present application.

[0032] Figure 7 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In the field of storage system design, the buffer mechanism is widely used to optimize data writing performance. Figure 1 Let me introduce it in detail.

[0034] Figure 1 It is a schematic diagram of a scenario in which a buffer mechanism is used to store data, provided in an embodiment of the present application.

[0035] Before understanding the principle of this method, some related concepts are introduced first.

[0036] The operating system is the core software of the computer system, responsible for managing the computer's hardware resources and providing various services to users and applications. Its main functions include process management, memory management, file management, device management, etc.

[0037] A buffer is a temporary storage area for storing data, usually used to temporarily store data during data transmission in order to improve data transmission efficiency.

[0038] An application is a program that a user runs in a computer system to implement specific functions and process data. An application can use the operating system to implement file reading and writing functions, that is, to use the operating system to read data from a file or write data to a file.

[0039] A disk is a storage device used to store data. It is usually composed of a disk platter and a read-write head and can store large amounts of data persistently.

[0040] User space is the area in the operating system allocated to user programs. User programs can directly access and operate data and resources in the user space.

[0041] Kernel space is the area in the operating system used to run the kernel and provide system-level services. User programs cannot directly access and operate kernel space and need to send requests to the kernel through system calls and other methods.

[0042] like Figure 1 As shown in the figure, assuming that the application submits a write request to the disk through the operating system, the operating system writes the data to the buffer of the physical memory, and the data temporarily resides in the buffer. The data is managed and processed in the kernel space of the operating system and transferred through the buffer. After a period of time, when the buffer is full or an explicit trigger refresh operation occurs, the operating system will uniformly transfer the data set in the buffer to the kernel space for further processing. Finally, when the data is ready, the operating system will pass the data to the disk and perform data write operations through the disk controller to store the data persistently on the disk.

[0043] However, this method faces a challenge in complex read-write interaction scenarios, namely, the instantaneous state management of data in the buffer. Specifically, when faced with the need to read or update data immediately, if the buffer contains relevant data, the buffer content must first be synchronized (Flush) to persistent storage (such as a disk file) to ensure that subsequent read or modification operations are based on the latest and consistent data state. This process not only requires the buffered data to be faithfully written back (Committing buffered data to disk), but also means that there will be delays in reloading or modifying data from disk. This disadvantage makes it difficult for this method to achieve efficient data reading and writing in scenarios with high concurrency or strict real-time requirements.

[0044] In view of the above problems, the present application proposes a data processing method, which can realize efficient reading and writing of data and avoid frequent I / O operations.

[0045] In order to further illustrate the technical solution of the present application, specific embodiments are provided below.

[0046] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application.

[0047] like Figure 2 As shown, the above method includes the following steps S201 to S204.

[0048] S201: Receive a data write request and data to be processed.

[0049] A data write request may be understood to refer to a request issued by an application or a system to write data to a storage device (eg, a disk, etc.).

[0050] The data to be processed refers to the data that is about to be written to the storage device.

[0051] Combining the principles above, it can be understood that the data reading and writing function is a bridge for the operating system to interact with the disk device. The data to be processed can include various types of data, such as text, images, audio, video, etc. The specific data form is not limited here and can be selected according to the actual situation.

[0052] S202: In response to a data write request, divide the data to be processed into at least one data segment according to a preset data segment size.

[0053] The preset data segment size refers to the size of each data segment set in advance during the data writing process, and the value here is 4KB, that is, 4096 bytes.

[0054] It is understood that the standard size of the operating system memory page is 4KB. When the preset data segment size matches the memory page size, it can ensure that the data segment occupies an integer number of memory pages, avoiding the problem of memory fragmentation. This can minimize the overhead of memory allocation and management and improve memory utilization.

[0055] S203. Write at least one data segment into a memory page in sequence, wherein the size of the memory page is equal to the value of a preset data segment size.

[0056] Memory page refers to the division of computer memory into blocks of fixed size. Each block is called a page, which is used to manage and exchange data.

[0057] The size of a memory page refers to the capacity of each memory page, which is a fixed value set by the computer operating system. Here, the memory page size is 4KB.

[0058] Since the memory page size is consistent with the data segment size, each data segment can be stored in exactly one memory page. This alignment can make data access more efficient, avoid the extra overhead caused by cross-page access, and improve memory access efficiency. At the same time, the exchange and transmission of data between different storage locations is simpler and more efficient, because data can be moved or copied in blocks without additional segmentation or merging operations on the data, allowing the system to manage and process data more efficiently, reducing the additional burden on the operating system during data writing and access, thereby improving the overall system performance and response speed, and maximizing the collaborative efficiency of hardware and software levels.

[0059] S204: Map the memory page to the page cache of the operating system kernel according to a preset mapping relationship to complete data writing.

[0060] The preset mapping relationship refers to a preset corresponding relationship between a memory page and a page cache of an operating system kernel during a data writing process.

[0061] For example, suppose an application needs to write a 1MB file to disk. In this process, the data write request is the request sent by the application to the operating system, and the data to be processed is the content of the 1MB file.

[0062] Assume that the preset data segment size is set to 4KB during the writing process. This means that the data to be processed will be divided into 4KB units to obtain multiple divided data segments.

[0063] According to the preset mapping relationship, the operating system will write these data segments into the memory page in sequence. Once the entire data segment is written into the memory page, the memory page will be mapped to the page cache of the operating system kernel to complete the data writing.

[0064] Specifically, a series of such pages can be collectively associated with the page cache of the kernel space through the memory mapping mechanism of the virtual memory management subsystem.

[0065] The virtual memory management subsystem is part of the operating system's memory management and is responsible for managing the mapping between virtual memory (also known as virtual address space) and physical memory. In modern operating systems, the virtual memory management subsystem maps the process's virtual address space to physical memory using data structures such as page tables, allowing the process to access a larger address space than the actual physical memory.

[0066] The page cache that collectively associates pages to the kernel space refers to mapping a series of pages from files (or other devices) to the memory through the memory mapping mechanism of the virtual memory management subsystem in the operating system, and caching these pages in the page cache of the kernel space.

[0067] The page cache is a memory cache used to store file data on disk, designed to improve the performance of accessing files.

[0068] For example, when an application accesses data in a file, the operating system maps the pages in the file to the application's virtual address space. When these pages are accessed, a page fault is triggered, and the operating system reads the corresponding data in the file into physical memory and caches these pages in the page cache of the kernel space. If other applications or processes also need to access the same file data, the operating system will directly obtain the data from the page cache without accessing the disk again.

[0069] By associating pages with the kernel space page cache, file access performance can be improved, disk I / O operations can be reduced, and access to file data can be accelerated. The kernel space page cache also allows multiple processes to share the same file data, avoiding duplicate data reads and improving system performance.

[0070] This mapping process not only implicitly ensures efficient bridging of data between physical memory and disk storage media, but also means that data writes to the kernel page cache can be treated as nearly instant disk write operations, greatly reducing the overhead of traditional I / O operations.

[0071] In one implementation, a target data volume is determined based on the amount of data to be processed and a preset data volume; and a target memory page quantity value is calculated based on the target data volume and the memory page size.

[0072] The amount of data to be processed can be understood as the amount of data in the file to be processed. The preset data amount is a custom value and can be set according to actual conditions. The following is an introduction with reference to a specific example.

[0073] It can be understood that from the user state perspective, the configuration logic of the number of pages in the file system strictly follows the actual size of the file, and a dynamic adjustment mechanism is used to adapt to the processing requirements of files of different sizes.

[0074] The specific Page allocation algorithm is as follows:

[0075] Page Count=(File Size+1) / Page Size.

[0076] Among them, Page Count represents the number of memory pages, which is calculated by rounding up; File Size represents the amount of data to be processed (file size), the preset data amount is 1, and Page Size represents the memory page size.

[0077] It can be understood that the preset data volume can be set to 0, 1, 2, 3, 4, 5, etc. according to the actual situation. Setting the preset data volume value to 1 here is a way of rounding up, which can ensure that one more page can be allocated when the file size is an integer multiple of the memory page size, thereby avoiding boundary condition errors. Compared with other values, the deviation is the smallest when the preset data volume value is 1. For example, assuming the memory page size is 1024 and the file size is 1020, if it is +5, then (1020+5) / 1024, rounded up, will allocate 2 pages, while (1020+1) / 1024 will only allocate one page.

[0078] Boundary condition errors can be avoided by setting a preset data amount. In some cases, even if the file size is an integer multiple of the memory page size, additional space may be required to store metadata or other auxiliary information due to the layout or alignment requirements of the internal data structure. Allocating an extra page can be used as a simple strategy to avoid such boundary condition errors.

[0079] Also, in some cases, allocating more space can provide a buffer for future data growth or format changes, so that even if data is increased later, there is no need to immediately reallocate space.

[0080] This approach ensures accurate configuration and efficient utilization of memory resources while maintaining overall system stability and responsiveness.

[0081] Combine the following Figure 3 Let's take a closer look at how to perform memory mapping.

[0082] Figure 3 It is a schematic diagram of a memory mapping scenario provided in an embodiment of the present application.

[0083] like Figure 3 As shown in the figure, a memory page is constructed, and N memory pages with a size of 4KB are mapped (Memory-Mapped) to the kernel's page cache through virtual addresses. Since there is an equivalent relationship between the pages in the kernel and the disk files (File), it can be considered that when the data is written to the kernel's page cache, the data is considered to be written to the disk once.

[0084] Through data write requests, the data to be processed is divided into preset data segment sizes according to the preset data segment size, so that it can be written and processed segment by segment later; the divided data segments are written into the memory pages one by one to ensure the persistence of the data in the memory; the memory pages are mapped to the page cache of the operating system kernel according to the preset mapping relationship, and the system can efficiently receive, process and store data write requests.

[0085] The data to be processed is highly refined into multiple data fragments, each of which is 4KB in size. This fine-grained sharding design allows the system to manage data storage and access more flexibly. When data is written to the corresponding memory page, the system can perform data operations directly in the memory without immediately flushing the data to the persistent storage medium, because the data has been effectively recorded and stored in the Page fragment in the memory. Since the data is available immediately after being written and there is no need to wait for flushing to the disk, the accessibility and immediacy of the data are significantly enhanced. This can speed up data reading and improve the overall responsiveness of the system, which is particularly suitable for the stringent requirements of high-performance computing and real-time data analysis scenarios.

[0086] Compared with traditional solutions, this application achieves fast data writing and reading by eliminating the step of performing flush to persistent storage, improves system performance and response speed, and enhances the immediate availability and accessibility of data, meeting the needs of high-performance computing and real-time data analysis scenarios.

[0087] In one implementation, writing at least one data segment into a memory page in sequence includes: calculating the amount of data to be processed to obtain the amount of data to be processed; when the size of the amount of data to be processed is greater than or equal to a preset fixed value, determining the memory page as a minimum data writing unit; writing at least one data segment into the minimum data writing unit in sequence until at least one data segment is completely written.

[0088] The default fixed value is the standard size of a memory page, which is 4KB.

[0089] Calculating the amount of data to be processed is to obtain the size of the data to be processed (in bytes). For example, the 1MB file mentioned above has a data size of 1MB. It can be understood that the data size of 1MB is greater than 4KB, that is, the data to be processed initially occupies a complete 4K page. It can also be understood that in the process of storing the 1MB file, the memory page is the basic operation unit.

[0090] Combine the following Figure 4 Let's take a closer look at how to write data to a memory page.

[0091] Figure 4 It is a flow chart of writing data into a memory page provided in an embodiment of the present application.

[0092] like Figure 4 As shown, when the amount of data to be processed is greater than 4KB (memory page size), it means that the first page memory when storing the data to be processed can be completely occupied, and the data to be processed is stored in sequence in the manner of 4KB segments, ..., 4KB segments, until the last data segment is stored. It can be understood that the last data segment may be completely occupied or may have space left. The specific storage situation is related to the actual amount of data and is not limited here.

[0093] The data to be processed can be divided into data segments according to a preset fixed value (such as 4K), and written into the memory page segment by segment. By dividing the data into fixed-size data segments, the efficiency of the write operation can be improved, the memory management overhead can be reduced, the memory access efficiency can be optimized, and the system burden can be reduced, thereby improving the overall system performance.

[0094] This storage method is closely synchronized with contemporary hard disk technology standards, such as Advanced Format, that is, by accurately matching the hardware alignment boundaries, reducing the addressing overhead and potential access delays in read and write operations, it can not only ensure the efficiency of the data access path, but also promote the improvement of the overall response speed of the system.

[0095] In one implementation, writing at least one data segment into a memory page in sequence also includes: when the amount of data to be processed is less than a preset fixed value, writing at least one data segment into the current memory page, obtaining a data segment corresponding to the next data to be processed, and writing it into the current memory page until the storage space of the current memory page is fully occupied.

[0096] It is understandable that the size of the data to be processed is likely to be less than 4KB, for example, the size of the data to be processed is 2KB. At this time, it means that the data to be processed cannot completely occupy an independent memory page. If the next data to be processed is stored in the next memory page, there will be remaining space in the memory space of the current memory page. In this case, in order to effectively use the memory space, the current memory page can be filled first, and then written to the next memory page according to the next data segment.

[0097] Combine the following Figure 5 Let me introduce it in detail.

[0098] Figure 5 It is a flow chart of writing data into a memory page provided in an embodiment of the present application.

[0099] like Figure 5 As shown, the data segment to be written can be determined by calculating the size of the current data to be processed. It is determined whether the remaining space of the current memory page is sufficient to accommodate the data segment of the current data to be processed. If it is sufficient, the data segment is directly written to the current memory page; if it is insufficient, a part of the data segment that meets the remaining space is first written to the current memory page, and the other part of the data segment is written to the next memory page. The above steps are repeated as needed until all the data to be processed are written to the memory page.

[0100] By processing data writing in this way, memory space can be used more efficiently, memory fragmentation problems can be avoided, and data segments can be stored in memory pages as completely as possible, thereby improving memory utilization and system performance.

[0101] In one implementation, metadata information of a memory page is recorded, and the metadata information includes at least one of the following: identification information of the memory page, starting position information of the memory page, size information of data written to the memory page, and file handle number information corresponding to the memory page; a data index is constructed based on the metadata information, and the data index is used to query the data to be processed.

[0102] Metadata information can be understood as descriptive information of data, which is used to record and manage the attributes, characteristics and organization of data.

[0103] It can be understood that after completing the data writing operation, the system obtains and registers key metadata parameters. As an example and not limitation, the metadata information can be the identification information (Page ID) of the memory page, the starting position information (Position) of the memory page, the size information (Size) of the data written to the memory page, and the file handle number information (File ID) corresponding to the memory page.

[0104] It can be understood that the purpose of recording the metadata information of the memory page is to describe the attribute information and feature information of the memory page, such as the size of the memory page, the size of the free space, the data storage location, etc. Based on this metadata information, a data index can be constructed to query and access the data to be processed in the memory page. Accordingly, the recording of metadata is the basis for achieving efficient data management and recovery.

[0105] As an example but not limitation, a metadata information table can be maintained in an operating system or database management system to record relevant attribute information of each memory page. When it is necessary to query or access data in the memory, the storage location can be quickly located according to the data index, thereby improving data access efficiency.

[0106] By building a data index, you can quickly locate the data in the memory page, reduce data access time, and improve system response speed. Data index can help quickly find and access data in memory, improve query efficiency, and is suitable for scenarios that require frequent query and retrieval of data.

[0107] In one implementation, a data query request is received, the data query request including at least one of the following: identification information of a memory page to be queried, starting position information of the memory page to be queried, and data size information of the data to be queried; in response to the data query request, the data to be queried is queried according to a data index.

[0108] It can be understood that the data query request corresponds to the metadata information mentioned above, that is, the data query is performed according to the stored metadata information.

[0109] Assume that there are multiple memory pages in a system, and each memory page has a corresponding metadata information record, including the identification information, size information, data storage location, etc. of the memory page. When the system receives a data query request, the request includes the identification information of the memory page to be queried. The system searches for the corresponding memory page in the metadata information based on the identification information, and obtains the starting position information and data size information of the memory page. Then the system builds a data index based on this information and queries the corresponding data to be queried.

[0110] Through data query requests and data indexing, and using indicators such as the identification information and starting position information of the memory page to be queried, the system can accurately locate the data to be queried, avoid unnecessary search operations, improve data positioning accuracy, and achieve efficient data query and access operations.

[0111] Compared with the traditional solution of using data pre-write log (WAL, Write-Ahead Logging) to persistent storage media or using direct I / O intervention to modify data, the data processing method provided in the present application has detailed metadata records for each data fragment (Page), so the system can quickly and accurately locate the target data without going through the process of writing logs to persistent storage media. It can also be understood that when writing data, the system can directly write the data to the corresponding Page fragment without waiting for the data to be written to the write-ahead log (WAL) and then written to the persistent storage medium, thereby improving the efficiency of writing data.

[0112] In addition, since the metadata contains the specific location information of the data in the Page, the system can directly locate the Page where the target data is located when it needs to modify the data, and perform the modification operation without the need for direct I / O intervention to modify the data. This direct positioning and modification method reduces the frequent access to the persistent storage medium and improves the flexibility and efficiency of data processing.

[0113] In other words, the method in the present application avoids the need to write a pre-log (WAL) to a persistent storage medium in the traditional solution, and does not require direct I / O for data modification, thereby improving the efficiency and flexibility of data processing.

[0114] In one implementation, querying the data to be queried according to the data index includes: determining a target memory page from the memory pages according to the identification information of the memory page to be queried; determining the offset of the data to be queried in the target memory page according to the starting position information of the memory page to be queried; determining the length information of the data to be queried according to the data size information of the data to be queried; determining and reading the data to be queried according to the offset of the data to be queried in the target memory page and the length information of the data to be queried.

[0115] Assume that there is a memory storage space in the system that contains multiple memory pages, and each memory page has corresponding identification information, starting position information, and size information. When the system receives a data query request, the request contains the identification information, starting position information, and data size information of the memory page to be queried.

[0116] First, the system finds the corresponding memory page in the metadata information based on the identification information of the memory page to be queried and determines the target memory page. Then, the system determines the offset of the data to be queried in the target memory page based on the starting position information of the memory page to be queried, and determines the length information of the data to be queried based on the data size information of the data to be queried. Next, the system determines the location of the data to be queried in the target memory page based on the offset and length information of the data to be queried and reads the data.

[0117] The following is a specific example to illustrate how to query data.

[0118] Assume that there are two memory pages in the system, Page1 and Page2, where Page1 starts at position 0 and has a size of 100, and Page2 starts at position 100 and has a size of 150. When the system receives a data query request, the request is for data starting at position 120 and having a length of 30 in Page2.

[0119] At this time, based on the identification information of the data to be queried, the system determines that the target memory page is Page2.

[0120] According to the starting position information of the data to be queried being 120, it is determined that the offset of the queried data in the target memory page is 20.

[0121] According to the data size information being 30, it is determined that the length information of the data to be queried is 30.

[0122] Based on the offset and length information, the system determines that the location of the data to be queried in the Page2 memory page is 140 (starting position 100 + offset 20), and reads data with a length of 30.

[0123] The specific numerical value can be determined according to the actual situation. This is just an example and not a limitation.

[0124] This method of querying the data to be queried based on the data index can accurately calculate the offset and length information, so that the system can efficiently locate and read the data to be queried, thereby improving the accuracy and efficiency of the query.

[0125] From the above, we can understand that by subdividing files into segments with a basic unit of 4KB and mapping them to the memory space, we can achieve micro-granular management of files. This process not only promotes independent metadata management of each 4K segment, ensuring accurate tracking and description of each data write, but also enables real-time monitoring and flexible operation of the instantaneous state of data, thereby achieving fine management of data status while maintaining efficient data writing performance.

[0126] By judging whether the amount of data can occupy a separate memory page for storage in different situations during the data writing process, it is ensured that all data writing operations can strictly follow the 4KB boundary alignment principle, which is consistent with the standard size of the memory page (Page Frame) of modern operating systems, thereby eliminating the loss of read and write efficiency caused by non-alignment. Through this alignment strategy, not only the continuity and consistency of data reading and writing are enhanced, but also the performance potential of the hardware storage subsystem and the upper-level software is maximized, ensuring that the system-level I / O operations achieve optimal efficiency, thereby improving the overall performance of data processing and storage.

[0127] It can also be understood that by utilizing the memory management mechanism, the data to be written is pre-loaded into the memory to achieve high-speed data caching. This strategy not only effectively bypasses the bottleneck link in the traditional I / O stack, but also makes full use of the high-speed read and write characteristics of RAM (random access memory) compared to disk media, greatly improving the throughput and response speed of data processing. With the help of memory mapping technology, direct memory access (DMA) of data is achieved, eliminating the overhead of kernel buffer copying, and ensuring efficient and seamless data transmission between memory and persistent storage media.

[0128] In addition, to ensure data integrity and consistency, the use of saved metadata information can ensure rapid recovery in the event of abnormal situations and avoid data loss, thereby improving write efficiency while ensuring system reliability and data security.

[0129] The above mainly introduces the method of the embodiment of the present application in conjunction with the accompanying drawings. It should be understood that although the various steps in the flowcharts involved in the embodiments described above are shown in sequence, these steps are not necessarily performed in sequence in the order shown in the figure. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a part of the steps or stages in other steps. A device of an embodiment of the present application is introduced below in conjunction with the accompanying drawings. For simplicity, when introducing the device below, it will be appropriately omitted, and the relevant content can refer to the relevant description in the method above, and will not be repeated.

[0130] Figure 6 It is a structural diagram of a data processing device provided in an embodiment of the present application.

[0131] like Figure 6As shown, the data processing device 1000 includes the following units.

[0132] The acquisition unit 1001 is used to receive a data writing request and data to be processed.

[0133] The processing unit 1002 is used to divide the data to be processed into at least one data segment according to a preset data segment size in response to a data write request.

[0134] The processing unit 1002 is also used to write at least one data segment into a memory page in sequence, and the size of the memory page is equal to the value of the preset data segment size.

[0135] The processing unit 1002 is further configured to map the memory page to the page cache of the operating system kernel according to a preset mapping relationship to complete data writing.

[0136] In one implementation, the device 1000 further includes a storage unit 1003, which can be used to store instructions and / or data, so as to implement the method in the above embodiment.

[0137] In one implementation, the processing unit 1002 is also used to calculate the data volume of the data to be processed and obtain the data volume to be processed; when the size of the data volume to be processed is greater than or equal to a preset fixed value, determine the memory page as the minimum data writing unit; write at least one data segment into the minimum data writing unit in sequence until at least one data segment is written.

[0138] It can be understood that by calculating the size of the amount of data to be processed and thereby determining a memory page as a minimum data writing unit, and performing a data writing operation according to the minimum data unit, effective use of the memory page can be ensured.

[0139] In one implementation, the processing unit 1002 is also used to, when the amount of data to be processed is less than a preset fixed value, write at least one data segment to the current memory page, obtain the data segment corresponding to the next data to be processed, and write it to the current memory page until the storage space of the current memory page is fully occupied.

[0140] It can be understood that by not immediately writing the data to be processed to a new memory page when the amount of data is less than a preset fixed value, but continuing to write the data segment to the current memory page, the frequency of memory page switching can be reduced, system overhead can be reduced, and the efficiency of data writing can be improved.

[0141] At the same time, switching memory pages after the storage space of the current memory page is fully occupied can also reduce the generation of memory fragmentation, avoid unnecessary waste of memory space, and further improve memory utilization. This method can optimize memory management to a certain extent and improve the overall performance and efficiency of the system.

[0142] In one implementation, the processing unit 1002 is also used to record metadata information of the memory page, the metadata information including at least one of the following: identification information of the memory page, starting position information of the memory page, size information of the data written to the memory page, and file handle number information corresponding to the memory page; a data index is constructed based on the metadata information, and the data index is used to query the data to be processed.

[0143] It can be understood that by recording the metadata information of the memory page and building the data index, the target memory page can be quickly located, the starting position and size of the data can be determined, and the required data can be quickly accessed and queried, thereby improving data processing efficiency. By recording metadata information and building data indexes, data consistency can be ensured. For example, when updating or deleting data, the data to be modified or deleted can be accurately found based on the information of the data index, ensuring the correctness and integrity of the data operation.

[0144] In one implementation, the processing unit 1002 is further configured to determine a target data volume according to the amount of data to be processed and a preset data volume; and calculate a target memory page quantity value according to the target data volume and the memory page size.

[0145] It can be understood that by determining the target data volume based on the amount of data to be processed and the preset data volume, over-allocation or under-allocation of memory resources can be avoided, thereby more effectively utilizing system resources.

[0146] In one implementation, the acquisition unit 1001 is also used to receive a data query request, which includes: identification information of the memory page to be queried, starting position information of the memory page to be queried, and data size information of the data to be queried; the processing unit 1002 is also used to respond to the data query request and query the data to be queried according to the data index.

[0147] Corresponding to the above, by using data indexes to perform data queries, the target memory page can be quickly located, and the starting position and size of the data can be determined, so as to quickly access and query the required data and improve data processing efficiency.

[0148] In one implementation, the processing unit 1002 is also used to determine a target memory page from the memory pages based on the identification information of the memory page to be queried; determine an offset of the data to be queried in the target memory page based on the starting position information of the memory page to be queried; determine length information of the data to be queried based on the data size information of the data to be queried; determine and read the data to be queried based on the offset of the data to be queried in the target memory page and the length information of the data to be queried.

[0149] It can be understood that by accurately describing the identification information, starting position, data size and other information, the specific location of the data to be queried in the memory can be accurately located, avoiding erroneous reading caused by fuzzy or biased data location. If the location and size information of the data in the memory are clear, the data to be queried can be directly located and read, reducing unnecessary data scanning and processing processes, and improving retrieval efficiency and speed.

[0150] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0151] Figure 7 Schematic diagram of the structure of the computer device provided in the embodiment of the present application. Figure 7 As shown, the computer device 3000 of this embodiment includes: at least one processor 3100 ( Figure 7 Only one processor, memory 3200, and a computer program 3210 stored in the memory 3200 and executable on at least one processor 3100 are shown. When the processor 3100 executes the computer program 3210, the computer device implements the steps in the above embodiments.

[0152] The processor 3100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0153] The memory 3200 may be an internal storage unit of the computer device 3000 in some embodiments, such as a hard disk or memory of the computer device 3000. The memory 3200 may also be an external storage device of the computer device 3000 in other embodiments, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 3000. Further, the memory 3200 may also include both an internal storage unit of the computer device 3000 and an external storage device. The memory 3200 is used to store an operating system, an application program, a boot loader (Boot Loader) data, and other programs, such as a program code of a computer program, etc. The memory 3200 may also be used to temporarily store data that has been output or is to be output.

[0154] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0155] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a computer device, the computer device can implement the steps in the above-mentioned method embodiments.

[0156] The embodiment of the present application provides a computer program product. When the computer program product is executed on a computer device, the computer device can implement the above-mentioned methods.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the computer device can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0158] It should be understood that the size of the sequence number of each step in the above-described embodiment does not mean the order of execution, and the order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiment of the present application. In the description, for the purpose of explanation rather than for limitation, specific details such as specific system structure, technology and the like are proposed, so as to thoroughly understand the embodiment of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed description of well-known systems, devices, circuits and methods is omitted to prevent unnecessary details from hindering the description of the present application.

[0159] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0160] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0161] Additionally, in the specification of the present application and the description of the appended claims, the terms “include,” “comprise,” “have,” and their variations all mean “including but not limited to,” unless otherwise specifically emphasized.

[0162] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0163] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0164] In the embodiments provided in the present application, it should be understood that the disclosed devices, computer equipment and methods can be implemented in other ways. For example, the device and computer equipment embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0165] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: include: Receive data writing request and data to be processed; In response to the data write request, dividing the data to be processed into at least one data segment according to a preset data segment size; Writing the at least one data segment into a memory page in sequence, wherein the size of the memory page is equal to the value of the preset data segment size; The memory page is mapped to the page cache of the operating system kernel according to a preset mapping relationship to complete data writing.

2. The method according to claim 1, characterized in that Writing the at least one data segment into the memory page in sequence includes: Calculating the amount of the data to be processed to obtain the amount of data to be processed; When the size of the amount of data to be processed is greater than or equal to the preset fixed value, determining the memory page as the minimum data writing unit; The at least one data segment is sequentially written into the minimum data writing unit until the at least one data segment is completely written.

3. The method according to claim 2, characterized in that The step of writing the at least one data segment into the memory page in sequence further includes: When the size of the data to be processed is smaller than the preset fixed value, the at least one data segment is written into the current memory page, and the data segment corresponding to the next data to be processed is obtained and written into the current memory page until the storage space of the current memory page is fully occupied.

4. The method according to claim 2 or claim 3, characterized in that: The method further comprises: Recording metadata information of the memory page, the metadata information including at least one of the following: identification information of the memory page, starting position information of the memory page, size information of data written to the memory page, and file handle number information corresponding to the memory page; A data index is constructed according to the metadata information, and the data index is used to query the data to be processed.

5. The method according to claim 4, characterized in that The method further comprises: Receive a data query request, wherein the data query request includes at least one of the following: identification information of a memory page to be queried, starting position information of the memory page to be queried, and data size information of the data to be queried; In response to the data query request, the data to be queried is queried according to the data index.

6. The method according to claim 2 or claim 3, characterized in that: The method further comprises: Determine the target data volume according to the to-be-processed data volume and the preset data volume; The target memory page quantity value is calculated according to the target data volume and the memory page size.

7. The method according to claim 5, characterized in that The querying of the to-be-queried data according to the data index comprises: Determining a target memory page from the memory pages according to the identification information of the memory page to be queried; Determine the offset of the data to be queried in the target memory page according to the starting position information of the memory page to be queried; Determine the length information of the data to be queried according to the data size information of the data to be queried; The data to be queried is determined and read according to the offset of the data to be queried in the target memory page and the length information of the data to be queried.

8. A data processing device, characterized in that: include: An acquisition unit, used for receiving a data writing request and data to be processed; a processing unit, configured to divide the to-be-processed data into at least one data segment according to a preset data segment size in response to a data write request; The processing unit is further used to write the at least one data segment into a memory page in sequence, the size of the memory page being equal to the value of the preset data segment size; The processing unit is also used to map the memory page to the page cache of the operating system kernel according to a preset mapping relationship to complete data writing.

9. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer device, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Method, device and equipment for randomly reading and writing data and storage medium

    CN119806411A

  • Method, device, equipment and storage medium for random reading and writing data

    CN119806411B

  • Data processing method and system

    CN120578339A

  • Data processing method and system

    CN120578339B