Data processing method, medium, apparatus, and computing device
By using the FFDB storage engine to convert data attributes and organizational structures between memory and disk files, the write and read efficiency issues in massive data storage are solved, achieving efficient data processing and reducing resource consumption and storage costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU NETEASE CLOUD MUSIC TECH CO LTD
- Filing Date
- 2022-12-19
- Publication Date
- 2026-07-21
AI Technical Summary
Existing storage engines cannot simultaneously achieve efficient writing and reading of massive amounts of data. In particular, in streaming data storage scenarios, the problems of random high-throughput writing, random high-efficiency reading, and large value storage have not been effectively solved.
A Flat File Database (FFDB) storage engine was designed. It improves data writing and reading efficiency by caching data in memory and converting it to read-only storage medium when a threshold is reached, and updating the data to disk files. It uses an inverted index structure to record data locations, organizes disk files using a linked list structure, and performs garbage collection periodically.
It significantly improves data writing and reading efficiency, reduces the consumption of storage and computing resources, enhances user experience, saves CPU and IO resources, and reduces the load on the storage engine.
Smart Images
Figure CN115793982B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this disclosure relate to the field of data processing, and more specifically, the embodiments of this disclosure relate to data processing methods, media, apparatus, and computing devices. Background Technology
[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.
[0003] Storage engines, broadly speaking, refer to NoSQL databases. By using different technologies and associated functionalities, they can store data from the database in files (or memory). Currently, several types of storage engines have emerged, such as hash-based storage engines, log-structured-merge-tree (LSM) storage engines, and B+ tree storage engines. These storage engines can meet different user storage needs.
[0004] However, the storage engines in related technologies cannot achieve high read and write efficiency for massive amounts of data at the same time. Therefore, there is an urgent need for a solution that can simultaneously achieve high efficiency in writing and reading massive amounts of data. Summary of the Invention
[0005] This disclosure provides a data processing method, medium, apparatus, and computing device to achieve the goal of simultaneously handling the efficient writing and reading of massive amounts of data.
[0006] In a first aspect of this disclosure, a data processing method is provided, comprising: in response to detecting a write request for first data, caching the first data in a first memory, wherein the first memory is a read-write storage medium; in response to completion of storage in the first memory, detecting whether the current storage amount of the first memory is greater than a preset storage amount; in response to the current storage amount of the first memory being greater than the preset storage amount, modifying the attributes of the first memory to obtain a second memory, wherein the second memory is a read-only storage medium; and updating the data in the second memory to a disk file.
[0007] In one embodiment of this disclosure, the data processing method further includes: using an inverted index structure to record the disk file where the data is located and the offset position of the data in the disk file.
[0008] In another embodiment of this disclosure, the disk file corresponds one-to-one with the second memory. The data processing method further includes: after updating the data in the second memory to the disk file, storing the disk file in a first-in-first-out (FIFO) data structure, which includes a linked list and / or a double-ended queue.
[0009] In yet another embodiment of this disclosure, the data processing method further includes: obtaining the storage time corresponding to the target disk file located at the end of the first-in-first-out data structure; determining whether the target disk file has expired based on the storage time; and deleting the target disk file if it has expired.
[0010] In yet another embodiment of this disclosure, the data processing method further includes: storing first data in a log file in response to a write request.
[0011] In another embodiment of this disclosure, the data processing method further includes: in response to receiving a request to read second data, reading the second data in a first memory; in response to not reading the second data in the first memory, reading the second data in a second memory; in response to not reading the second data in the second memory, obtaining the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file; and reading the second data according to the identification information and the target offset position.
[0012] In another embodiment of this disclosure, obtaining the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file includes: obtaining the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file from the inverted index structure.
[0013] In another embodiment of this disclosure, the data processing method further includes: after obtaining the second memory, in response to the number of second memory being greater than a preset number, deleting K second memory locations, where K = MN, M is the number of second memory locations, and N is the preset number.
[0014] In another embodiment of this disclosure, modifying the attributes of the first memory to obtain the second memory includes: modifying the target flag bit of the first memory to obtain the second memory, wherein the target flag bit is used to identify the attributes of the memory.
[0015] In a second aspect of the present disclosure, a medium is provided that stores computer execution instructions, which, when executed by a processor, are used to implement the data processing method of the first aspect.
[0016] In a third aspect of this disclosure, a data processing apparatus is provided, comprising: a caching module, configured to cache the first data in a first memory in response to detecting a write request for the first data, wherein the first memory is a read-write storage medium; a detection module, configured to detect whether the current storage amount of the first memory is greater than a preset storage amount in response to storage completion in the first memory; a modification module, configured to modify the attributes of the first memory to obtain a second memory in response to the current storage amount of the first memory being greater than the preset storage amount, wherein the second memory is a read-only storage medium; and an update module, configured to update the data in the second memory to a disk file.
[0017] In a fourth aspect of the present disclosure, a computing device is provided, comprising: a processor and a memory connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the data processing method as described in the first aspect.
[0018] According to the data processing method, medium, apparatus, and computing device of this disclosure, when the amount of data in a readable and writable storage medium exceeds a threshold, the readable and writable storage medium is converted into a read-only storage medium to cache the written data, and the data in the read-only storage medium is updated to a disk file, wherein the file in the disk is stored in a linked list structure, thereby significantly improving the efficiency of data writing and reading, and reducing the consumption of storage and computing resources, bringing a better experience to the user. Attached Figure Description
[0019] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:
[0020] Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method according to the embodiments of this disclosure;
[0021] Figure 2 This is a flowchart illustrating the data processing method according to an embodiment of the present disclosure;
[0022] Figure 3 This is a schematic diagram illustrating the data writing process according to an embodiment of the present disclosure;
[0023] Figure 4 This is a schematic diagram of the disk file storage structure according to an embodiment of the present disclosure;
[0024] Figure 5 This is a schematic diagram illustrating the data reading process according to an embodiment of the present disclosure;
[0025] Figure 6This is a schematic diagram of the structure of the storage medium according to an embodiment of the present disclosure;
[0026] Figure 7 This is a schematic diagram of the structure of the data processing apparatus according to an embodiment of the present disclosure;
[0027] Figure 8 This is a schematic diagram of the structure of a computing device according to an embodiment of the present disclosure.
[0028] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation
[0029] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.
[0030] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.
[0031] According to embodiments of this disclosure, a data processing method, medium, apparatus, and computing device are proposed. It should be understood herein that the terms used refer to the following:
[0032] Storage Engine: A general term for non-relational databases.
[0033] Flat File Database (FFDB): A storage engine suitable for random storage of massive streaming data.
[0034] Streaming data: a method of producing and organizing data in a chronological order.
[0035] Garbage collection (GC) in databases mainly refers to the cleanup of expired and useless data.
[0036] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.
[0037] In addition, the data involved in this disclosure may be data authorized by the user or fully authorized by all parties. The collection, dissemination and use of the data shall comply with the requirements of relevant national laws and regulations. The implementation methods / executives of this disclosure may be combined with each other.
[0038] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview
[0040] The inventors have discovered that in the design and implementation of storage engines, the relevant technologies are mainly divided into three types of storage engines: hash structure, log-structured merge-tree (LSM) structure, and B+ tree structure.
[0041] Among them, the hash structure storage engine typically stores the actual data in memory, using a hash table data structure to construct the in-memory data storage location. It features high data retrieval efficiency and suitability for point-to-point query scenarios. However, because the index of the hash table structure stores indexes of data locations—logically continuous data that is physically discontinuous—it is unsuitable for matching or range searches. Furthermore, due to its low data read / write efficiency, it is generally used for in-memory data storage.
[0042] LSM (Low-Side Module) storage engines are disk-based storage that sacrifices some read throughput to maximize write throughput. Update operations are first written to a memtable in memory, and then all at once to disk after a threshold is reached. Files on disk are divided into multiple levels, and periodic merge operations are performed to clean up expired, deleted, and duplicate data, and to sort the data. However, because LSM storage uses an append-only update approach, maximizing write throughput at the expense of read throughput, and because of large-scale merge operations, additional write amplification overhead occurs. In scenarios with large data volumes (large values), merge pressure can be enormous, even severely impacting the performance of the storage engine. Therefore, LSM storage is not suitable for scenarios with high read performance requirements.
[0043] B+ tree storage, also a disk-based storage method, is primarily used in read-heavy, write-light scenarios. Because it's based on a page-based data organization, its data read / write efficiency is relatively high. Furthermore, since logically adjacent data is also physically adjacent, matching and range searches are also efficient. However, because all data is stored in a single tree, B+ tree splitting and merging operations occur during large-scale update operations, leading to significant random write issues. Moreover, disk random write performance is poor, especially for solid-state drives (SSDs), where write amplification exists, causing a sharp drop in storage engine performance. Therefore, B+ trees are not suitable for high-write-throughput scenarios.
[0044] Therefore, this disclosure provides a data processing method that can design an FFDB storage engine for the storage characteristics of streaming data to solve the problems of random high-throughput writing, random high-efficiency reading, large values and massive data, and realize low-cost and high-efficiency reading and writing of massive streaming data.
[0045] Specifically, the FFDB storage engine records a Write Ahead Log (WAL) file and introduces a memory structure for caching written data. When the cached data in this memory structure exceeds a threshold, it is directly flushed to the disk file. The disk file is organized in a First-In-First-Out (FIFO) manner according to a linked list structure, thus converting random writes into sequential writes. Furthermore, during the flushing of data to the disk file, the file number and offset information of the data are used as an inverted index and stored in the index structure, which can accelerate random data retrieval. In addition, since streaming data has a fixed expiration window, the engine periodically retrieves the file at the end of the linked list to determine if it has expired. If it has expired, the file is directly deleted, thus achieving the lowest-cost deletion of expired data, saving storage resources, and improving the efficiency of data writing.
[0046] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.
[0047] Application Scenarios Overview
[0048] First refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method according to embodiments of this disclosure. Figure 1In this architecture, the storage engine FFDB can include a memory buffer, an immutable memory buffer, an inverted index storage structure, a disk, and a read module. The memory buffer is used for writing and reading data, the immutable memory buffer is used for read-only data, the inverted index is used to store the file and offset where the data is stored, the disk file is used to store the data in the immutable memory buffer, and the read module is used to read data from the memory buffer, the immutable memory buffer, or the disk file and return the data to the user.
[0049] In this scenario, data can be represented as (k, v), where k can represent the keyword information corresponding to the data (key), and v can represent the data (value). When data is written, it is first written to the log file WAL, then to the memory buffer. When the memory buffer's data storage capacity reaches a threshold, it is converted into an immutable memory buffer, and the data in the immutable memory buffer is flushed to the disk file. Each immutable memory buffer is stored in a disk file, and all disk files are stored in a first-in, first-out linked list. Therefore, the data writing efficiency is improved.
[0050] In this scenario, after the data is flushed to the Disk File, the Inverted Index records the Disk File where the data is located and the offset position of the data in the Disk File. The recorded content can be represented as (k, offset), where k can be used to represent the keyword information of the data, and offset can be used to represent the disk file where the data is located and the offset position.
[0051] In this scenario, data can be read by retrieving the value 'v' corresponding to the data 'k'. First, the data is read from the Memory Buffer. If it can be read from the Memory Buffer, the data read is successful; otherwise, it is read from the Immutable Memory Buffer. If it can be read from the Immutable Memory Buffer, the data read is successful; otherwise, the data read is performed using the Disk File and offset position recorded by the Inverted Index (i.e., offset). The data 'k' is then read from the Disk File, and the corresponding value 'v' is used to read the data. If it is read from the Disk File, the data read is successful; otherwise, the data read fails. Therefore, this improves data reading efficiency.
[0052] In this scenario, to save storage resources, the storage engine FFDB can also perform GC (garbage collection) automatically in the background in real time.
[0053] Exemplary methods
[0054] The following is combined Figure 1 Application scenarios, refer to Figure 2 This document describes a data processing method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.
[0055] refer to Figure 2 , Figure 2 This is a flowchart illustrating the data processing method according to an embodiment of the present disclosure, such as... Figure 2 As shown, the data processing method includes the following steps:
[0056] S201: In response to the detection of a write request for the first data, cache the first data in the first memory.
[0057] In this step, the first memory is designated as a read-write storage medium.
[0058] Specifically, the first data can be represented as (k, v), where k can represent the keyword information corresponding to the first data, or be denoted as key, and v can represent the first data, or be denoted as value. The data writing process can be as follows: Figure 3 As shown.
[0059] Figure 3 This is a schematic diagram illustrating the data writing process according to an embodiment of the present disclosure. Figure 3In this context, when it is necessary to write the first data, the first data can be written to the first memory buffer, which is readable and writable.
[0060] Optionally, the first memory location can be a skip list structure.
[0061] S202: In response to the completion of storage in the first memory, detect whether the current storage amount in the first memory is greater than the preset storage amount.
[0062] Specifically, since the size of the first memory is limited, after writing the first data into the first memory, it is necessary to check the current storage capacity of the first memory. If the current storage capacity of the first memory is greater than or equal to the preset storage capacity, it means that no more new data can be written into the first memory. If the current storage capacity of the first memory is less than the preset storage capacity, it means that new data can continue to be written into the first memory.
[0063] S203: In response to the fact that the current storage amount of the first memory is greater than the preset storage amount, the attributes of the first memory are modified to obtain the second memory.
[0064] In this step, the second memory is designated as a read-only storage medium.
[0065] Specifically, such as Figure 3 As shown, if the current storage amount of the first memory is detected to be greater than or equal to the preset storage amount, that is, no new data can be written to the first memory, the first memory with read and write properties can be converted into the second memory with read-only properties, the immutable memory buffer. The second memory is only used to read data and cannot modify or write data.
[0066] Optionally, when new data needs to be written into memory, a new first memory buffer can be allocated to store the new data.
[0067] S204: Update the data in the second memory to the disk file.
[0068] Specifically, such as Figure 3 As shown, after modifying the attributes of the first memory to obtain the second memory, the data in the second memory can be directly flushed to the disk file. That is, the data in the second memory is copied to the disk file without being deleted, so as to speed up the subsequent data reading speed.
[0069] The data processing method disclosed herein significantly improves the efficiency of data writing and reading, and reduces the consumption of storage and computing resources, thus providing a better user experience, when the current storage capacity of the first memory with read-write properties is greater than the preset storage capacity. This is achieved by converting the first memory into a second memory with read-only properties to cache the written data and updating the data in the second memory to the disk file.
[0070] In one embodiment of this disclosure, the data processing method further includes: using an inverted index structure to record the disk file where the data is located and the offset position of the data in the disk file.
[0071] In this embodiment, such as Figure 3 As shown, after updating the data in the second memory to the disk file, the disk file where the data is located and the offset position of the data in the disk file can be recorded to facilitate subsequent data reading.
[0072] Specifically, recording the disk file where the data resides can be achieved by recording the disk file number. When recording the disk file and offset of the data, an inverted index structure can be used. The recorded content can be represented as (k, offset), where k can represent the key information of the data, and offset can represent the disk file and offset of the data. Using an inverted index structure to record the disk file and offset of the data can accelerate random access to data on the disk.
[0073] Alternatively, the inverted index structure can be a B+ tree or an LSM tree.
[0074] In another embodiment of this disclosure, the disk file corresponds one-to-one with the second memory. The data processing method further includes: after updating the data in the second memory to the disk file, storing the disk file in a first-in-first-out (FIFO) data structure, which includes a linked list and / or a double-ended queue.
[0075] In this embodiment, since there is at least one second memory location, and the data in each second memory location is stored in a disk file, there is a one-to-one correspondence between the disk file and the second memory location. To facilitate the management of the disk files, they can be stored in a first-in, first-out (FIFO) data structure, such as... Figure 4 As shown. Figure 4 This is a schematic diagram of the disk file storage structure according to an embodiment of the present disclosure. Figure 4 In the example of a linked list, when the first memory is converted to the second memory, the data in the second memory will be flushed to a disk file, which is stored at the head of the linked list.
[0076] In yet another embodiment of this disclosure, the data processing method further includes: obtaining the storage time corresponding to the target disk file located at the end of the first-in-first-out data structure; determining whether the target disk file has expired based on the storage time; and deleting the target disk file if it has expired.
[0077] In this embodiment, to conserve disk storage resources, garbage collection (GC) can be performed periodically or in real-time. Since disk files are stored in a first-in, first-out (FIFO) data structure, disk files that enter the FIFO structure earlier (i.e., those at the end) have an earlier storage time, while those entering later (i.e., those at the beginning) have a later storage time. Therefore, the storage time of target disk files at the end can be detected in real-time. If this storage time exceeds a preset storage duration, the target disk file is determined to be expired and can be deleted to free up disk space.
[0078] Optionally, the storage time of the disk file is the time when the disk file enters the first-in-first-out data structure.
[0079] Optionally, internal garbage collection (GC) can be performed automatically on the disk.
[0080] In yet another embodiment of this disclosure, the data processing method further includes: storing first data in a log file in response to a write request.
[0081] In this embodiment, in response to a write request for first data, in addition to writing the first data to the first memory, the first data can also be written to the log file WAL, such as... Figure 3 As shown.
[0082] Specifically, WAL files can be used for data recovery in storage engines under abnormal scenarios, ensuring that data already written to the storage engine is not lost, such as extreme abnormal scenarios like abnormal process termination, abnormal machine shutdown, or power failure.
[0083] In another embodiment of this disclosure, the data processing method further includes: in response to receiving a request to read second data, reading the second data in a first memory; in response to not reading the second data in the first memory, reading the second data in a second memory; in response to not reading the second data in the second memory, obtaining the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file; and reading the second data according to the identification information and the target offset position.
[0084] In this embodiment, the first memory buffer stores the latest data; the second memory buffer, an immutable memory buffer, stores immutable data, which originates from the first memory. Therefore, the second memory buffer stores the latest data, second only to the first memory. The data stored on the disk is the oldest data. Thus, data can be read from the first memory first; if it cannot be read from the first memory, it can be read from the second memory; if it cannot be read from the second memory, it can be read from the disk, thereby improving data reading speed.
[0085] Specifically, the process of reading the second data can be as follows: Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the data reading process according to an embodiment of this disclosure. Figure 5 In the process, the second data can be read from the first memory. If the second data can be read from the first memory, the reading of the second data is successful. If the second data cannot be read from the first memory, the second data can be read from the second memory. If the second data can be read from the second memory, the reading of the second data is successful. If the second data cannot be read from the second memory, the second data can be read from the disk. If the second data can be read from the disk, the reading of the second data is successful. If the second data cannot be read from the disk, the reading of the second data fails.
[0086] Specifically, since the disk file containing the data and its offset within the disk file are recorded when the data in the second memory are flushed to the disk file, the second data can be read using the disk file containing the second data and its offset within the disk file.
[0087] In another embodiment of this disclosure, obtaining the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file includes: obtaining the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file from the inverted index structure.
[0088] In this embodiment, the second data can be represented as (k, v), where k can represent the keyword information of the second data, or be denoted as key, and v can represent the second data, or be denoted as value. Figure 5 As shown, when reading the second data, v can be obtained by reading k of the second data, thereby realizing the reading of the second data.
[0089] Specifically, since the inverted index structure records the disk file where the data is stored and the offset position of the data within that disk file (i.e., the inverted index structure records the data's (k, offset), when reading the second data from the disk, the identifier information of the disk file containing the second data and the target offset position of the second data within that disk file, i.e., offset, can be obtained from the inverted index structure. Based on the offset, the corresponding k for the second data can be determined, and then the corresponding v can be obtained, enabling fast reading of the second data.
[0090] Optionally, the identification information of the disk file can be the disk file number, preset identifier, etc.
[0091] Optionally, when writing data to the first memory or disk file, the write time can be recorded. Therefore, when reading data, the Time To Live (TTL) can be read simultaneously, and the data's keyword information, the data itself, and its TTL can be returned to the user together.
[0092] In another embodiment of this disclosure, the data processing method further includes: after obtaining the second memory, in response to the number of second memory being greater than a preset number, deleting K second memory locations, where K = MN, M is the number of second memory locations, and N is the preset number.
[0093] In this embodiment, the number of second memory is limited. Therefore, after converting the first memory into second memory, the number of second memory in the current storage engine can be detected. If the number of second memory is greater than the preset number, the excess second memory can be deleted. For example, if the current number of second memory M is 6 and the preset number N is 5, the excess second memory out of the 6 can be deleted to free up storage space and thus save storage resources.
[0094] Optionally, when deleting redundant secondary memory, you can delete any of it, or you can delete the earliest converted secondary memory in chronological order.
[0095] In another embodiment of this disclosure, modifying the attributes of the first memory to obtain the second memory includes: modifying the target flag bit of the first memory to obtain the second memory, wherein the target flag bit is used to identify the attributes of the memory.
[0096] In this embodiment, when converting the first memory to the second memory, the attributes of the first memory can be directly modified. The attributes of the first memory can be identified by the target flag bit of the first memory. Therefore, by modifying the target flag bit of the first memory, the attributes of the first memory can be changed, thereby realizing the conversion of the first memory to the second memory, which can improve the data read and write efficiency.
[0097] This data processing method, by designing a storage engine for massive streaming data storage scenarios, solves the problems of low request performance and high storage costs in scenarios involving random real-time read / write, large value records, and fixed expiration windows under massive data storage. Compared with related technologies, under the same online traffic, pressure, and storage configuration, the technical solution provided by this disclosure greatly saves central processing unit (CPU) resources, input / output (IO) resources, and reduces the load on the storage engine, significantly improving data storage efficiency in streaming massive data scenarios. In practical use, compared with related technologies, this disclosure saves approximately 90% of CPU and IO resources. In other scenarios, the technical solution of this disclosure can save even more CPU and IO resources and reduce the load to a greater extent.
[0098] Exemplary media
[0099] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 6 The medium for exemplary embodiments of this disclosure will be described.
[0100] Figure 6 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present disclosure, with reference to... Figure 6 As shown, the medium can be a storage medium 60 storing a program product for implementing the above-described method according to embodiments of the present disclosure. This storage medium can be a portable compact disc read-only memory (CD-ROM) and includes computer-executable instructions for causing a computing device to execute the data processing method provided in this disclosure. However, the program product of this disclosure is not limited to this.
[0101] The program product disclosed herein can take the form of any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0102] A readable signal medium may include data signals propagated in baseband or as part of a carrier wave, carrying computer-executed instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.
[0103] Computer-executable instructions for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The computer-executable instructions can be executed entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).
[0104] Exemplary device
[0105] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 7 The data processing apparatus of the exemplary embodiments of this disclosure will be described to implement the method in any of the above method embodiments. The implementation principle and technical effect are similar, and will not be repeated here.
[0106] Figure 7 This is a schematic diagram of the structure of the data processing apparatus according to an embodiment of the present disclosure, such as... Figure 7 As shown, the data processing apparatus 700 includes:
[0107] The caching module 701 is used to cache the first data in the first memory in response to the detection of a write request for the first data, wherein the first memory is a read-write storage medium.
[0108] The detection module 702 is used to detect whether the current storage amount in the first memory is greater than the preset storage amount in response to the completion of storage in the first memory;
[0109] Modification module 703 is used to modify the attributes of the first memory in response to the current storage amount of the first memory being greater than the preset storage amount, so as to obtain the second memory, the attribute of which is read-only storage medium;
[0110] Update module 704 is used to update the data in the second memory to the disk file.
[0111] Optionally, the data processing apparatus 700 further includes a recording module (not shown) for recording the disk file containing the data and the offset position of the data in the disk file using an inverted index structure.
[0112] Optionally, the disk file corresponds one-to-one with the second memory. The data processing device 700 further includes: a first storage module (not shown), used to update the data in the second memory to the disk file and then store the disk file in a first-in-first-out (FIFO) data structure, which includes a linked list and / or a double-ended queue.
[0113] Optionally, the data processing device 700 further includes: a first deletion module (not shown), used to obtain the storage time corresponding to the target disk file at the end of the first-in-first-out data structure; determine whether the target disk file has expired based on the storage time; and delete the target disk file if it has expired.
[0114] Optionally, the data processing apparatus 700 further includes a second storage module (not shown) for storing first data in a log file in response to a write request.
[0115] Optionally, the data processing apparatus 700 further includes: a reading module (not shown), configured to read the second data from the first memory in response to receiving a request to read the second data; read the second data from the second memory in response to not reading the second data from the first memory; obtain the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file in response to not reading the second data from the second memory; and read the second data according to the identification information and the target offset position.
[0116] Optionally, the reading module is also used to obtain the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file by obtaining the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file from the inverted index structure.
[0117] Optionally, the data processing device 700 further includes a second deletion module (not shown), used to delete K second memory units after obtaining the second memory units, in response to the number of second memory units being greater than a preset number, where K = MN, M is the number of second memory units, and N is the preset number.
[0118] Optionally, the modification module 703 is also used to modify the attributes of the first memory to obtain the second memory by modifying the target flag bit of the first memory to obtain the second memory, wherein the target flag bit is used to identify the attributes of the memory.
[0119] Exemplary computing device
[0120] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 8 A computing device according to an exemplary embodiment of the present disclosure will be described.
[0121] Figure 8 The computing device 80 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0122] Figure 8 This is a schematic diagram of the structure of a computing device according to an embodiment of this disclosure. Figure 8 As shown, the computing device 80 is presented in the form of a general-purpose computing device. The components of the computing device 80 may include, but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including the processing unit 801 and the storage unit 802). The at least one storage unit 802 includes a memory storing computer-executable instructions; the at least one processing unit 801 includes a processor that executes the computer-executable instructions to implement the methods described above.
[0123] The 803 bus includes a data bus, a control bus, and an address bus.
[0124] Storage unit 802 may include readable media in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 8023.
[0125] Storage unit 802 may also include a program / utility 8025 having a set (at least one) program module 8024, such program module 8024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0126] The computing device 80 can also communicate with one or more external devices 804 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 805. Furthermore, the computing device 80 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 806. Figure 8As shown, network adapter 806 communicates with other modules of computing device 80 via bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 80, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0127] It should be noted that although several units / modules or sub-units / modules of the data processing apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.
[0128] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0129] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims
1. A data processing method, characterized in that, include: In response to the detection of a write request for first data, the first data is cached in first memory, wherein the attributes of the first memory are readable and writable storage medium; In response to the completion of storage in the first memory, it is detected whether the current storage amount in the first memory is greater than the preset storage amount; In response to the fact that the current storage amount of the first memory is greater than the preset storage amount, the attributes of the first memory are modified to obtain the second memory, and the attributes of the second memory are read-only storage medium; The data in the second memory is copied to the corresponding disk file, wherein the disk file corresponds one-to-one with the second memory. The disk file is stored in a first-in-first-out data structure; Obtain the storage time of the target disk file located at the end of the first-in-first-out data structure; Based on the storage time, determine whether the target disk file has expired; If expired, the target disk file will be deleted.
2. The data processing method according to claim 1, characterized in that, Also includes: An inverted index structure is used to record the disk file where the data is located and the offset position of the data in the disk file.
3. The data processing method according to claim 1 or 2, wherein the first-in-first-out data structure includes a linked list and / or a double-ended queue.
4. The data processing method according to claim 1 or 2, characterized in that, Also includes: In response to the write request, the first data is stored in the log file.
5. The data processing method according to claim 1 or 2, characterized in that, Also includes: In response to receiving a read request for the second data, the second data is read from the first memory; In response to the fact that the second data is not read from the first memory, the second data is read from the second memory; In response to the fact that the second data is not read from the second memory, the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file are obtained; The second data is read based on the identification information and the target offset position.
6. The data processing method according to claim 5, characterized in that, The step of obtaining the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file includes: Obtain the identification information of the disk file containing the second data and the target offset position of the second data in the disk file from the inverted index structure.
7. The data processing method according to claim 1 or 2, characterized in that, After obtaining the second memory, the process also includes: In response to the fact that the number of second memory locations exceeds the preset number, K second memory locations are deleted, where K=MN, M is the number of second memory locations, and N is the preset number.
8. The data processing method according to claim 1 or 2, characterized in that, The step of modifying the attributes of the first memory to obtain the second memory includes: The target flag of the first memory is modified to obtain the second memory, wherein the target flag is used to identify the attributes of the memory.
9. A medium, characterized in that, The medium stores computer execution instructions, which, when executed, are used to implement the data processing method as described in any one of claims 1 to 8.
10. A data processing apparatus, characterized in that, include: A caching module is used to cache the first data in a first memory in response to the detection of a write request for the first data, wherein the first memory is a read-write storage medium. The detection module is used to detect whether the current storage amount in the first memory is greater than a preset storage amount in response to the completion of storage in the first memory; The modification module is used to modify the attributes of the first memory in response to the current storage amount of the first memory being greater than the preset storage amount, so as to obtain the second memory, wherein the attribute of the second memory is read-only storage medium. The update module is used to update the data in the second memory to the disk file, wherein the disk file corresponds one-to-one with the second memory. The first storage module is used to store the disk file into a first-in-first-out data structure; The first deletion module is used to obtain the storage time corresponding to the target disk file located at the end of the first-in-first-out data structure; determine whether the target disk file has expired based on the storage time; and delete the target disk file if it has expired.
11. The data processing apparatus according to claim 10, characterized in that, Also includes: The recording module is used to record the disk file where the data is located and the offset position of the data in the disk file using an inverted index structure.
12. The data processing apparatus according to claim 10 or 11, characterized in that, The first-in-first-out (FIFO) data structure includes linked lists and / or deques.
13. The data processing apparatus according to claim 10 or 11, characterized in that, Also includes: The second storage module is used to store the first data in a log file in response to the write request.
14. The data processing apparatus according to claim 10 or 11, characterized in that, Also includes: A reading module is configured to read the second data from the first memory in response to receiving a read request for the second data; In response to the fact that the second data is not read from the first memory, the second data is read from the second memory; In response to the fact that the second data is not read in the second memory, the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file are obtained; the second data is read according to the identification information and the target offset position.
15. The data processing apparatus according to claim 14, characterized in that, The reading module is also used to obtain the identification information of the disk file where the second data is located and the target offset position of the second data in the disk file through the following methods: Obtain the identification information of the disk file containing the second data and the target offset position of the second data in the disk file from the inverted index structure.
16. The data processing apparatus according to claim 10 or 11, characterized in that, Also includes: The second deletion module is used to delete K second memory locations after obtaining the second memory location, in response to the number of second memory locations being greater than a preset number, where K=MN, M is the number of second memory locations, and N is the preset number.
17. The data processing apparatus according to claim 10 or 11, characterized in that, The modification module is also used to modify the attributes of the first memory to obtain the second memory in the following ways: The target flag of the first memory is modified to obtain the second memory, wherein the target flag is used to identify the attributes of the memory.
18. A computing device, comprising: A processor, and a memory connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the data processing method as described in any one of claims 1 to 8.