Data Processing Method, Apparatus, Computer Device, and Storage Medium

Abstract

An embodiment of the present invention relates to a data processing method, apparatus, computer device, and storage medium, including: obtaining application data to be stored, and caching the application data in a first storage space corresponding to a cache disk according to a preset data structure; detecting in real time the data storage status of all the first storage spaces of the cache disk; when the data storage status is a first status, transferring the data stored in the first storage space corresponding to the first status to a second storage space corresponding to a main storage disk; when the data storage status is a second status, controlling a garbage collection program to clear the data stored in the first storage space corresponding to the second status. Thus, by allowing the garbage collection program to execute, but not allowing the data storage status of the storage space to be changed during the garbage collection process, problems such as the cache disk not processing data storage during the garbage collection process causing service interruption and large latency can be avoided, and the data processing efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer data storage, and more particularly to a data processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] "Digital transformation" has recently become a buzzword across industries. It redefines business processes (processes, scenarios, relationships, and participants), encompassing IT technologies such as computing, storage, transmission, and interaction. Storage, as the IT foundation of digital transformation, places enormous demands on it. International Data Corporation predicts that the total amount of global data will reach 175ZB by 2025. Consequently, the storage industry is pursuing high-performance, highly reliable, and low-cost-per-capacity storage systems. Most performance bottlenecks in IT systems stem from storage, but the current unit price of SSDs is over 10 times that of HDDs. The I / O models of most business systems adhere to the principle of locality, making the use of small-capacity SSDs as cache for larger HDDs a cost-effective, accelerated storage solution for some business systems, meeting the requirements for high performance and low cost per capacity.

[0003] Block cache (Bcache) is a commonly used HDD acceleration solution in the industry, but the space management bucket state machine in Bcache currently has the following design flaws: the bucket state machine is a hierarchical state machine. After entering the second-level state machine, all bucket states will be changed. At this time, buckets are required not to be eliminated and reused. At this time, there may be many or few available buckets. In the case of a large number of random small IO writes, when entering the second-level state machine, the cache space elimination process cannot be started during the metadata garbage collection (Garbage Collection, GC) process. When there is no cache disk space, the write IO is stuck, which manifests itself in the business as zero performance. The long zeroing time causes the business IO to not return for a long time, which has the risk of causing business interruption. For example Figure 1 As shown in the figure, the existing 3-node Ceph storage test uses 128 concurrent 8K random writes for 780 seconds. The business-side statistics show that the zeroing time is as long as 30 seconds. Summary of the Invention

[0004] In view of this, in order to solve the above technical problems or part of the technical problems, the embodiments of the present invention provide a data processing method, apparatus, computer equipment and storage medium.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0006] Obtain application data to be stored, and cache the application data in a corresponding first storage space of a cache disk according to a preset data structure;

[0007] Real-time detect the data storage status of all the first storage spaces of the cache disk;

[0008] When the data storage status is a first status, transfer the data stored in the first storage space corresponding to the first status to a second storage space corresponding to a main memory disk;

[0009] When the data storage status is a second status, control a garbage collection program to clear the data stored in the first storage space corresponding to the second status.

[0010] In a possible implementation manner, the method further includes:

[0011] When the garbage collection program inspects all the first storage spaces, control the data storage status of all the first storage spaces to remain unchanged.

[0012] In a possible implementation manner, the method further includes:

[0013] Based on the data structure corresponding to the data stored in the first storage space, determine the data storage status of all the first storage spaces of the cache disk.

[0014] In a possible implementation manner, the method further includes:

[0015] Obtain the data storage location identified in the data structure;

[0016] If the data stored in the current first storage space does not exist in the main memory disk, determine that the data storage status of the current first storage space is the first status;

[0017] If the data stored in the current first storage space exists in the main memory disk, determine that the data storage status of the current first storage space is the second status.

[0018] In a possible implementation manner, the method further includes:

[0019] Release the first storage space after the data is cleared for re-caching data.

[0020] In a possible implementation manner, there is one cache disk and multiple main memory disks;

[0021] Full data mapping exists between the cache disk and the multiple main memory disks.

[0022] In a second aspect, an embodiment of the present invention provides a data processing device, including:

[0023] An acquisition module, configured to acquire application data to be stored, and cache the application data in a first storage space corresponding to a cache disk according to a preset data structure;

[0024] A detection module, configured to detect in real time the data storage status of all the first storage spaces of the cache disk;

[0025] A storage module, configured to transfer the data stored in the first storage space corresponding to the first state to a second storage space corresponding to a main storage disk when the data storage status is the first state;

[0026] A garbage collection module, configured to control a garbage collection program to clear the data stored in the first storage space corresponding to the second state when the data storage status is the second state.

[0027] In a possible implementation manner, the apparatus further includes:

[0028] A control module, configured to control the data storage status of all the first storage spaces to remain unchanged when the garbage collection program inspects all the first storage spaces.

[0029] In a third aspect, an embodiment of the present invention provides a computer device, including: a processor and a memory, where the processor is configured to execute a data processing program stored in the memory to implement the data processing method described in the first aspect above.

[0030] In a fourth aspect, an embodiment of the present invention provides a storage medium, including: the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method described in the first aspect above.

[0031] The data processing solution provided by the embodiment of the present invention caches the application data to be stored in the first storage space corresponding to the cache disk according to a preset data structure by acquiring the application data to be stored; detects in real time the data storage status of all the first storage spaces of the cache disk; transfers the data stored in the first storage space corresponding to the first state to the second storage space corresponding to the main storage disk when the data storage status is the first state; controls the garbage collection program to clear the data stored in the first storage space corresponding to the second state when the data storage status is the second state. Compared with the prior art, in the metadata garbage collection process, the cache disk does not process data storage, and in terms of business, the business performance is zero. In this solution, by allowing the garbage collection program to execute, but not allowing the data storage status of the storage space to be changed during the garbage collection process, it is possible to avoid the problems of business interruption and large delay caused by the cache disk not processing data storage during the garbage collection process, and improve the data processing efficiency. Description of the Drawings

[0032] Figure 1 Performance curve diagram of cache disk data storage in the prior art;

[0033] Figure 2 Flow schematic diagram of a data processing method provided by an embodiment of the present invention;

[0034] Figure 3 Flow schematic diagram of another data processing method provided by an embodiment of the present invention;

[0035] Figure 4 Data mapping architecture diagram of a cache disk and multiple main memory disks provided by an embodiment of the present invention;

[0036] Figure 5 Data structure diagram provided by an embodiment of the present invention;

[0037] Figure 6 Bucket space management structure diagram provided by an embodiment of the present invention;

[0038] Figure 7 State machine schematic diagram of the prior art;

[0039] Figure 8 State machine schematic diagram provided by an embodiment of the present invention;

[0040] Figure 9 Performance comparison curve diagram of cache disk data storage in the prior art and an embodiment of the present invention;

[0041] Figure 10 Structure schematic diagram of a data processing device provided by an embodiment of the present invention;

[0042] Figure 11 Structure schematic diagram of a computer device provided by an embodiment of the present invention. Detailed Embodiments

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0044] For ease of understanding of the embodiments of the present invention, the following will further explain with specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation to the embodiments of the present invention.

[0045] Figure 2 A schematic flowchart of a data processing method provided by an embodiment of the present invention is as follows Figure 2 As shown, the method specifically includes:

[0046] The present invention mainly optimizes the disk-level cache module in the existing Linux kernel, uses Bcache for data management. The Bcache module abstracts high-speed media such as SSD or SCM as a cache disk, and supports three modes: writeback, writethrough, and writearound. In the writeback mode, data is cached in the cache disk and then organized in an order-friendly manner for the main memory disk such as HDD, and the cached data is written to the main memory disk.

[0047] The data mapping method of Bcache can be regarded as a variant of full associative mapping. Inside Bcache, the main memory disk is abstracted as a device logical ID number, and the entire cache disk is abstracted as a cache set (cache_set), so as to realize the full data mapping relationship between a cache disk and multiple main memory disks, corresponding to realizing the logic that multiple main memory disks share a cache disk, which simplifies the maintenance operation. The architecture is as follows Figure 4 As shown.

[0048] S21. Obtain application data to be stored, and cache the application data in a first storage space corresponding to the cache disk according to a preset data structure.

[0049] In an embodiment of the present invention, the application data to be stored is obtained and cached in the first storage space corresponding to the cache disk according to a preset data structure. Due to the full data mapping relationship between the cache disk and multiple main memory disks, except for the cache metadata position in the cache disk, a space in the cache disk may be mapped to any space of the main memory disk. All the metadata of the application data I / O is organized into a B+ tree data structure, which can accelerate the query speed and the swap-in and swap-out speed between memory and disk. The organization relationship between data and metadata is as follows Figure 5 As shown.

[0050] The data layout of the cache disk adopts a segmented layout method, and the space management is based on buckets as the unit for space management. Among them, the first bucket is used to store cache superblock information, including basic information of the cache. The position of the superblock bucket on the cache disk is fixed.

[0051] Next to the superblock bucket is the log bucket, which currently has multiple functions, including: 1. Recording application IO metadata for cache update acceleration; 2. The metadata B+ tree root bucket location; 3. The location of the linked list header that stores all bucket information; 4. The location of the bucket that stores all main storage disk UUIDs. The location of the log bucket is also fixed on the cache disk.

[0052] The remaining buckets include: 1. B+ tree bucket; 2. bucket that stores all bucket information; 3. bucket that stores UUID, which is randomly assigned and determined as metadata bucket; 4. data bucket. Figure 6 A bucket space management structure diagram is shown. In the embodiment of the present invention, the first storage space can be a bucket.

[0053] The basic unit of Bcache disk space management is bucket. Each application data to be stored needs to be allocated bucket space after entering the cache disk. After the data is written to the bucket, it is inserted into the B+ tree, and the write IO is completed in one go. In this way, when data is written to the bucket in writeback mode and enters the data occupied state, the Bcache state machine currently used by the community can only migrate the bucket from the data occupied state to the data free state through the garbage collection second-level state machine. The state machine is as follows: Figure 7 shown.

[0054] As can be seen from the state machine diagram, after the garbage collection process starts, it enters the second-level state machine, where all buckets are migrated to the GC_MARK=0 state. At this point, if an eviction event occurs, both occupied and unoccupied buckets will be evicted and reused, leading to data inconsistencies between the cache and storage disks. The existing solution is to disallow eviction events during the garbage collection process. During this time, the available space in the Bcache bucket may be high or low. In the case of a large number of small I / Os, garbage collection may last over 40 seconds. Once the last bucket is exhausted, write I / O performance will be affected, resulting in write I / O hung, meaning that write I / O performance will drop to zero. Repeated reads from the same location will prevent the cache disk from caching new data, resulting in read performance comparable to that of an HDD.

[0055] S22 : Detect the data storage status of all first storage spaces of the cache disk in real time.

[0056] Change the double-layer state machine of the first storage space of Bcache into a single-layer state machine, and real-time detect the data storage status of all the first storage spaces of the cache disk. Among them, an event where dirty_data = 0 can be added to the first storage space, and this event represents the data storage status of the first storage space. After caching data in the first storage space and the data has not been stored in the main memory disk, it is determined that dirty_data = 1, that is, the data storage status of the first storage space is the first state, and the first state means that the data stored in the first storage space does not exist in the main memory disk.

[0057] Optionally, when no data is cached in the first storage space, it is determined that dirty_data = 0, that is, the data storage status of the first storage space is the second state, and the second state means that there is no cached data in the first storage space.

[0058] S23. When the data storage status is the first state, transfer the data stored in the first storage space corresponding to the first state to the second storage space corresponding to the main memory disk.

[0059] When the data storage status of the first storage space is the first state, transfer the data stored in the first storage space corresponding to the first state to the second storage space corresponding to the main memory disk.

[0060] S24. When the data storage status is the second state, control the garbage collection program to clear the data stored in the first storage space corresponding to the second state.

[0061] When the data storage status of the first storage space is the second state, control the garbage collection program to clear the data stored in the first storage space corresponding to the second state.

[0062] It should be noted that when the garbage collection program inspects all the first storage spaces, control the data storage status of all the first storage spaces to remain unchanged, that is, the double-layer state machine of the first storage space becomes a single-layer state machine, and no longer changes the data storage status according to the state machine.

[0063] The data processing method provided by an embodiment of the present invention includes obtaining application data to be stored and caching the application data in a first storage space corresponding to a cache disk according to a preset data structure; detecting in real time the data storage status of all the first storage spaces of the cache disk; when the data storage status is a first status, transferring the data stored in the first storage space corresponding to the first status to a second storage space corresponding to a main storage disk; when the data storage status is a second status, controlling a garbage collection program to clear the data stored in the first storage space corresponding to the second status. Compared with the prior art, in the process of metadata garbage collection, the cache disk does not process data storage, which is manifested as the business performance being zero in terms of business. With this method, by allowing the garbage collection program to execute but not allowing the data storage status of the storage space to be changed during the garbage collection process, it is possible to avoid problems such as business interruption and large latency caused by the cache disk not processing data storage during the garbage collection process, and improve data processing efficiency.

[0064] Figure 3 is a flowchart of another data processing method provided by an embodiment of the present invention, as Figure 3 shown, the method specifically includes:

[0065] S31. Obtain the data storage location identified in the data structure.

[0066] In an embodiment of the present invention, application data to be stored is obtained and the application data is cached in a first storage space corresponding to a cache disk according to a preset data structure. Due to the full mapping relationship between the cache disk and multiple main storage disks, in addition to the cache metadata location in the cache disk, a space in the cache disk may be mapped to any space in the main storage disk. All the metadata of the application data I / O is organized into a B+ tree data structure, which can accelerate the query speed and the swapping speed between memory and disk. The organizational relationship between data and metadata is as Figure 5 shown.

[0067] According to the organizational relationship between data and metadata, the data storage location identified in the data structure (existing only in the cache disk or existing in both the cache disk and the main storage disk) can be obtained.

[0068] S32. If the data stored in the current first storage space does not exist in the main storage disk, determine that the data storage status of the current first storage space is the first status.

[0069] S33. If the data stored in the current first storage space exists in the main storage disk, determine that the data storage status of the current first storage space is the second status.

[0070] After caching data in the first storage space and the data does not exist in the main disk, determine that dirty_data = 1, that is, the data storage state of the first storage space is the first state, and the first state means that the data stored in the first storage space does not exist in the main disk.

[0071] Optionally, when no data is cached in the first storage space, determine that dirty_data = 0, that is, the data storage state of the first storage space is the second state, and the second state means that no cached data exists in the first storage space.

[0072] S34. Control the garbage collection program to clear the data stored in the first storage space corresponding to the second state.

[0073] S35. Release the first storage space after data clearing for re-caching data.

[0074] When the data storage state of the first storage space is the second state, control the garbage collection program to clear the data stored in the first storage space corresponding to the second state, and release the first storage space after data clearing for re-caching data.

[0075] It should be noted that when the garbage collection program inspects all the first storage spaces, control the data storage state of all the first storage spaces to remain unchanged, that is, the double-layer state machine of the first storage space becomes a single-layer state machine, and no longer changes the data storage state according to the state machine. During the garbage collection process, allow eviction events to occur, and prohibit the state migration of the first storage space from being triggered during garbage collection. For example Figure 8 This is the schematic diagram of the single-layer state machine provided by the embodiment of the present invention.

[0076] Figure 9 This is the performance comparison curve graph of the cache disk data storage between the prior art and the embodiment of the present invention. It can be seen from the figure that the present invention solves the problem of the write IO performance being zero or having a second-level time delay during the use of Bcache; for 3-node ceph, 128 concurrency, 8KB random write, the performance is improved by more than 40%, the average IOPS is increased from 25914.2 to 36412.9, and the average time delay is decreased from 4.9ms to 3.5ms.

[0077] The data processing method provided by the embodiment of the present invention includes obtaining application data to be stored, and caching the application data in the corresponding first storage space of the cache disk according to a preset data structure; detecting the data storage status of all the first storage spaces of the cache disk in real time; when the data storage status is the first state, transferring the data stored in the first storage space corresponding to the first state to the corresponding second storage space of the main memory disk; when the data storage status is the second state, controlling a garbage collection program to clear the data stored in the first storage space corresponding to the second state. By this method, by allowing the garbage collection program to execute but not allowing the data storage status of the storage space to be changed during the garbage collection process, it is possible to avoid problems such as the cache disk not processing data storage during garbage collection, causing service interruption and large latency, and improving data processing efficiency.

[0078] Figure 10 It is a schematic structural diagram of a data processing device provided by an embodiment of the present invention, specifically including:

[0079] An acquisition module 1001, configured to obtain application data to be stored, and cache the application data in the corresponding first storage space of the cache disk according to a preset data structure;

[0080] A detection module 1002, configured to detect the data storage status of all the first storage spaces of the cache disk in real time;

[0081] A storage module 1003, configured to transfer the data stored in the first storage space corresponding to the first state to the corresponding second storage space of the main memory disk when the data storage status is the first state;

[0082] A garbage collection module 1004, configured to control a garbage collection program to clear the data stored in the first storage space corresponding to the second state when the data storage status is the second state;

[0083] A control module 1005, configured to control the data storage status of all the first storage spaces to remain unchanged when the garbage collection program inspects all the first storage spaces.

[0084] In a possible implementation manner, the storage module 1003 is specifically configured to determine the data storage status of all the first storage spaces of the cache disk based on the data structure corresponding to the data stored in the first storage space.

[0085] In a possible implementation, the storage module 1003 is further configured to obtain the data storage location identified in the data structure; if the data stored in the current first storage space does not exist in the main disk, determine that the data storage status of the current first storage space is the first status; if the data stored in the current first storage space exists in the main disk, determine that the data storage status of the current first storage space is the second status.

[0086] In a possible implementation, the control module 1005 is specifically configured to release the first storage space after data is cleared for re-caching data.

[0087] The data processing device provided in this embodiment may be the data processing device as shown in Figure 10 and can execute all steps of the data processing method as shown in Figure 2-3 to achieve the technical effects of the data processing method as shown in Figure 2-3 For specific details, please refer to Figure 2-3 the relevant description. For the sake of brevity, it will not be elaborated here.

[0088] Figure 11 FIG. is a schematic structural diagram of a computer device provided in an embodiment of the present invention. Figure 11 The computer device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504, and other user interfaces 503. Each component in the computer device 500 is coupled together through a bus system 505. It can be understood that the bus system 505 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 11 all kinds of buses are labeled as the bus system 505.

[0089] Among them, the user interface 503 may include a display, a keyboard, or a pointing device (such as a mouse, a trackball, a touchpad, or a touch screen, etc.).

[0090] It can be understood that the memory 502 in the embodiments of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synch link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The memory 502 described herein is intended to include but not be limited to these and any other suitable types of memory.

[0091] In some embodiments, the memory 502 stores the following elements, executable units or data structures, or subsets or supersets thereof: an operating system 5021 and an application program 5022.

[0092] Among them, the operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., and is used to implement various basic services and process hardware-based tasks. The application program 5022 includes various application programs, such as a media player and a browser, etc., and is used to implement various application services. The program for implementing the method of the embodiments of the present invention can be included in the application program 5022.

[0093] In the embodiments of the present invention, by invoking the programs or instructions stored in the memory 502, specifically, the programs or instructions stored in the application program 5022, the processor 501 is used to execute the method steps provided in each method embodiment, for example, including:

[0094] Obtain the application data to be stored, and cache the application data in the corresponding first storage space of the cache disk according to a preset data structure; detect the data storage status of all the first storage spaces of the cache disk in real time; when the data storage status is the first status, transfer the data stored in the first storage space corresponding to the first status to the corresponding second storage space of the main memory disk; when the data storage status is the second status, control the garbage collection program to clear the data stored in the first storage space corresponding to the second status.

[0095] In a possible implementation manner, when the garbage collection program performs a patrol inspection on all the first storage spaces, control the data storage status of all the first storage spaces to remain unchanged.

[0096] In a possible implementation manner, determine the data storage status of all the first storage spaces of the cache disk based on the data structure corresponding to the data stored in the first storage space.

[0097] In a possible implementation manner, obtain the data storage location identified in the data structure; if the data stored in the current first storage space does not exist in the main memory disk, determine that the data storage status of the current first storage space is the first status; if the data stored in the current first storage space exists in the main memory disk, determine that the data storage status of the current first storage space is the second status.

[0098] In a possible implementation manner, release the first storage space after the data is cleared for re-caching data.

[0099] In a possible implementation manner, there is one cache disk and multiple main memory disks; there is a full mapping of data between the cache disk and the multiple main memory disks.

[0100] The method disclosed in the embodiments of the present invention above can be applied to the processor 501 or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed by the integrated logic circuit in hardware or instructions in software form in the processor 501. The above-mentioned processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software units in the decoding processor. The software unit may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and combines its hardware to complete the steps of the above method.

[0101] It can be understood that these embodiments described herein can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or a combination thereof.

[0102] For software implementation, the techniques described herein can be implemented by units that execute the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented inside or outside the processor.

[0103] The computer device provided in this embodiment may be the computer device shown in Figure 11 and can execute all steps of the data processing method shown in Figure 2-3 , thereby achieving the technical effects of the data processing method shown in Figure 2-3 . For specific details, please refer to Figure 2-3 for relevant descriptions. For the sake of brevity, no further elaboration will be provided here.

[0104] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). One or more programs are stored in this storage medium. Among them, the storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include a combination of the above types of memory.

[0105] When one or more programs in the storage medium can be executed by one or more processors to implement the data processing method executed on the computer device side as described above.

[0106] The processor is used to execute the data processing program stored in the memory to implement the following steps of the data processing method executed on the computer device side:

[0107] Obtain the application data to be stored, and cache the application data in the corresponding first storage space of the cache disk according to a preset data structure; continuously detect the data storage status of all the first storage spaces of the cache disk; when the data storage status is the first state, transfer the data stored in the first storage space corresponding to the first state to the corresponding second storage space of the main storage disk; when the data storage status is the second state, control the garbage collection program to clear the data stored in the first storage space corresponding to the second state.

[0108] In a possible implementation manner, when the garbage collection program inspects all the first storage spaces, control the data storage status of all the first storage spaces to remain unchanged.

[0109] In a possible implementation manner, determine the data storage status of all the first storage spaces of the cache disk based on the data structure corresponding to the data stored in the first storage space.

[0110] In a possible implementation manner, obtain the data storage location identified in the data structure; if the data stored in the current first storage space does not exist in the main storage disk, determine that the data storage status of the current first storage space is the first state; if the data stored in the current first storage space exists in the main storage disk, determine that the data storage status of the current first storage space is the second state.

[0111] In a possible implementation, the first storage space after data clearing is released for re-caching data.

[0112] In a possible implementation, there is one cache disk and multiple main memory disks; there is full data mapping between the cache disk and the multiple main memory disks.

[0113] Those skilled in the art should further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0114] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be implemented by hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0115] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, Including: Obtain application data to be stored, and cache the application data in a first storage space corresponding to a cache disk according to a preset data structure, where the metadata of the application data I / O is organized into a B+ tree data structure; Convert the double-layer state machine of the first storage space into a single-layer state machine, and real-time detect the data storage status of all first storage spaces of the cache disk; When the data storage status is the first status, transfer the data stored in the first storage space corresponding to the first status to a second storage space corresponding to a main memory disk; The first status indicates that the data stored in the first storage space does not exist in the main memory disk; When the data storage status is the second status, control a garbage collection program to clear the data stored in the first storage space corresponding to the second status; The second status indicates that there is no cached data in the first storage space; When the garbage collection program inspects all first storage spaces, control the data storage status of all first storage spaces to remain unchanged.

2. The method according to claim 1, characterized in that The real-time detection of the data storage status of all first storage spaces of the cache disk includes: Based on the data structure corresponding to the data stored in the first storage space, determine the data storage status of all first storage spaces of the cache disk.

3. The method according to claim 2, characterized in that The determining the data storage status of all first storage spaces of the cache disk based on the data structure corresponding to the data stored in the first storage space includes: Obtain the data storage location identified in the data structure; If the data stored in the current first storage space does not exist in the main memory disk, determine that the data storage status of the current first storage space is the first status; If the data stored in the current first storage space exists in the main memory disk, determine that the data storage status of the current first storage space is the second status.

4. The method according to claim 3, wherein After the controlling the garbage collection program to clear the data stored in the first storage space corresponding to the second status when the data storage status is the second status, the method further includes: Release the first storage space after the data is cleared for re-caching data.

5. The method according to claim 1, wherein There is one cache disk and multiple main memory disks; There is a full data mapping between the cache disk and multiple main memory disks.

6. A data processing device, characterized in that, Including: An obtaining module, configured to obtain application data to be stored, and cache the application data in a first storage space corresponding to a cache disk according to a preset data structure, where the metadata of the application data I / O is organized into a B+ tree data structure; A detecting module, configured to convert the double-layer state machine of the first storage space into a single-layer state machine, and real-time detect the data storage status of all first storage spaces of the cache disk; A storing module, configured to transfer the data stored in the first storage space corresponding to the first status to a second storage space corresponding to a main memory disk when the data storage status is the first status; The first status indicates that the data stored in the first storage space does not exist in the main memory disk; A garbage collection module, configured to control a garbage collection program to clear the data stored in the first storage space corresponding to the second status when the data storage status is the second status; The second status indicates that there is no cached data in the first storage space; A control module, configured to control the data storage status of all the first storage spaces to remain unchanged when the garbage collection program performs a patrol inspection on all the first storage spaces.

7. A computer device, characterized in that, It includes: A processor and a memory, where the processor is configured to execute a data processing program stored in the memory to implement the data processing method according to any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method according to any one of claims 1 to 5.

Images