Memory and hard disk integrated storage data processing method, system, device and medium

Abstract

The present application provides a method, system, device and medium for processing data stored in an integrated manner of memory and hard disk. The method utilizes a distributed storage data access method and determines high-heat user data based on the heat information of candidate user data. High-heat user data will be stored in the memory, and candidate user data will be stored in the hard disk. When a user accesses data, first the first cuckoo filter and the second cuckoo filter will filter out invalid user keys during access to reduce the performance pressure on the storage device. For high-heat user data, it will be obtained through memory reading. When the memory cannot obtain the required user data, the hard disk data can be accessed for reading. Therefore, the memory can be called for commonly used user data to increase the reading speed; and when the memory fails, since the hard disk also stores a copy of the user data, no data loss will occur, thereby combining the advantages of the two storage methods.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, system, device and medium for processing data stored in an integrated manner using a memory and a hard disk. Background Art

[0002] A key-value store is a data storage system that stores data in the form of key values, similar to a map in Java. The entire database can be understood as a large map, where each key corresponds to a unique value. Key-value distributed storage systems have fast query speeds, can store large amounts of data, and support high concurrency. They are well suited for queries based on primary keys, but cannot perform complex conditional queries.

[0003] Currently, some key-value storage systems use an all-in-memory architecture to avoid disk read / write bottlenecks and achieve extremely high performance. However, in practice, such systems are only used as front-end caches for databases and rarely serve as the final destination for data. The limitations of in-memory databases are that data stored in memory is easily lost in accidents such as system crashes, and security cannot be guaranteed. Furthermore, the price and energy consumption of memory are still much higher than those of disks, making them uneconomical. Using hard disk storage, data can be persisted on the hard disk. Hard disk storage is much cheaper than in-memory storage, and expansion costs are low. However, the disadvantage of hard disks is that access speeds are slower than memory, making them unsuitable for scenarios where users are sensitive to response times. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a memory and hard disk integrated storage data processing method, system, device and medium, which complement each other and combine the advantages of memory storage and hard disk storage in the key-value storage method to solve the problems of slow reading and writing speeds and easy data loss due to downtime.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a method for processing data stored in an integrated memory and hard disk, the method comprising:

[0006] Obtaining a candidate user key value and corresponding candidate user data, and storing the candidate user data in full to a hard disk, determining popularity information of the candidate user data, determining high-popularity user data from the candidate user data based on the popularity information, and storing the high-popularity user data in full to a memory, wherein the high-popularity user data includes candidate user data whose popularity information meets a preset threshold, and the popularity information includes at least the number of times the candidate user data has been accessed;

[0007] Performing a first update process on the first cuckoo filter according to the high-hot user key value corresponding to the high-hot user data, and performing a second update process on the second cuckoo filter according to the candidate user key value corresponding to the candidate user data;

[0008] Filtering the accessed user key value using the first cuckoo filter after the first update process, if the high-hot user key value is successfully obtained by filtering the first cuckoo filter after the first update process, requesting to access the memory, and if the high-hot user data corresponding to the high-hot user key value is read in the memory, completing the user data reading, and if the high-hot user data corresponding to the high-hot user key value cannot be read in the memory, requesting to access the hard disk to complete the reading of the high-hot user data;

[0009] If filtering by the first cuckoo filter fails, further filtering is performed using the second cuckoo filter after the second update processing. If filtering by the second cuckoo filter after the second update processing successfully obtains the candidate user key value, access to the hard disk is requested to complete the reading of the data in the candidate user data except the high-popularity user data.

[0010] In some embodiments, determining popularity information of the candidate user data and determining high-popularity user data from the candidate user data based on the popularity information includes:

[0011] Establish a popularity information hash table and store it on the hard disk. The data stored in the popularity information hash table includes the candidate user key value and the corresponding popularity information. The popularity information includes a window array, a window cursor, and a popularity value. Each element of the window array is used to store the number of accesses to the candidate user key value in a time segment. The window array is used to store the number of accesses for N consecutive time segments, where N is a positive integer. The size of the window array is N and the initial value of each element is 0. The window cursor is used to point to an element in the window array to determine whether the number of accesses needs to be updated for the time segment. The popularity value is used to determine whether the candidate user data corresponding to the candidate user key value is written to the memory. The initial value of the popularity value is 0.

[0012] If the sum of all elements in the window array is greater than or equal to a preset access count threshold, the heat value is increased by 1; if the heat value is equal to the preset heat threshold, the candidate user data corresponding to the candidate user key value is written into the memory, and then the window cursor is increased by 1 to point to the next element in the window array and the next element is updated; if the heat value is not equal to the preset heat threshold, the window cursor is increased by 1 to point to the next element in the window array and the next element is updated;

[0013] If the sum of all elements in the window array is less than a preset access count threshold, the heat value is reduced by 1; if the heat value is equal to the heat threshold minus 1, the candidate user data corresponding to the candidate user key value is deleted from the memory, and then the window cursor is increased by 1 to point to the next element in the window array and the next element is updated; if the heat value is not equal to the heat threshold minus 1, the window cursor is increased by 1 to point to the next element in the window array and the next element is updated.

[0014] In some embodiments, determining popularity information of the candidate user data and determining high-popularity user data from the candidate user data based on the popularity information further includes:

[0015] If the popularity value is equal to 0, deleting the popularity information of the candidate user key from the popularity information hash table;

[0016] If the heat value is not equal to 0, the window cursor is incremented by 1 to point to the next element in the window array and the next element is updated.

[0017] In some embodiments, the memory and hard disk integrated storage data processing method further includes:

[0018] Acquire first data to be written, where the first data to be written corresponds to the candidate user key value;

[0019] If the second cuckoo filter after the second update process successfully filters and obtains the candidate user key value, a request is made to access the hard disk to complete writing of the first data to be written.

[0020] In some embodiments, the memory and hard disk integrated storage data processing method further includes:

[0021] Storing all the candidate user data in the memory and hard disk respectively;

[0022] performing a third update process on the first cuckoo filter according to the candidate user key value corresponding to the candidate user data;

[0023] The accessed user key value is filtered using the first cuckoo filter after the third update processing. If the candidate user key value is successfully obtained by filtering the first cuckoo filter after the third update processing, a request is made to access the memory to complete the reading of the candidate user data. If the memory fails, a request is made to access the hard disk to complete the reading of the candidate user data, and the candidate user data is written into the memory.

[0024] In some embodiments, the memory and hard disk integrated storage data processing method further includes:

[0025] Acquire second data to be written, where the second data to be written corresponds to the candidate user key value;

[0026] If the first cuckoo filter after the third update process successfully filters and obtains the candidate user key value, a request is made to access the memory to complete writing of the second data to be written, and the second data to be written is asynchronously written to the hard disk.

[0027] In some embodiments, the memory and hard disk integrated storage data processing method further includes:

[0028] Storing all the candidate user data in a hard disk;

[0029] performing a fourth update process on the second cuckoo filter according to the candidate user key value corresponding to the candidate user data;

[0030] Filtering the accessed user key value using the second cuckoo filter after the fourth update process;

[0031] If the candidate user key value is successfully obtained by filtering the second cuckoo filter after the fourth update process, requesting access to the hard disk to complete reading of the candidate user data;

[0032] Acquire third data to be written, where the third data to be written corresponds to the candidate user key value;

[0033] If the second cuckoo filter after the fourth update process successfully obtains the candidate user key value, a request is made to access the hard disk to complete writing of the third data to be written.

[0034] To achieve the above objectives, a second aspect of an embodiment of the present application provides a memory and hard disk integrated storage data processing application system, comprising:

[0035] An integrated storage data processing module includes an upper-layer application, an integrated storage data processing SDK toolkit, a first cuckoo filter, and a second cuckoo filter. The upper-layer application is used to call the integrated storage data processing SDK toolkit, and the integrated storage data processing SDK toolkit is used to execute the memory and hard disk integrated storage data processing method described in the first aspect. The first cuckoo filter and the second cuckoo filter are both used to filter accessed user key values.

[0036] The storage module includes a hard disk and a memory, wherein the hard disk is used to store the candidate user data, and the memory is used to store the high-popularity user data or the candidate user data.

[0037] To achieve the above-mentioned objectives, a third aspect of the embodiments of the present application provides a computer device, including:

[0038] at least one memory;

[0039] at least one processor;

[0040] at least one computer program;

[0041] The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement:

[0042] A method for processing data stored in an integrated memory and hard disk as described in the first aspect above.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute a memory and hard disk integrated storage data processing method described in the first aspect above.

[0044] The present application proposes a memory and hard disk integrated storage data processing method, system, device and medium, which utilizes a distributed storage data access method, that is, accessing candidate user data by corresponding candidate user key values, and determining high-hot user data from the candidate user data based on its popularity information. The high-hot user data will be stored in the memory, while the candidate user data will be stored in the hard disk. Therefore, when a user accesses data, the first cuckoo filter and the second cuckoo filter will first filter out invalid user keys during access to prevent a flood attack caused by a large number of invalid key accesses from causing cache breakdown, thereby coping with the performance pressure brought by the memory and hard disk, greatly enhancing the robustness of the application; then, for high-hot user data with a relatively higher access frequency, it will be obtained through memory reading. When the memory cannot obtain the required user data, the hard disk can finally be accessed to read the required user data. Therefore, by using this integrated storage for data processing, the memory can be called for frequently used user data to improve the reading speed; and when the memory fails, such as when the system crashes, the hard disk also stores a copy of the user data, so no data loss will occur, thus combining the advantages of the two storage methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a hybrid storage mode of a method for processing data stored in memory and hard disk integrated according to an embodiment of the present application;

[0046] Figure 2 This is a flowchart of a method for determining high-profile user data provided by an embodiment of the present application;

[0047] Figure 3 This is a flow chart of a high-speed storage mode of a method for processing data stored in an integrated manner using memory and a hard disk, as provided in an embodiment of the present application;

[0048] Figure 4 This is a flow chart of a low-speed storage mode of a method for processing data stored in an integrated memory and hard disk provided in an embodiment of the present application;

[0049] Figure 5 Schematic diagram of a memory and hard disk integrated storage data processing application system provided in an embodiment of the present application;

[0050] Figure 6 This is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0054] First, some terms used in this application are explained:

[0055] Distributed key-value store: A type of NoSQL storage. Its data is organized, indexed, and stored in the form of key-value pairs. Key-value stores are well-suited for business data that doesn't involve excessive data relationships or business relationships. They can also effectively reduce the number of disk reads and writes, offering better read and write performance than SQL database storage. Commonly used high-performance KV storage systems include Redis, Memcached, etcd, and Zookeeper. Redis is primarily used to cache business data. In some embodiments, Redis is often used as the memory in the embodiments. Redis is essentially a key-value in-memory database. The entire database is loaded into memory for operations, and database data is periodically flushed to the hard disk through asynchronous operations for storage. Because it operates purely in memory, Redis offers excellent performance, processing over 100,000 read and write operations per second. However, its main drawback is that its database capacity is limited by physical memory, making it unsuitable for high-performance read and write operations on large amounts of data. Therefore, Redis's suitability is primarily limited to high-performance operations and calculations on smaller data volumes.

[0056] Cuckoo Filter: The Cuckoo Filter consists of an array with 4-byte elements, which can store 4 fingerprints, each occupying 1 byte (128 types). Multiple seats on the same array element are contiguous in memory, effectively utilizing CPU cache.

[0057] The Cuckoo filter includes the following operations: Insertion: First, the data's fingerprint and hash value are calculated. Another hash value is then calculated using the fingerprint and hash value. The two hash values are mapped to two locations (since two locations are calculated, each storing four fingerprints, a maximum of eight identical objects can be stored). Next, during insertion, two attempts are made to insert the fingerprint into the location. If both attempts fail, a random fingerprint is removed and a new location is found for that fingerprint (the capacity is expanded after the maximum number of attempts is exceeded). Expansion: If the array is too small, circular attempts can occur. A maximum number of attempts can be set. If this number is exceeded, the capacity is expanded and the location of each fingerprint is recalculated. Search: The hash value is calculated to obtain two elements. The fingerprints of the eight locations in the two elements are compared. If the comparison is successful, the data exists. If the hash value and fingerprint match, a false positive occurs (a small probability). Delete: Because each object's fingerprint is stored in a single location, data can be deleted by deleting the fingerprint. The delete function cannot be used if more than eight identical objects are stored. If two data items have the same hash value and fingerprint, a false deletion may occur. Update: This involves deleting and then adding a new fingerprint.

[0058] The advantages of the Cuckoo filter are: low memory access times; simple hash function calculation; a delete operation exists, and if the number of identical data does not exceed 8, the delete operation is available; however, because the calculated fingerprint information is stored, there is a possibility of accidental deletion.

[0059] The disadvantages of the cuckoo filter are: the memory space is not connected, the CPU consumption is high; it is easy to have a filling cycle problem, and the original data is kicked out due to a hash conflict, but the original data still has conflicts; when deleting data, hash conflicts will cause accidental deletions, and if there is a misjudgment in the query, the deletion will also be accidental.

[0060] A hash table is a data structure that allows direct access based on a key value. In other words, it accesses records by mapping a key value to a location in the table, speeding up lookups. This mapping function is called a hash function, and the array storing the records is called a hash table. Given a table M, there exists a function f(key). If, for any given key value, key, the function returns the address of the record in the table containing that key, then the table M is called a hash table, and the function f(key) is called a hash function. In a hash table, we can use the hash function to quickly access target data in an array. If a hash conflict occurs, a linked list is used for storage. This allows us to handle large and small amounts of data. Because of their flexibility in data storage and efficiency in data lookups, hash tables are frequently used in programming languages, such as associative arrays.

[0061] Currently, Redis is widely used as a high-performance cache in various businesses, such as the financial and insurance sectors. However, Redis currently has some drawbacks, such as high memory usage, poor data reliability, and the cumbersome task of maintaining cache and storage consistency. However, simply replacing Redis with a hard drive has significant disadvantages: access speeds are slower than in-memory, making it unsuitable for scenarios where users are sensitive to response time.

[0062] Based on this, the embodiments of the present application provide a memory and hard disk integrated storage data processing method, system, device and medium, which can combine a storage method similar to Redis memory with hard disk storage to overcome the shortcomings of using memory and hard disk separately.

[0063] The embodiments of the present application provide a method, system, device and medium for processing memory and hard disk integrated storage data, which are specifically illustrated through the following embodiments. First, a method for processing memory and hard disk integrated storage data in the embodiments of the present application is described.

[0064] The embodiment of the present application provides a method for processing memory and hard disk integrated storage data, which relates to the field of distributed storage technology. The embodiment of the present application provides a method for processing memory and hard disk integrated storage data, which can be applied to a terminal, a server side, or a software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements information extraction methods, etc., but is not limited to the above forms.

[0065] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0066] See also Figure 1 , Figure 1 This is an optional flow chart of a hybrid storage mode of the memory and hard disk integrated storage data processing method provided in an embodiment of the present application. Figure 1 The method may specifically include but is not limited to steps S110 to S140.

[0067] Step S110: Obtain candidate user key values and corresponding candidate user data, store all candidate user data in a hard disk, determine popularity information of the candidate user data, determine high-popularity user data from the candidate user data based on the popularity information, and store all high-popularity user data in a memory. The high-popularity user data includes candidate user data whose popularity information meets a preset threshold, and the popularity information includes at least the number of times the candidate user data has been accessed.

[0068] Step S120: performing a first update process on the first cuckoo filter according to the high-frequency user key value corresponding to the high-frequency user data, and performing a second update process on the second cuckoo filter according to the candidate user key value corresponding to the candidate user data;

[0069] Step S130: Filtering the accessed user key value using the first cuckoo filter after the first update process. If the high-hot user key value is successfully obtained through the first cuckoo filter after the first update process, a request is made to access the memory. If the high-hot user data corresponding to the high-hot user key value is read in the memory, the user data reading is completed. If the high-hot user data corresponding to the high-hot user key value cannot be read in the memory, a request is made to access the hard disk to complete the reading of the high-hot user data.

[0070] Step S140: If the filtering by the first cuckoo filter fails, further filtering is performed using the second cuckoo filter after the second update process. If the filtering by the second cuckoo filter after the second update process successfully obtains the candidate user key value, a request is made to access the hard disk to complete the reading of the data in the candidate user data except the high-popularity user data.

[0071] In some embodiments, in steps S110 to S140, a distributed storage data access method is used, i.e., candidate user data is accessed by corresponding candidate user key values. High-hot user data is determined based on the popularity information of the candidate user data. The high-hot user data is stored in the memory, while the candidate user data is stored in the hard disk. Therefore, when a user accesses data, the first cuckoo filter and the second cuckoo filter will first filter out invalid user keys during access to prevent a flood attack caused by a large number of invalid key accesses from causing cache breakdown, thereby coping with the performance pressure brought by the memory and hard disk, greatly enhancing the robustness of the application. Then, for high-hot user data that is accessed relatively frequently, it will be obtained by reading from the memory. When the memory cannot obtain the required user data, the hard disk can finally be accessed to read the required user data. Therefore, by using this integrated storage for data processing, the memory can be called for frequently used user data to increase the reading speed. When the memory fails, such as when the system crashes, the hard disk also stores a copy of the user data, so no data loss will occur, thus combining the advantages of the two storage methods.

[0072] In step S110 of some embodiments, the user data established by the entire system, i.e., the candidate user data, will be fully stored in the hard disk for long-term preservation of the data; and the user data that needs to be frequently called and processed, i.e., the high-popularity user data selected from the candidate user data, will be stored in the memory for fast reading.

[0073] In step S120 of some embodiments, it can be understood that the candidate user data initially stored in the hard disk will be filtered using the second cuckoo to eliminate other access key values that are not related to the candidate user key value. Therefore, in the second cuckoo filter, the key of valid access, that is, the candidate user key value, needs to be updated to the second cuckoo filter; similarly, for the high-heat user data initially stored in the memory, the first cuckoo will be filtered to eliminate other access key values that are not related to the high-heat user key value. Therefore, in the first cuckoo filter, the key of valid access, that is, the high-heat user key value, needs to be updated to the first cuckoo filter.

[0074] In step S130 of some embodiments, it can be understood that for the key-value storage system, the value corresponding to the key, that is, the user data, is read through key access. Therefore, the first cuckoo filter is used to filter out a large number of invalid keys during key access. Only when the access key belongs to a high-popularity user key value, the access is established to request access to the corresponding high-popularity user data in the memory.

[0075] If the high-profile user data cannot be read from the memory when access is established, it may be due to a memory failure or downtime, or the high-profile user data that may be accessed has been deleted from the memory. Therefore, in this case, in order to successfully complete the reading of the high-profile user data, the hard disk will be accessed. It is understandable that the candidate user data stored on the hard disk includes the high-profile user data. Through the corresponding query access, the required high-profile user data can be read.

[0076] In step S140 of some embodiments, it can be understood that if no high-popularity user key value is obtained after filtering by the first cuckoo filter, it means that the key accessed at this time does not belong to the high-popularity user key value, but may be other candidate user key values with a lower access frequency. Therefore, it is necessary to further filter through the second cuckoo filter. If a candidate user key value is obtained by filtering, the corresponding candidate user data in the hard disk is read.

[0077] It should be noted that if the access key is still not matched after passing through the first cuckoo filter and the second cuckoo filter, it means that the relevant information of the access key at this time is not stored in the storage module, that is, not stored in the memory or hard disk, and the data reading result at this time is failed.

[0078] See also Figure 2 , Figure 2 This is a flowchart of a method for determining high-profile user data in some embodiments of the present application. In some embodiments of the present application, the method for determining high-profile user data specifically includes but is not limited to steps S111 to S113. Figure 2 These three steps are introduced in detail.

[0079] Step S111: Create a popularity information hash table and store it on the hard disk. The data stored in the popularity information hash table includes candidate user key values and corresponding popularity information. The popularity information includes a window array, a window cursor, and a popularity value. Each element of the window array is used to store the number of accesses to the candidate user key value in a time segment. The window array is used to store the number of accesses to N consecutive time segments, where N is a positive integer. The size of the window array is N and the initial value of each element is 0. The window cursor is used to point to an element in the window array to determine whether the number of accesses to a time segment needs to be updated. The popularity value is used to determine whether the candidate user data corresponding to the candidate user key value is written into the memory. The initial value of the popularity value is 0.

[0080] Step S112: If the sum of all elements in the window array is greater than or equal to a preset access count threshold, the popularity value is increased by 1. If the popularity value is equal to the preset popularity threshold, the candidate user data corresponding to the candidate user key value is written into the memory, and then the window cursor is increased by 1 to point to the next element in the window array and the next element is updated. If the popularity value is not equal to the preset popularity threshold, the window cursor is increased by 1 to point to the next element in the window array and the next element is updated.

[0081] Step S113: If the sum of all elements in the window array is less than the preset access count threshold, the heat value is reduced by 1. If the heat value is equal to the heat threshold minus 1, the candidate user data corresponding to the candidate user key value is deleted from the memory, and then the window cursor is increased by 1 to point to the next element in the window array and the next element is updated. If the heat value is not equal to the heat threshold minus 1, the window cursor is increased by 1 to point to the next element in the window array and the next element is updated.

[0082] In step S111 of some embodiments, it can be understood that high-popularity user data is determined based on popularity information, and the popularity information is represented in the form of a hash table, that is, a popularity information hash table, which specifically is a data structure of multiple hashes composed of multiple candidate user key values and corresponding multiple popularity information in pairs to form a hash table, wherein each popularity information includes a window array, a window cursor, and a popularity value.

[0083] The following is an explanation of the heat information using a specific embodiment. For a certain candidate user key value, assuming that the total duration of the heat information statistics is 10 minutes, and a time segment is set to 1 minute, the size of the window array is set to 10 elements. The value of each element represents the number of visits to the candidate user key value in a time segment, and the window cursor is used to point to the element that needs to be updated in the window array. Specifically, the first element records the number of visits in the first 1 minute, the second element records the number of visits in the next 1 minute, and so on. The window array will be filled with 10 1-minute visits. Assuming that it starts from time 0, at 1 minute, the position pointed to by the window cursor is the 1st element. Since time is continuous, it can be understood that at 10 minutes, the position pointed to by the window cursor is the 10th element. When the time reaches 11 minutes, the cursor will return to the first element, pointing to the first element to update the access count of the candidate user key value in the 11th minute. At this time, the access count originally recorded in the 1st minute will be replaced by the access count in the 11th minute. Similarly, when the time reaches 12 minutes, the window cursor will point to the second element, and the initial access count in the 2nd minute will be replaced by the access count in the 12th minute. And so on, the window cursor will cycle over time. Therefore, it can be understood that the window array always stores the access count for a continuous period of 10 minutes, and the access count is updated every 1 minute.

[0084] In step S112 of some embodiments, based on the aforementioned embodiments, the following specifically explains how to determine high-hot user data through heat information. For each total duration, that is, each 10-minute duration, all elements of the window array are first added together to obtain the total number of visits to the candidate user key value within this 10-minute period. When this total number of visits is greater than a preset visit threshold, the heat value is increased by 1. When the heat value is equal to the preset heat threshold, the candidate user data corresponding to the candidate user key value is written to the memory, that is, the candidate user data is high-hot user data; if the heat value is not equal to the preset heat threshold, the elements in the window array continue to be updated over time.

[0085] It can be understood that the above judgment process can be performed once every 1 minute. For every consecutive 10 minutes, if the candidate user key value access always meets the characteristics of high-frequency access in a continuous time period, that is, the total number of accesses is greater than the preset access number threshold, then it can be understood that the heat value continues to accumulate in this continuous time period to reach the heat threshold size, and the candidate user data is finally considered to be high-heat user data.

[0086] In step S113 of some embodiments, based on the aforementioned embodiments, it can be understood that, for the high-hot data at this time, once the total number of accesses is less than the preset access threshold, the hot value is reduced by 1, indicating that the candidate user key-value access no longer meets the characteristics of high-frequency access, that is, the hot value is equal to the hot threshold minus 1. At this time, the candidate user data is no longer high-hot user data, and the candidate user data will be deleted from the memory and the next element will be updated. If the hot value is not equal to the hot threshold minus 1, it is some other situation, and the subsequent access count update of the next element will be performed normally.

[0087] See also Figure 2 , Figure 2 This is a flowchart of a method for determining high-profile user data in some embodiments of the present application. In some embodiments of the present application, the method for determining high-profile user data further includes but is not limited to step S114 and step S115. Figure 2 These two steps are introduced in detail.

[0088] Step S114: if the popularity value is equal to 0, then delete the popularity information of the candidate user key from the popularity information hash table;

[0089] Step S115: If the heat value is not equal to 0, the window cursor is incremented by 1 to point to the next element in the window array and the next element is updated.

[0090] In steps S114 to S115 of some embodiments, it can be understood that when the total number of accesses to a certain candidate user data maintains a low-frequency access for a long period of time, and when its heat value is as low as 0, it is considered that the candidate user data does not have the possibility of being identified as high-heat user data, and the corresponding hash will be deleted from the heat information hash table to improve the efficiency of the overall data processing operation, reduce storage pressure and improve performance.

[0091] See also Figure 1 , Figure 1 This is an optional flow chart of a hybrid storage mode of the memory and hard disk integrated storage data processing method in some embodiments of the present application. In some embodiments of the present application, the hybrid storage mode of the memory and hard disk integrated storage data processing method specifically further includes steps S150 and S160. Figure 1 These two steps are introduced in detail.

[0092] Step S150: obtaining first data to be written, where the first data to be written corresponds to the candidate user key value;

[0093] Step S160: If the second cuckoo filter after the second update process successfully filters and obtains the candidate user key value, a request is made to access the hard disk to complete writing of the first data to be written.

[0094] In steps S150 and S160 of some embodiments, it can be understood that by utilizing the memory and hard disk integrated storage data processing method of the embodiments of the present invention, on the basis of implementing the user data reading of steps S110 to S140, the user data can be further written.

[0095] See also Figure 3 , Figure 3 This is an optional flow chart of the high-speed storage mode of the memory and hard disk integrated storage data processing method in some embodiments of the present application. In some embodiments of the present application, the high-speed storage mode of the memory and hard disk integrated storage data processing method specifically includes but is not limited to steps S210 to S230. Figure 3 These three steps are introduced in detail.

[0096] Step S210: storing all candidate user data in the memory and hard disk respectively;

[0097] Step S220: performing a third update process in the first cuckoo filter according to the candidate user key value corresponding to the candidate user data;

[0098] Step S230: Filter the accessed user key value using the first cuckoo filter after the third update processing. If the first cuckoo filter after the third update processing successfully obtains the candidate user key value, request access to the memory to complete the reading of the candidate user data. If the memory fails, request access to the hard disk to complete the reading of the candidate user data and write the candidate user data into the memory.

[0099] In some embodiments, steps S210 through S230 can be understood to represent the hybrid storage mode of an embodiment of the present invention. Based on this, steps S210 through S230 represent the data reading steps of the high-speed storage mode of an embodiment of the present invention. Specifically, candidate user data is stored in both memory and hard disk. After filtering by the first cuckoo filter, a request is made to access the memory to read the candidate user data. If the memory fails, a request is made to access the hard disk to complete the user data read. In high-speed storage mode, all candidate user data is accessed and read at high speed through memory. Therefore, it can be understood that data read speed is traded for storage space.

[0100] See also Figure 3 , Figure 3This is an optional flow chart of the high-speed storage mode of the memory and hard disk integrated storage data processing method in some embodiments of the present application. In some embodiments of the present application, the high-speed storage mode of the memory and hard disk integrated storage data processing method specifically includes but is not limited to steps S240 and S250. Figure 3 These two steps are introduced in detail.

[0101] Step S240: obtaining second data to be written, where the second data to be written corresponds to the candidate user key value;

[0102] Step S250: If the first cuckoo filter after the third update process successfully filters and obtains the candidate user key value, a request is made to access the memory to complete writing of the second data to be written, and the second data to be written is asynchronously written to the hard disk.

[0103] In steps S240 and S250 of some embodiments, it is understood that, based on steps S210 to S230, the high-speed storage mode of the present invention can further perform a data write operation. Specifically, after the first cuckoo filter is successfully filtered, the data is first written to the memory and then asynchronously written to the hard disk.

[0104] See also Figure 4 , Figure 4 This is an optional flow chart of the low-speed storage mode of the memory and hard disk integrated storage data processing method in some embodiments of the present application. In some embodiments of the present application, the low-speed storage mode of the memory and hard disk integrated storage data processing method specifically includes but is not limited to steps S310 to S360. Figure 4 These six steps are introduced in detail.

[0105] Step S310: storing all candidate user data in the hard disk;

[0106] Step S320: performing a fourth update process on the second cuckoo filter according to the candidate user key value corresponding to the candidate user data;

[0107] Step S330: Filter the accessed user key value using the second cuckoo filter after the fourth update process;

[0108] Step S340: If the candidate user key value is successfully obtained by filtering the second cuckoo filter after the fourth update process, a request is made to access the hard disk to complete reading of the candidate user data;

[0109] Step S350: obtaining third data to be written, where the third data to be written corresponds to the candidate user key value;

[0110] Step S360: If the second cuckoo filter after the fourth update process successfully obtains the candidate user key value, a request is made to access the hard disk to complete writing of the third data to be written.

[0111] In some embodiments, steps S310 to S360 may also include a low-speed storage mode. Specifically, this mode utilizes only the hard disk, and data can be read from or written to the hard disk after successful filtering by the second cuckoo filter. In this mode, the performance pressure on the memory is reduced, but this comes at the expense of slower read and write speeds.

[0112] The embodiment of the present application provides a method for processing data stored in an integrated memory and hard disk. By utilizing a distributed storage data access method, that is, accessing candidate user data by corresponding candidate user key values, specifically, selecting memory or hard disk access according to a hybrid storage mode, a high-speed storage mode, and a low-speed storage mode, the reading and writing of user data are completed; high-heat user data is determined from the candidate user data according to its heat information, specifically, by performing relevant judgment operations on the heat information carried by the heat information hash table, high-heat user data can be determined; high-heat user data will be stored in the memory, and candidate user data will be stored in the hard disk. Therefore, when a user accesses data, first the first cuckoo filter and the second cuckoo filter will filter out invalid user keys during access to prevent a flood attack caused by a large number of invalid key accesses from causing cache breakdown, thereby coping with the performance pressure brought by the memory and hard disk, and greatly enhancing the robustness of the application; then for high-heat user data with a relatively higher access frequency, it will be obtained by reading from the memory. When the memory cannot obtain the required user data, the hard disk can finally be accessed to read the required user data. Therefore, by utilizing this integrated storage for data processing, frequently used user data can be accessed from the internal memory to improve read speed. Furthermore, in the event of a memory failure, such as a system outage, data loss is prevented because the hard disk also stores a copy of the user data, thereby combining the advantages of both storage methods. Furthermore, high-speed and low-speed storage modes can be employed, thereby enhancing the versatility of the data processing method of the present invention to accommodate diverse scenarios.

[0113] See also Figure 5 The embodiment of the present application also provides a memory and hard disk integrated storage data processing application system, which can implement the above-mentioned memory and hard disk integrated storage data processing method. The application system includes an integrated storage data processing module and a storage module.

[0114] The integrated storage data processing module includes an upper-layer application 410, an integrated storage data processing SDK toolkit 420, a first cuckoo filter 430, and a second cuckoo filter 440. The upper-layer application 410 is used to call the integrated storage data processing SDK toolkit 420, and the integrated storage data processing SDK toolkit 420 is used to execute any one of the memory and hard disk integrated storage data processing methods in the embodiments of the present invention. The first cuckoo filter 430 and the second cuckoo filter 440 are both used to filter accessed user key values.

[0115] The storage module includes a hard disk 450 and a memory 460 . The hard disk 450 is used to store candidate user data, and the memory 460 is used to store high-hot user data or candidate user data.

[0116] It can be understood that the user calls the integrated storage data processing SDK toolkit 420 through the upper-layer application 410 via the API, and the integrated storage data processing SDK toolkit 420 integrates the functions of implementing any memory and hard disk integrated storage data processing method in the embodiments of the present invention.

[0117] It should be noted that the memory and hard disk integrated storage data processing application system of the embodiment of the present application is used to implement the above-mentioned memory and hard disk integrated storage data processing method. The use of some modules or components in the application system of the embodiment of the present application corresponds to the above-mentioned data processing method. Please refer to the above-mentioned data processing method for the specific processing process, which will not be repeated here.

[0118] The embodiment of the present application provides a memory and hard disk integrated storage data processing application system, which can implement the above-mentioned memory and hard disk integrated storage data processing method. When running the storage data processing SDK toolkit, by utilizing the data access method of distributed storage, that is, by correspondingly accessing the candidate user data by the candidate user key value, the high-heat user data is determined from the candidate user data according to its heat information. The high-heat user data will be stored in the memory, and the candidate user data will be stored in the hard disk. Therefore, when the user accesses the data, first the first cuckoo filter and the second cuckoo filter will filter out invalid user keys during access to prevent a flood attack caused by a large number of invalid key accesses from causing cache breakdown, thereby coping with the performance pressure brought by the memory and the hard disk, and greatly enhancing the robustness of the application; then, for the high-heat user data that is accessed relatively more frequently, it will be obtained through memory reading. When the memory cannot obtain the required user data, the hard disk can finally be accessed to read the required user data. Therefore, when using this integrated storage for data processing, the memory can be called for frequently used user data to improve the reading speed; and when the memory fails, such as a crash, the hard disk also stores a copy of the user data, so there will be no data loss, thus combining the advantages of the two storage methods.

[0119] The present application also provides a computer device comprising: at least one memory, at least one processor, and at least one computer program, wherein the at least one computer program is stored in the at least one memory and the at least one processor executes the at least one computer program to implement any of the methods for processing data stored in integrated memory and hard disk according to the aforementioned embodiments. The computer device may be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0120] See also Figure 6 , Figure 6 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0121] The processor 510 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0122] The memory 520 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 520 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 520 and is called by the processor 510 to execute a text information extraction method of the embodiments of this application;

[0123] Input / output interface 530, used to implement information input and output;

[0124] Communication interface 540, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0125] bus 550 , which transmits information between various components of the device (e.g., processor 510 , memory 520 , input / output interface 530 , and communication interface 540 );

[0126] The processor 510 , the memory 520 , the input / output interface 530 and the communication interface 540 are connected to each other in communication within the device via a bus 550 .

[0127] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute any one of the memory and hard disk integrated storage data processing methods in the above embodiments.

[0128] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0129] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0130] It will be understood by those skilled in the art that Figures 1 to 6 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0132] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0133] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0134] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0139] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for processing data stored in memory and hard disk integration, characterized in that: The method comprises: Obtaining a candidate user key value and corresponding candidate user data, and storing the candidate user data in full to a hard disk, determining popularity information of the candidate user data, determining high-popularity user data from the candidate user data based on the popularity information, and storing the high-popularity user data in full to a memory, wherein the high-popularity user data includes candidate user data whose popularity information meets a preset threshold, and the popularity information includes at least the number of times the candidate user data has been accessed; Performing a first update process on the first cuckoo filter according to the high-hot user key value corresponding to the high-hot user data, and performing a second update process on the second cuckoo filter according to the candidate user key value corresponding to the candidate user data; Filtering the accessed user key value using the first cuckoo filter after the first update process, if the high-hot user key value is successfully obtained by filtering the first cuckoo filter after the first update process, requesting to access the memory, and if the high-hot user data corresponding to the high-hot user key value is read in the memory, completing the user data reading, and if the high-hot user data corresponding to the high-hot user key value cannot be read in the memory, requesting to access the hard disk to complete the reading of the high-hot user data; If filtering by the first cuckoo filter fails, further filtering is performed using the second cuckoo filter after the second update processing. If filtering by the second cuckoo filter after the second update processing successfully obtains the candidate user key value, access to the hard disk is requested to complete the reading of the data in the candidate user data except the high-popularity user data.

2. The method for processing data stored in memory and hard disk integration according to claim 1, characterized in that: The determining the popularity information of the candidate user data, and determining high-popularity user data from the candidate user data according to the popularity information, includes: Establish a popularity information hash table and store it on the hard disk. The data stored in the popularity information hash table includes the candidate user key value and the corresponding popularity information. The popularity information includes a window array, a window cursor, and a popularity value. Each element of the window array is used to store the number of accesses to the candidate user key value in a time segment. The window array is used to store the number of accesses for N consecutive time segments, where N is a positive integer. The size of the window array is N and the initial value of each element is 0. The window cursor is used to point to an element in the window array to determine whether the number of accesses needs to be updated for the time segment. The popularity value is used to determine whether the candidate user data corresponding to the candidate user key value is written to the memory. The initial value of the popularity value is 0. If the sum of all elements in the window array is greater than or equal to a preset access count threshold, the heat value is increased by 1; if the heat value is equal to the preset heat threshold, the candidate user data corresponding to the candidate user key value is written into the memory, and then the window cursor is increased by 1 to point to the next element in the window array and the next element is updated; if the heat value is not equal to the preset heat threshold, the window cursor is increased by 1 to point to the next element in the window array and the next element is updated; If the sum of all elements in the window array is less than a preset access count threshold, the heat value is reduced by 1; if the heat value is equal to the heat threshold minus 1, the candidate user data corresponding to the candidate user key value is deleted from the memory, and then the window cursor is increased by 1 to point to the next element in the window array and the next element is updated; if the heat value is not equal to the heat threshold minus 1, the window cursor is increased by 1 to point to the next element in the window array and the next element is updated.

3. The method for processing data stored in memory and hard disk integration according to claim 2, characterized in that: The determining of popularity information of the candidate user data, and determining high-popularity user data from the candidate user data according to the popularity information, further includes: If the popularity value is equal to 0, deleting the popularity information of the candidate user key from the popularity information hash table; If the heat value is not equal to 0, the window cursor is incremented by 1 to point to the next element in the window array and the next element is updated.

4. The method for processing data stored in memory and hard disk integration according to claim 1, characterized in that: The method further comprises: Acquire first data to be written, where the first data to be written corresponds to the candidate user key value; If the second cuckoo filter after the second update process successfully filters and obtains the candidate user key value, a request is made to access the hard disk to complete writing of the first data to be written.

5. The method for processing data stored in memory and hard disk integration according to claim 1, characterized in that: The method further comprises: Storing all the candidate user data in the memory and hard disk respectively; performing a third update process on the first cuckoo filter according to the candidate user key value corresponding to the candidate user data; The accessed user key value is filtered using the first cuckoo filter after the third update processing. If the candidate user key value is successfully obtained by filtering the first cuckoo filter after the third update processing, a request is made to access the memory to complete the reading of the candidate user data. If the memory fails, a request is made to access the hard disk to complete the reading of the candidate user data, and the candidate user data is written into the memory.

6. The method for processing data stored in memory and hard disk integration according to claim 5, characterized in that: The method further comprises: Acquire second data to be written, where the second data to be written corresponds to the candidate user key value; If the first cuckoo filter after the third update process successfully filters and obtains the candidate user key value, a request is made to access the memory to complete writing of the second data to be written, and the second data to be written is asynchronously written to the hard disk.

7. The method for processing data stored in memory and hard disk integration according to claim 1, characterized in that: The method further comprises: Storing all the candidate user data in a hard disk; performing a fourth update process on the second cuckoo filter according to the candidate user key value corresponding to the candidate user data; Filtering the accessed user key value using the second cuckoo filter after the fourth update process; If the candidate user key value is successfully obtained by filtering the second cuckoo filter after the fourth update process, requesting access to the hard disk to complete reading of the candidate user data; Acquire third data to be written, where the third data to be written corresponds to the candidate user key value; If the second cuckoo filter after the fourth update process successfully obtains the candidate user key value, a request is made to access the hard disk to complete writing of the third data to be written.

8. A memory and hard disk integrated storage data processing application system, characterized in that: include: An integrated storage data processing module, comprising an upper-layer application, an integrated storage data processing SDK toolkit, a first cuckoo filter, and a second cuckoo filter, wherein the upper-layer application is used to call the integrated storage data processing SDK toolkit, the integrated storage data processing SDK toolkit is used to execute the memory and hard disk integrated storage data processing method according to any one of claims 1 to 7, and the first cuckoo filter and the second cuckoo filter are both used to filter accessed user key values; The storage module includes a hard disk and a memory, wherein the hard disk is used to store the candidate user data, and the memory is used to store the high-popularity user data or the candidate user data.

9. A computer device, characterized in that: include: at least one memory; at least one processor; at least one computer program; The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement: A method for processing data stored in an integrated memory and hard disk as claimed in any one of claims 1 to 7.

10. A storage medium, wherein the storage medium is a computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to cause a computer to execute: A method for processing data stored in an integrated memory and hard disk as claimed in any one of claims 1 to 7.

Images